[jira] [Commented] (SPARK-26265) deadlock between TaskMemoryManager and BytesToBytesMap$MapIterator

2018-12-05 Thread qian han (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711079#comment-16711079 ] qian han commented on SPARK-26265: -- # There are hundreds of thousand application running on our cluster

[jira] [Commented] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711068#comment-16711068 ] Apache Spark commented on SPARK-26288: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26288: Assignee: (was: Apache Spark) > add initRegisteredExecutorsDB in

[jira] [Assigned] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26288: Assignee: Apache Spark > add initRegisteredExecutorsDB in ExternalShuffleService >

[jira] [Created] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

2018-12-05 Thread weixiuli (JIRA)
weixiuli created SPARK-26288: Summary: add initRegisteredExecutorsDB in ExternalShuffleService Key: SPARK-26288 URL: https://issues.apache.org/jira/browse/SPARK-26288 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

2018-12-05 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711014#comment-16711014 ] Takeshi Yamamuro commented on SPARK-26182: -- This is an expected behaviour and a known issue,

[jira] [Updated] (SPARK-26182) Cost increases when optimizing scalaUDF

2018-12-05 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26182: - Issue Type: Improvement (was: Bug) > Cost increases when optimizing scalaUDF >

[jira] [Commented] (SPARK-26287) Don't need to create an empty spill file when memory has no records

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711004#comment-16711004 ] Apache Spark commented on SPARK-26287: -- User 'wangjiaochun' has created a pull request for this

[jira] [Assigned] (SPARK-26287) Don't need to create an empty spill file when memory has no records

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26287: Assignee: (was: Apache Spark) > Don't need to create an empty spill file when memory

[jira] [Commented] (SPARK-26287) Don't need to create an empty spill file when memory has no records

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711001#comment-16711001 ] Apache Spark commented on SPARK-26287: -- User 'wangjiaochun' has created a pull request for this

[jira] [Assigned] (SPARK-26287) Don't need to create an empty spill file when memory has no records

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26287: Assignee: Apache Spark > Don't need to create an empty spill file when memory has no

[jira] [Created] (SPARK-26287) Don't need to create an empty spill file when memory has no records

2018-12-05 Thread wangjiaochun (JIRA)
wangjiaochun created SPARK-26287: Summary: Don't need to create an empty spill file when memory has no records Key: SPARK-26287 URL: https://issues.apache.org/jira/browse/SPARK-26287 Project: Spark

[jira] [Assigned] (SPARK-26286) Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26286: Assignee: (was: Apache Spark) > Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test >

[jira] [Assigned] (SPARK-26286) Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26286: Assignee: Apache Spark > Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test >

[jira] [Commented] (SPARK-26286) Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710997#comment-16710997 ] Apache Spark commented on SPARK-26286: -- User 'wangjiaochun' has created a pull request for this

[jira] [Created] (SPARK-26286) Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test

2018-12-05 Thread wangjiaochun (JIRA)
wangjiaochun created SPARK-26286: Summary: Add MAXIMUM_PAGE_SIZE_BYTES Exception unit test Key: SPARK-26286 URL: https://issues.apache.org/jira/browse/SPARK-26286 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26285: Assignee: (was: Apache Spark) > Add a metric source for accumulators (aka

[jira] [Assigned] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26285: Assignee: Apache Spark > Add a metric source for accumulators (aka AccumulatorSource) >

[jira] [Commented] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710975#comment-16710975 ] Apache Spark commented on SPARK-26285: -- User 'abellina' has created a pull request for this issue:

[jira] [Commented] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710974#comment-16710974 ] Apache Spark commented on SPARK-26285: -- User 'abellina' has created a pull request for this issue:

[jira] [Commented] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Alessandro Bellina (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710970#comment-16710970 ] Alessandro Bellina commented on SPARK-26285: I can't assign this issue, but I am putting up

[jira] [Created] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-05 Thread Alessandro Bellina (JIRA)
Alessandro Bellina created SPARK-26285: -- Summary: Add a metric source for accumulators (aka AccumulatorSource) Key: SPARK-26285 URL: https://issues.apache.org/jira/browse/SPARK-26285 Project:

[jira] [Commented] (SPARK-26261) Spark does not check completeness temporary file

2018-12-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710902#comment-16710902 ] Hyukjin Kwon commented on SPARK-26261: -- It would be easy to verify if the codes are posted together

[jira] [Commented] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors

2018-12-05 Thread Alan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710898#comment-16710898 ] Alan commented on SPARK-12312: -- I agree! Can we please get this implemented as soon as possible?  This

[jira] [Commented] (SPARK-26261) Spark does not check completeness temporary file

2018-12-05 Thread Jialin LIu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710883#comment-16710883 ] Jialin LIu commented on SPARK-26261: Our initial test is: We start a word count workflow including

[jira] [Resolved] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26275. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23236

[jira] [Assigned] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26275: Assignee: Hyukjin Kwon > Flaky test: pyspark.mllib.tests.test_streaming_algorithms >

[jira] [Reopened] (SPARK-25148) Executors launched with Spark on K8s client mode should prefix name with spark.app.name

2018-12-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-25148: > Executors launched with Spark on K8s client mode should prefix name with > spark.app.name

[jira] [Commented] (SPARK-25148) Executors launched with Spark on K8s client mode should prefix name with spark.app.name

2018-12-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710726#comment-16710726 ] Marcelo Vanzin commented on SPARK-25148: Actually there was a separate bug for the same issue.

[jira] [Resolved] (SPARK-25148) Executors launched with Spark on K8s client mode should prefix name with spark.app.name

2018-12-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25148. Resolution: Duplicate > Executors launched with Spark on K8s client mode should prefix

[jira] [Resolved] (SPARK-25148) Executors launched with Spark on K8s client mode should prefix name with spark.app.name

2018-12-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25148. Resolution: Cannot Reproduce This seems to work for me locally. Executor pods are

[jira] [Comment Edited] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710630#comment-16710630 ] shane knapp edited comment on SPARK-26282 at 12/5/18 9:02 PM: -- and the

[jira] [Commented] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710630#comment-16710630 ] shane knapp commented on SPARK-26282: - and the centos workers are updated: {noformat} [

[jira] [Commented] (SPARK-26281) Duration column of task table should be executor run time instead of real duration

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710622#comment-16710622 ] Apache Spark commented on SPARK-26281: -- User 'shahidki31' has created a pull request for this

[jira] [Updated] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26233: -- Fix Version/s: 2.4.1 2.3.3 2.2.3 > Incorrect decimal

[jira] [Updated] (SPARK-26284) Spark History server object vs file storage behavior difference

2018-12-05 Thread Damien Doucet-Girard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Doucet-Girard updated SPARK-26284: - Description: I am using the spark history server in order to view

[jira] [Created] (SPARK-26284) Spark History server object vs file storage behavior difference

2018-12-05 Thread Damien Doucet-Girard (JIRA)
Damien Doucet-Girard created SPARK-26284: Summary: Spark History server object vs file storage behavior difference Key: SPARK-26284 URL: https://issues.apache.org/jira/browse/SPARK-26284

[jira] [Updated] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26282: -- Summary: Update JVM to 8u191 on jenkins workers (was: update jvm on jenkins workers) >

[jira] [Resolved] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2018-12-05 Thread Pawan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pawan resolved SPARK-25919. --- Resolution: Fixed This was fixed by Hive in later versions of Jar which are not currently used by Spark

[jira] [Commented] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710565#comment-16710565 ] shane knapp commented on SPARK-26282: - ubuntu workers are done... {noformat} [

[jira] [Updated] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26282: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Update JVM to 8u191 on

[jira] [Commented] (SPARK-26282) Update JVM to 8u191 on jenkins workers

2018-12-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710558#comment-16710558 ] Dongjoon Hyun commented on SPARK-26282: --- +1, great! > Update JVM to 8u191 on jenkins workers >

[jira] [Commented] (SPARK-26282) update jvm on jenkins workers

2018-12-05 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710540#comment-16710540 ] shane knapp commented on SPARK-26282: - looks like 191 is the most current java8...  deploying that

[jira] [Closed] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2018-12-05 Thread Pawan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pawan closed SPARK-25919. - > Date value corrupts when tables are "ParquetHiveSerDe" formatted and target > table is Partitioned >

[jira] [Commented] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2018-12-05 Thread Pawan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710535#comment-16710535 ] Pawan commented on SPARK-25919: --- I just figured out why is this the issue. Its because of the hive-exec

[jira] [Commented] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710508#comment-16710508 ] Apache Spark commented on SPARK-26283: -- User 'shahidki31' has created a pull request for this

[jira] [Commented] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710506#comment-16710506 ] Apache Spark commented on SPARK-26283: -- User 'shahidki31' has created a pull request for this

[jira] [Assigned] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26283: Assignee: (was: Apache Spark) > When zstd compression enabled, Inprogress

[jira] [Assigned] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26283: Assignee: Apache Spark > When zstd compression enabled, Inprogress application in the

[jira] [Commented] (SPARK-26282) update jvm on jenkins workers

2018-12-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710451#comment-16710451 ] Sean Owen commented on SPARK-26282: --- Yes the latest Java 8 JDK (_192?) is best. That may well be one

[jira] [Created] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread ABHISHEK KUMAR GUPTA (JIRA)
ABHISHEK KUMAR GUPTA created SPARK-26283: Summary: When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running Key: SPARK-26283 URL:

[jira] [Commented] (SPARK-26283) When zstd compression enabled, Inprogress application in the history server appUI showing finished job as running

2018-12-05 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710444#comment-16710444 ] shahid commented on SPARK-26283: Thanks. I am working on it. > When zstd compression enabled,

[jira] [Created] (SPARK-26282) update jvm on jenkins workers

2018-12-05 Thread shane knapp (JIRA)
shane knapp created SPARK-26282: --- Summary: update jvm on jenkins workers Key: SPARK-26282 URL: https://issues.apache.org/jira/browse/SPARK-26282 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-26281) Duration column of task table should be executor run time instead of real duration

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26281: Assignee: Apache Spark > Duration column of task table should be executor run time

[jira] [Created] (SPARK-26281) Duration column of task table should be executor run time instead of real duration

2018-12-05 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-26281: -- Summary: Duration column of task table should be executor run time instead of real duration Key: SPARK-26281 URL: https://issues.apache.org/jira/browse/SPARK-26281

[jira] [Assigned] (SPARK-26281) Duration column of task table should be executor run time instead of real duration

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26281: Assignee: (was: Apache Spark) > Duration column of task table should be executor run

[jira] [Commented] (SPARK-26281) Duration column of task table should be executor run time instead of real duration

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710403#comment-16710403 ] Apache Spark commented on SPARK-26281: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-26278) V2 Streaming sources cannot be written to V1 sinks

2018-12-05 Thread Seth Fitzsimmons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710397#comment-16710397 ] Seth Fitzsimmons commented on SPARK-26278: -- I was thinking specifically of the SerializedOffset

[jira] [Commented] (SPARK-26222) Scan: track file listing time

2018-12-05 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710303#comment-16710303 ] Yuanjian Li commented on SPARK-26222: - Leave some thoughts for further discussion: * There's one

[jira] [Commented] (SPARK-26021) -0.0 and 0.0 not treated consistently, doesn't match Hive

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710266#comment-16710266 ] Apache Spark commented on SPARK-26021: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-26280) Spark will read entire CSV file even when limit is used

2018-12-05 Thread Amir Bar-Or (JIRA)
Amir Bar-Or created SPARK-26280: --- Summary: Spark will read entire CSV file even when limit is used Key: SPARK-26280 URL: https://issues.apache.org/jira/browse/SPARK-26280 Project: Spark Issue

[jira] [Updated] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26273: -- Priority: Minor (was: Major) > Add OneHotEncoderEstimator as alias to OneHotEncoder >

[jira] [Commented] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710219#comment-16710219 ] Apache Spark commented on SPARK-25132: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-26273. - Resolution: Won't Fix > Add OneHotEncoderEstimator as alias to OneHotEncoder >

[jira] [Commented] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710216#comment-16710216 ] Apache Spark commented on SPARK-25132: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Commented] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710192#comment-16710192 ] Liang-Chi Hsieh commented on SPARK-26273: - For now the idea collected from the PR is we don't

[jira] [Assigned] (SPARK-26279) Remove unused method in Logging

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26279: Assignee: (was: Apache Spark) > Remove unused method in Logging >

[jira] [Commented] (SPARK-2629) Improved state management for Spark Streaming (mapWithState)

2018-12-05 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710144#comment-16710144 ] Dan Dutrow commented on SPARK-2629: --- This PR should not reference SPARK-2629 > Improved state

[jira] [Assigned] (SPARK-26279) Remove unused method in Logging

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26279: Assignee: Apache Spark > Remove unused method in Logging >

[jira] [Commented] (SPARK-26279) Remove unused method in Logging

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710109#comment-16710109 ] Apache Spark commented on SPARK-26279: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Updated] (SPARK-26279) Remove unused method in Logging

2018-12-05 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26279: - Summary: Remove unused method in Logging (was: Remove unused methods in Logging) > Remove

[jira] [Created] (SPARK-26279) Remove unused methods in Logging

2018-12-05 Thread Chenxiao Mao (JIRA)
Chenxiao Mao created SPARK-26279: Summary: Remove unused methods in Logging Key: SPARK-26279 URL: https://issues.apache.org/jira/browse/SPARK-26279 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-24417) Build and Run Spark on JDK11

2018-12-05 Thread M. Le Bihan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710091#comment-16710091 ] M. Le Bihan edited comment on SPARK-24417 at 12/5/18 1:53 PM: -- Hello, 

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2018-12-05 Thread M. Le Bihan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710091#comment-16710091 ] M. Le Bihan commented on SPARK-24417: - Hello,  Unaware if the problem with the JDK 11, I used it

[jira] [Created] (SPARK-26278) V2 Streaming sources cannot be written to V1 sinks

2018-12-05 Thread Justin Polchlopek (JIRA)
Justin Polchlopek created SPARK-26278: - Summary: V2 Streaming sources cannot be written to V1 sinks Key: SPARK-26278 URL: https://issues.apache.org/jira/browse/SPARK-26278 Project: Spark

[jira] [Assigned] (SPARK-26277) WholeStageCodegen metrics should be tested with whole-stage codegen enabled

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26277: Assignee: (was: Apache Spark) > WholeStageCodegen metrics should be tested with

[jira] [Commented] (SPARK-26270) Having clause does not work with explode anymore

2018-12-05 Thread Olli Kuonanoja (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710026#comment-16710026 ] Olli Kuonanoja commented on SPARK-26270: Makes sense, thanks [~mgaido] > Having clause does not

[jira] [Commented] (SPARK-26277) WholeStageCodegen metrics should be tested with whole-stage codegen enabled

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710037#comment-16710037 ] Apache Spark commented on SPARK-26277: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Commented] (SPARK-26277) WholeStageCodegen metrics should be tested with whole-stage codegen enabled

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710039#comment-16710039 ] Apache Spark commented on SPARK-26277: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26277) WholeStageCodegen metrics should be tested with whole-stage codegen enabled

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26277: Assignee: Apache Spark > WholeStageCodegen metrics should be tested with whole-stage

[jira] [Created] (SPARK-26277) WholeStageCodegen metrics should be tested with whole-stage codegen enabled

2018-12-05 Thread Chenxiao Mao (JIRA)
Chenxiao Mao created SPARK-26277: Summary: WholeStageCodegen metrics should be tested with whole-stage codegen enabled Key: SPARK-26277 URL: https://issues.apache.org/jira/browse/SPARK-26277 Project:

[jira] [Resolved] (SPARK-26276) Broken link on download page

2018-12-05 Thread Sebb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebb resolved SPARK-26276. -- Resolution: Invalid Wrong project > Broken link on download page > > >

[jira] [Created] (SPARK-26276) Broken link on download page

2018-12-05 Thread Sebb (JIRA)
Sebb created SPARK-26276: Summary: Broken link on download page Key: SPARK-26276 URL: https://issues.apache.org/jira/browse/SPARK-26276 Project: Spark Issue Type: Bug Components: Deploy

[jira] [Commented] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709994#comment-16709994 ] Apache Spark commented on SPARK-26275: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26275: - Priority: Minor (was: Major) > Flaky test: pyspark.mllib.tests.test_streaming_algorithms >

[jira] [Assigned] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26275: Assignee: Apache Spark > Flaky test: pyspark.mllib.tests.test_streaming_algorithms >

[jira] [Assigned] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26275: Assignee: (was: Apache Spark) > Flaky test:

[jira] [Commented] (SPARK-26151) Return partial results for bad CSV records

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709991#comment-16709991 ] Apache Spark commented on SPARK-26151: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Created] (SPARK-26275) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2018-12-05 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26275: Summary: Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction Key: SPARK-26275 URL:

[jira] [Commented] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709970#comment-16709970 ] Apache Spark commented on SPARK-26233: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-26149) Read UTF8String from Parquet/ORC may be incorrect

2018-12-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709975#comment-16709975 ] Hyukjin Kwon commented on SPARK-26149: -- Thanks for details, [~yumwang] > Read UTF8String from

[jira] [Commented] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709968#comment-16709968 ] Apache Spark commented on SPARK-26233: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-26233) Incorrect decimal value with java beans and first/last/max... functions

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709964#comment-16709964 ] Apache Spark commented on SPARK-26233: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-26270) Having clause does not work with explode anymore

2018-12-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709953#comment-16709953 ] Marco Gaido commented on SPARK-26270: - This is caused by SPARK-25708. You can find more details on

[jira] [Resolved] (SPARK-26270) Having clause does not work with explode anymore

2018-12-05 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-26270. - Resolution: Invalid > Having clause does not work with explode anymore >

[jira] [Commented] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709886#comment-16709886 ] Apache Spark commented on SPARK-26273: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709882#comment-16709882 ] Apache Spark commented on SPARK-26273: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26273: Assignee: Apache Spark > Add OneHotEncoderEstimator as alias to OneHotEncoder >

[jira] [Assigned] (SPARK-26273) Add OneHotEncoderEstimator as alias to OneHotEncoder

2018-12-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26273: Assignee: (was: Apache Spark) > Add OneHotEncoderEstimator as alias to OneHotEncoder

[jira] [Created] (SPARK-26274) Download page must link to https://www.apache.org/dist/spark for current releases

2018-12-05 Thread Sebb (JIRA)
Sebb created SPARK-26274: Summary: Download page must link to https://www.apache.org/dist/spark for current releases Key: SPARK-26274 URL: https://issues.apache.org/jira/browse/SPARK-26274 Project: Spark

  1   2   >