[jira] [Created] (SPARK-31079) Add RuleExecutor metrics in Explain Formatted

2020-03-06 Thread Xin Wu (Jira)
Xin Wu created SPARK-31079: -- Summary: Add RuleExecutor metrics in Explain Formatted Key: SPARK-31079 URL: https://issues.apache.org/jira/browse/SPARK-31079 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30541) Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite

2020-03-06 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053762#comment-17053762 ] Gabor Somogyi commented on SPARK-30541: --- Going to check them next week... > Flaky test:

[jira] [Updated] (SPARK-30541) Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite

2020-03-06 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-30541: -- Attachment: unit-tests_OK.log unit-tests_NOK.log

[jira] [Commented] (SPARK-30541) Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite

2020-03-06 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053760#comment-17053760 ] Gabor Somogyi commented on SPARK-30541: --- As I see there are 2 problems: * Kafka broker is not

[jira] [Updated] (SPARK-31078) outputOrdering should handle aliases correctly

2020-03-06 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terry Kim updated SPARK-31078: -- Description: Currently, `outputOrdering` doesn't respect aliases. Thus, the following would produce

[jira] [Created] (SPARK-31078) outputOrdering should handle aliases correctly

2020-03-06 Thread Terry Kim (Jira)
Terry Kim created SPARK-31078: - Summary: outputOrdering should handle aliases correctly Key: SPARK-31078 URL: https://issues.apache.org/jira/browse/SPARK-31078 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-31064) New Parquet Predicate Filter APIs with multi-part Identifier Support

2020-03-06 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-31064: --- Assignee: DB Tsai > New Parquet Predicate Filter APIs with multi-part Identifier Support >

[jira] [Resolved] (SPARK-31064) New Parquet Predicate Filter APIs with multi-part Identifier Support

2020-03-06 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-31064. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27824

[jira] [Created] (SPARK-31077) Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel

2020-03-06 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-31077: -- Summary: Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel Key: SPARK-31077 URL: https://issues.apache.org/jira/browse/SPARK-31077 Project: Spark

[jira] [Created] (SPARK-31076) Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time

2020-03-06 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31076: -- Summary: Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time Key: SPARK-31076 URL: https://issues.apache.org/jira/browse/SPARK-31076 Project:

[jira] [Updated] (SPARK-31072) Default to ParquetOutputCommitter even after configuring s3a committer as "partitioned"

2020-03-06 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31072: - Summary: Default to ParquetOutputCommitter even after configuring s3a committer

[jira] [Updated] (SPARK-31072) Default to ParquetOutputCommitter even after configuring committer as "partitioned"

2020-03-06 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31072: - Summary: Default to ParquetOutputCommitter even after configuring committer as

[jira] [Commented] (SPARK-30961) Arrow enabled: to_pandas with date column fails

2020-03-06 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053717#comment-17053717 ] Bryan Cutler commented on SPARK-30961: -- Just to be clear, this is only an issue with Spark 2.4.x.

[jira] [Resolved] (SPARK-30961) Arrow enabled: to_pandas with date column fails

2020-03-06 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-30961. -- Resolution: Won't Fix Thanks [~KevinAppel] and [~nicornk] for the info, I'll go ahead and

[jira] [Commented] (SPARK-29367) pandas udf not working with latest pyarrow release (0.15.0)

2020-03-06 Thread Alexander Tronchin-James (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053715#comment-17053715 ] Alexander Tronchin-James commented on SPARK-29367: -- The fix suggested above to add 

[jira] [Created] (SPARK-31075) Add documentation for ALTER TABLE ... ADD PARTITION

2020-03-06 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31075: Summary: Add documentation for ALTER TABLE ... ADD PARTITION Key: SPARK-31075 URL: https://issues.apache.org/jira/browse/SPARK-31075 Project: Spark

[jira] [Updated] (SPARK-24640) size(null) returns null

2020-03-06 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24640: -- Labels: (was: api bulk-closed) > size(null) returns null > > >

[jira] [Updated] (SPARK-31041) Show Maven errors from within make-distribution.sh

2020-03-06 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Description: This works: {code:java} ./dev/make-distribution.sh \ --pip \

[jira] [Updated] (SPARK-31041) Show Maven errors from within make-distribution.sh

2020-03-06 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Summary: Show Maven errors from within make-distribution.sh (was: Make arguments to

[jira] [Created] (SPARK-31074) Avro serializer should not fail when a nullable Spark field is written to a non-null Avro column

2020-03-06 Thread Kyrill Alyoshin (Jira)
Kyrill Alyoshin created SPARK-31074: --- Summary: Avro serializer should not fail when a nullable Spark field is written to a non-null Avro column Key: SPARK-31074 URL:

[jira] [Updated] (SPARK-31073) Add "shuffle write time" to task metrics summary in StagePage.

2020-03-06 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-31073: --- Summary: Add "shuffle write time" to task metrics summary in StagePage. (was: Add shuffle

[jira] [Created] (SPARK-31073) Add shuffle write time to task metrics summary in StagePage.

2020-03-06 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-31073: -- Summary: Add shuffle write time to task metrics summary in StagePage. Key: SPARK-31073 URL: https://issues.apache.org/jira/browse/SPARK-31073 Project: Spark

[jira] [Comment Edited] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

2020-03-06 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053543#comment-17053543 ] Felix Kizhakkel Jose edited comment on SPARK-31072 at 3/6/20, 3:48 PM:

[jira] [Commented] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

2020-03-06 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053543#comment-17053543 ] Felix Kizhakkel Jose commented on SPARK-31072: -- [~steve_l],  I have seen some issues you

[jira] [Resolved] (SPARK-31054) Turn on deprecation in Scala REPL/spark-shell by default

2020-03-06 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31054. -- Resolution: Won't Fix > Turn on deprecation in Scala REPL/spark-shell by default >

[jira] [Created] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

2020-03-06 Thread Felix Kizhakkel Jose (Jira)
Felix Kizhakkel Jose created SPARK-31072: Summary: Default to ParquetOutputCommitter even after configuring setting committer as "partitioned" Key: SPARK-31072 URL:

[jira] [Updated] (SPARK-31071) Spark Encoders.bean() should allow marking non-null fields in its Spark schema

2020-03-06 Thread Kyrill Alyoshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyrill Alyoshin updated SPARK-31071: Summary: Spark Encoders.bean() should allow marking non-null fields in its Spark schema

[jira] [Created] (SPARK-31071) Spark Encoders.bean() should allow setting non-null fields in its Spark schema

2020-03-06 Thread Kyrill Alyoshin (Jira)
Kyrill Alyoshin created SPARK-31071: --- Summary: Spark Encoders.bean() should allow setting non-null fields in its Spark schema Key: SPARK-31071 URL: https://issues.apache.org/jira/browse/SPARK-31071

[jira] [Created] (SPARK-31070) make skew join split skewed partitions more evenly

2020-03-06 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-31070: --- Summary: make skew join split skewed partitions more evenly Key: SPARK-31070 URL: https://issues.apache.org/jira/browse/SPARK-31070 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-30899) CreateArray/CreateMap's data type should not depend on SQLConf.get

2020-03-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30899: --- Assignee: Rakesh Raushan > CreateArray/CreateMap's data type should not depend on

[jira] [Resolved] (SPARK-30899) CreateArray/CreateMap's data type should not depend on SQLConf.get

2020-03-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30899. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27657

[jira] [Created] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

2020-03-06 Thread Xiaoju Wu (Jira)
Xiaoju Wu created SPARK-31069: - Summary: high cpu caused by chunksBeingTransferred in external shuffle service Key: SPARK-31069 URL: https://issues.apache.org/jira/browse/SPARK-31069 Project: Spark

[jira] [Updated] (SPARK-31011) Failed to register signal handler for PWR

2020-03-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31011: - Affects Version/s: (was: 3.0.0) 3.1.0 > Failed to register signal

[jira] [Updated] (SPARK-31030) Backward Compatibility for Parsing and Formatting Datetime

2020-03-06 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-31030: Summary: Backward Compatibility for Parsing and Formatting Datetime (was: Backward Compatibility

[jira] [Updated] (SPARK-31030) Backward Compatibility for Parsing Datetime

2020-03-06 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-31030: Description: *Background* In Spark version 2.4 and earlier, datetime parsing, formatting and

[jira] [Resolved] (SPARK-30279) Support 32 or more grouping attributes for GROUPING_ID

2020-03-06 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-30279. -- Fix Version/s: 3.1.0 Assignee: Takeshi Yamamuro Resolution: Fixed