[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415923#comment-17415923 ] Dongjoon Hyun commented on SPARK-33772: --- Could you paste your command line and the

[jira] [Updated] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36777: -- Parent: SPARK-33772 Issue Type: Sub-task (was: Improvement) > Move Java 17 on GitHub

[jira] [Commented] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415922#comment-17415922 ] Apache Spark commented on SPARK-36777: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36777: Assignee: Kousuke Saruta (was: Apache Spark) > Move Java 17 on GitHub Actions from EA to

[jira] [Assigned] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36777: Assignee: Apache Spark (was: Kousuke Saruta) > Move Java 17 on GitHub Actions from EA to

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415921#comment-17415921 ] L. C. Hsieh commented on SPARK-36673: - The schema after union looks incorrect. By th

[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415920#comment-17415920 ] Apache Spark commented on SPARK-36767: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36767: Assignee: (was: Apache Spark) > ArrayMin/ArrayMax/SortArray/ArraySort add comment an

[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415918#comment-17415918 ] Apache Spark commented on SPARK-36767: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36767: Assignee: Apache Spark > ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT >

[jira] [Updated] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36777: --- Issue Type: Improvement (was: Bug) > Move Java 17 on GitHub Actions from EA to LTS release

[jira] [Created] (SPARK-36777) Move Java 17 on GitHub Actions from EA to LTS release

2021-09-15 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-36777: -- Summary: Move Java 17 on GitHub Actions from EA to LTS release Key: SPARK-36777 URL: https://issues.apache.org/jira/browse/SPARK-36777 Project: Spark Iss

[jira] [Commented] (SPARK-36735) Adjust overhead of cached relation for DPP

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415911#comment-17415911 ] Apache Spark commented on SPARK-36735: -- User 'viirya' has created a pull request fo

[jira] [Resolved] (SPARK-36732) Upgrade ORC to 1.6.11

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36732. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33971 [https://

[jira] [Assigned] (SPARK-36732) Upgrade ORC to 1.6.11

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36732: - Assignee: Dongjoon Hyun > Upgrade ORC to 1.6.11 > - > >

[jira] [Created] (SPARK-36776) Partition filter of DataSourceV2ScanRelation can not push down when select none dataSchema from FileScan

2021-09-15 Thread suheng.cloud (Jira)
suheng.cloud created SPARK-36776: Summary: Partition filter of DataSourceV2ScanRelation can not push down when select none dataSchema from FileScan Key: SPARK-36776 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415891#comment-17415891 ] Mridul Muralidharan commented on SPARK-36772: - Thanks [~Gengliang.Wang] ! An

[jira] [Commented] (SPARK-36587) Migrate CreateNamespaceStatement to v2 command framework

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415884#comment-17415884 ] Apache Spark commented on SPARK-36587: -- User 'huaxingao' has created a pull request

[jira] [Assigned] (SPARK-36775) Add documentation for ANSI store assignment rules

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36775: Assignee: Apache Spark (was: Gengliang Wang) > Add documentation for ANSI store assignme

[jira] [Commented] (SPARK-36775) Add documentation for ANSI store assignment rules

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415883#comment-17415883 ] Apache Spark commented on SPARK-36775: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-36775) Add documentation for ANSI store assignment rules

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36775: Assignee: Gengliang Wang (was: Apache Spark) > Add documentation for ANSI store assignme

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Ye Zhou (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415879#comment-17415879 ] Ye Zhou commented on SPARK-36772: - I will work on this one and post PR ASAP. > Finalize

[jira] [Created] (SPARK-36775) Add documentation for ANSI store assignment rules

2021-09-15 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-36775: -- Summary: Add documentation for ANSI store assignment rules Key: SPARK-36775 URL: https://issues.apache.org/jira/browse/SPARK-36775 Project: Spark Issue T

[jira] [Assigned] (SPARK-36745) ExtractEquiJoinKeys should return the original predicates on join keys

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36745: --- Assignee: Yannis Sismanis > ExtractEquiJoinKeys should return the original predicates on jo

[jira] [Resolved] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36745. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33985 [https://gith

[jira] [Updated] (SPARK-36745) ExtractEquiJoinKeys should return the original predicates on join keys

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36745: Summary: ExtractEquiJoinKeys should return the original predicates on join keys (was: Cleanup pa

[jira] [Commented] (SPARK-36774) Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415867#comment-17415867 ] Apache Spark commented on SPARK-36774: -- User 'JoshRosen' has created a pull request

[jira] [Commented] (SPARK-36774) Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415869#comment-17415869 ] Apache Spark commented on SPARK-36774: -- User 'JoshRosen' has created a pull request

[jira] [Updated] (SPARK-36755) array_overlap should handle Double.NaN and Float.NaN

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36755: Fix Version/s: 3.0.4 > array_overlap should handle Double.NaN and Float.NaN >

[jira] [Resolved] (SPARK-36702) array_union should not produce duplicated result

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36702. - Fix Version/s: 3.0.4 3.1.3 3.2.0 Assignee: angersz

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415864#comment-17415864 ] Wenchen Fan commented on SPARK-36673: - This seems like a bug. [~viirya] what do you

[jira] [Created] (SPARK-36774) Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite

2021-09-15 Thread Josh Rosen (Jira)
Josh Rosen created SPARK-36774: -- Summary: Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite Key: SPARK-36774 URL: https://issues.apache.org/jira/browse/SPARK-36774 Project: Spark

[jira] [Commented] (SPARK-36678) Migrate SHOW TABLES to use V2 command by default

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415863#comment-17415863 ] Wenchen Fan commented on SPARK-36678: - Temp views don't belong to any catalog. I thi

[jira] [Commented] (SPARK-36773) Fix the uts to check the parquet compression

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415854#comment-17415854 ] Apache Spark commented on SPARK-36773: -- User 'BelodengKlaus' has created a pull req

[jira] [Commented] (SPARK-36773) Fix the uts to check the parquet compression

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415853#comment-17415853 ] Apache Spark commented on SPARK-36773: -- User 'BelodengKlaus' has created a pull req

[jira] [Assigned] (SPARK-36773) Fix the uts to check the parquet compression

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36773: Assignee: Apache Spark > Fix the uts to check the parquet compression > -

[jira] [Assigned] (SPARK-36773) Fix the uts to check the parquet compression

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36773: Assignee: (was: Apache Spark) > Fix the uts to check the parquet compression > --

[jira] [Updated] (SPARK-36773) Fix the uts to check the parquet compression

2021-09-15 Thread jingpan xiong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingpan xiong updated SPARK-36773: -- Summary: Fix the uts to check the parquet compression (was: Fix the uts to check the parquet

[jira] [Created] (SPARK-36773) Fix the uts to check the parquet codec

2021-09-15 Thread jingpan xiong (Jira)
jingpan xiong created SPARK-36773: - Summary: Fix the uts to check the parquet codec Key: SPARK-36773 URL: https://issues.apache.org/jira/browse/SPARK-36773 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36702) array_union should not produce duplicated result

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415841#comment-17415841 ] Apache Spark commented on SPARK-36702: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415810#comment-17415810 ] Gengliang Wang commented on SPARK-36772: [~mridulm80]Thanks for the ping. I will

[jira] [Comment Edited] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415803#comment-17415803 ] Mridul Muralidharan edited comment on SPARK-36772 at 9/15/21, 11:55 PM: --

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415803#comment-17415803 ] Mridul Muralidharan commented on SPARK-36772: - +CC [~Gengliang.Wang] > Fina

[jira] [Created] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

2021-09-15 Thread Mridul Muralidharan (Jira)
Mridul Muralidharan created SPARK-36772: --- Summary: FinalizeShuffleMerge fails with an exception due to attempt id not matching Key: SPARK-36772 URL: https://issues.apache.org/jira/browse/SPARK-36772

[jira] [Created] (SPARK-36771) Fix `pop` of Categorical Series

2021-09-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36771: Summary: Fix `pop` of Categorical Series Key: SPARK-36771 URL: https://issues.apache.org/jira/browse/SPARK-36771 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36771) Fix `pop` of Categorical Series

2021-09-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415796#comment-17415796 ] Xinrong Meng commented on SPARK-36771: -- I am working on that. > Fix `pop` of Categ

[jira] [Commented] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Guibin Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415787#comment-17415787 ] Guibin Zhang commented on SPARK-36770: -- [https://github.com/apache/spark/pull/34010

[jira] [Issue Comment Deleted] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Guibin Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guibin Zhang updated SPARK-36770: - Comment: was deleted (was: [https://github.com/apache/spark/pull/34010]  ) > Improve run-time

[jira] [Commented] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415786#comment-17415786 ] Apache Spark commented on SPARK-36770: -- User 'guibin' has created a pull request fo

[jira] [Assigned] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36770: Assignee: (was: Apache Spark) > Improve run-time performance for window function firs

[jira] [Assigned] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36770: Assignee: Apache Spark > Improve run-time performance for window function first and last

[jira] [Commented] (SPARK-34378) Support extra optional Avro fields in AvroSerializer

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415783#comment-17415783 ] Apache Spark commented on SPARK-34378: -- User 'xkrogen' has created a pull request f

[jira] [Assigned] (SPARK-34378) Support extra optional Avro fields in AvroSerializer

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34378: Assignee: (was: Apache Spark) > Support extra optional Avro fields in AvroSerializer

[jira] [Assigned] (SPARK-34378) Support extra optional Avro fields in AvroSerializer

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34378: Assignee: Apache Spark > Support extra optional Avro fields in AvroSerializer > -

[jira] [Resolved] (SPARK-36735) Adjust overhead of cached relation for DPP

2021-09-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36735. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33975 [https://gith

[jira] [Assigned] (SPARK-36735) Adjust overhead of cached relation for DPP

2021-09-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36735: --- Assignee: L. C. Hsieh > Adjust overhead of cached relation for DPP > --

[jira] [Resolved] (SPARK-36759) Upgrade Scala to 2.12.15

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36759. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33999 [https://

[jira] [Assigned] (SPARK-36759) Upgrade Scala to 2.12.15

2021-09-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36759: - Assignee: Dongjoon Hyun > Upgrade Scala to 2.12.15 > > >

[jira] [Comment Edited] (SPARK-31646) Remove unused registeredConnections counter from ShuffleMetrics

2021-09-15 Thread Yongjun Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415740#comment-17415740 ] Yongjun Zhang edited comment on SPARK-31646 at 9/15/21, 8:27 PM: -

[jira] [Comment Edited] (SPARK-31646) Remove unused registeredConnections counter from ShuffleMetrics

2021-09-15 Thread Yongjun Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415740#comment-17415740 ] Yongjun Zhang edited comment on SPARK-31646 at 9/15/21, 8:27 PM: -

[jira] [Commented] (SPARK-31646) Remove unused registeredConnections counter from ShuffleMetrics

2021-09-15 Thread Yongjun Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415740#comment-17415740 ] Yongjun Zhang commented on SPARK-31646: --- HI [~mauzhang], [~dongjoon], [~marek.simu

[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12

2021-09-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415725#comment-17415725 ] Takuya Ueshin commented on SPARK-34276: --- SPARK-36696 was fixed by upgrading parque

[jira] [Resolved] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36696. --- Fix Version/s: 3.2.0 Resolution: Fixed I confirmed the example file can be read after

[jira] [Comment Edited] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-15 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415703#comment-17415703 ] DB Tsai edited comment on SPARK-36696 at 9/15/21, 7:21 PM: --- Th

[jira] [Comment Edited] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-15 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415703#comment-17415703 ] DB Tsai edited comment on SPARK-36696 at 9/15/21, 7:20 PM: --- Th

[jira] [Commented] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-15 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415703#comment-17415703 ] DB Tsai commented on SPARK-36696: - This issue is addressed by https://issues.apache.org/

[jira] [Resolved] (SPARK-36726) Upgrade Parquet to 1.12.1

2021-09-15 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-36726. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33969 [https://github.com/a

[jira] [Assigned] (SPARK-36726) Upgrade Parquet to 1.12.1

2021-09-15 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-36726: --- Assignee: Chao Sun > Upgrade Parquet to 1.12.1 > - > > Key:

[jira] [Updated] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame

2021-09-15 Thread Guibin Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guibin Zhang updated SPARK-36770: - Summary: Improve run-time performance for window function first and last against UnboundedFollow

[jira] [Updated] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window frame

2021-09-15 Thread Guibin Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guibin Zhang updated SPARK-36770: - Description: *Context* The *UnboundedFollowingWindowFunctionFrame* has the time complexity of

[jira] [Created] (SPARK-36770) Improve run-time performance for window function first and last against UnboundedFollowing window frame

2021-09-15 Thread Guibin Zhang (Jira)
Guibin Zhang created SPARK-36770: Summary: Improve run-time performance for window function first and last against UnboundedFollowing window frame Key: SPARK-36770 URL: https://issues.apache.org/jira/browse/SPARK-

[jira] [Resolved] (SPARK-36722) Problems with update function in koalas - pyspark pandas.

2021-09-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36722. --- Fix Version/s: 3.2.0 Assignee: dgd_contributor Resolution: Fixed Issue resol

[jira] [Commented] (SPARK-36769) Improve `filter` of single-indexed DataFrame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415673#comment-17415673 ] Apache Spark commented on SPARK-36769: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-36769) Improve `filter` of single-indexed DataFrame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36769: Assignee: (was: Apache Spark) > Improve `filter` of single-indexed DataFrame > --

[jira] [Assigned] (SPARK-36769) Improve `filter` of single-indexed DataFrame

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36769: Assignee: Apache Spark > Improve `filter` of single-indexed DataFrame > -

[jira] [Created] (SPARK-36769) Improve `filter` of single-indexed DataFrame

2021-09-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36769: Summary: Improve `filter` of single-indexed DataFrame Key: SPARK-36769 URL: https://issues.apache.org/jira/browse/SPARK-36769 Project: Spark Issue Type: Sub-

[jira] [Commented] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415631#comment-17415631 ] Willi Raschkowski commented on SPARK-36768: --- In case you wonder why we care or

[jira] [Commented] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415630#comment-17415630 ] Willi Raschkowski commented on SPARK-36768: --- In the debugger I see [on this l

[jira] [Commented] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415621#comment-17415621 ] Willi Raschkowski commented on SPARK-36768: --- This also reproduces on master at

[jira] [Updated] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-36768: -- Description: Spark seems in some cases unable to resolve attributes that contain multi

[jira] [Updated] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-36768: -- Description: Spark seems in some cases unable to resolve attributes that contain multi

[jira] [Created] (SPARK-36768) Cannot resolve attribute with table reference

2021-09-15 Thread Willi Raschkowski (Jira)
Willi Raschkowski created SPARK-36768: - Summary: Cannot resolve attribute with table reference Key: SPARK-36768 URL: https://issues.apache.org/jira/browse/SPARK-36768 Project: Spark Issue

[jira] [Commented] (SPARK-36740) collection operators should handle duplicated NaN

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415608#comment-17415608 ] Apache Spark commented on SPARK-36740: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36740) collection operators should handle duplicated NaN

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36740: Assignee: Apache Spark > collection operators should handle duplicated NaN >

[jira] [Assigned] (SPARK-36740) collection operators should handle duplicated NaN

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36740: Assignee: (was: Apache Spark) > collection operators should handle duplicated NaN > -

[jira] [Commented] (SPARK-36740) collection operators should handle duplicated NaN

2021-09-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415607#comment-17415607 ] Apache Spark commented on SPARK-36740: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415588#comment-17415588 ] Shockang commented on SPARK-36767: -- [~angerszhuuu] OK > ArrayMin/ArrayMax/SortArray/A

[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415584#comment-17415584 ] angerszhu commented on SPARK-36767: --- [~Shockang] Sorry, I have write the pr. > Arra

[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415583#comment-17415583 ] Shockang commented on SPARK-36767: -- working on this. > ArrayMin/ArrayMax/SortArray/Ar

[jira] [Updated] (SPARK-36766) Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag

2021-09-15 Thread Sungwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sungwon updated SPARK-36766: Description: CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize the fileSyst

[jira] [Created] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT

2021-09-15 Thread angerszhu (Jira)
angerszhu created SPARK-36767: - Summary: ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT Key: SPARK-36767 URL: https://issues.apache.org/jira/browse/SPARK-36767 Project: Spark Issue Typ

[jira] [Issue Comment Deleted] (SPARK-30175) Eliminate warnings: part 5

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Comment: was deleted (was: I'm working on this.) > Eliminate warnings: part 5 > ---

[jira] [Created] (SPARK-36766) Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag

2021-09-15 Thread Sungwon (Jira)
Sungwon created SPARK-36766: --- Summary: Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag Key: SPARK-36766 URL: https://issues.apache.org/jira/browse/SPARK-36766 Project: Spark

[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchD

[jira] [Commented] (SPARK-30177) Eliminate warnings: part 7

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415564#comment-17415564 ] Shockang commented on SPARK-30177: -- After careful inspection, I found that except for t

[jira] [Issue Comment Deleted] (SPARK-30177) Eliminate warnings: part 7

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Comment: was deleted (was: After careful inspection, I found that except for the last one, everything e

[jira] [Issue Comment Deleted] (SPARK-30177) Eliminate warnings: part 7

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Comment: was deleted (was: I'm working on this.) > Eliminate warnings: part 7 > ---

[jira] [Resolved] (SPARK-36755) array_overlap should handle Double.NaN and Float.NaN

2021-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36755. - Fix Version/s: 3.1.3 3.2.0 Assignee: angerszhu Resolution: Fix

[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchD

[jira] [Updated] (SPARK-30177) Eliminate warnings: part 7

2021-09-15 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Description: -/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala- -Warning

  1   2   >