[jira] [Updated] (SPARK-36744) Support IO encryption for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-36744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-36744: Description: As a follow up to SPARK-36705, push-based shuffle is not compatible with IO encryption. We

[jira] [Updated] (SPARK-36744) Support IO encryption for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-36744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Minchu Yang updated SPARK-36744: Summary: Support IO encryption for push-based shuffle (was: support IO encryption for push-based

[jira] [Created] (SPARK-36744) support IO encryption for push-based shuffle

Minchu Yang created SPARK-36744: --- Summary: support IO encryption for push-based shuffle Key: SPARK-36744 URL: https://issues.apache.org/jira/browse/SPARK-36744 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-36031. -- Resolution: Done > Keep same behavior with pandas for operations of series with nan >

[jira] [Commented] (SPARK-36733) Perf issue in SchemaPruning when a struct has many fields

[ https://issues.apache.org/jira/browse/SPARK-36733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414438#comment-17414438 ] Apache Spark commented on SPARK-36733: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36733) Perf issue in SchemaPruning when a struct has many fields

[ https://issues.apache.org/jira/browse/SPARK-36733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36733: Assignee: (was: Apache Spark) > Perf issue in SchemaPruning when a struct has many

[jira] [Assigned] (SPARK-36733) Perf issue in SchemaPruning when a struct has many fields

[ https://issues.apache.org/jira/browse/SPARK-36733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36733: Assignee: Apache Spark > Perf issue in SchemaPruning when a struct has many fields >

[jira] [Commented] (SPARK-36733) Perf issue in SchemaPruning when a struct has many fields

[ https://issues.apache.org/jira/browse/SPARK-36733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414437#comment-17414437 ] Apache Spark commented on SPARK-36733: -- User 'sarutak' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-25075. -- Fix Version/s: 3.2.0 Assignee: Sean R. Owen Resolution: Fixed > Build and

[jira] [Updated] (SPARK-36712) Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile

[ https://issues.apache.org/jira/browse/SPARK-36712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-36712: - Priority: Minor (was: Major) > Published 2.13 POM lists `scala-parallel-collections` only in

[jira] [Assigned] (SPARK-36712) Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile

[ https://issues.apache.org/jira/browse/SPARK-36712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-36712: Assignee: Lukas Rytz > Published 2.13 POM lists `scala-parallel-collections` only in

[jira] [Resolved] (SPARK-36712) Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile

[ https://issues.apache.org/jira/browse/SPARK-36712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-36712. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33948

[jira] [Commented] (SPARK-33888) JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

[ https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414255#comment-17414255 ] Wenchen Fan commented on SPARK-33888: - This was reverted and replaced by 

[jira] [Reopened] (SPARK-33888) JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

[ https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-33888: - > JDBC SQL TIME type represents incorrectly as TimestampType, it should be > physical Int in

[jira] [Resolved] (SPARK-33888) JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

[ https://issues.apache.org/jira/browse/SPARK-33888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33888. - Assignee: (was: Duc Hoa Nguyen) Resolution: Invalid > JDBC SQL TIME type represents

[jira] [Resolved] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

[ https://issues.apache.org/jira/browse/SPARK-36736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36736. - Resolution: Fixed Issue resolved by pull request 33966

[jira] [Commented] (SPARK-36743) Backporting SPARK-36327 changes into Spark 2.4 version

[ https://issues.apache.org/jira/browse/SPARK-36743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414242#comment-17414242 ] Senthil Kumar commented on SPARK-36743: --- [~hyukjin.kwon], [~dongjoon] > Backporting SPARK-36327

[jira] [Updated] (SPARK-36743) Backporting SPARK-36327 changes into Spark 2.4 version

[ https://issues.apache.org/jira/browse/SPARK-36743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Senthil Kumar updated SPARK-36743: -- Summary: Backporting SPARK-36327 changes into Spark 2.4 version (was: Backporting changes

[jira] [Created] (SPARK-36743) Backporting changes into Spark 2.4 version

Senthil Kumar created SPARK-36743: - Summary: Backporting changes into Spark 2.4 version Key: SPARK-36743 URL: https://issues.apache.org/jira/browse/SPARK-36743 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32285) Add PySpark support for nested timestamps with arrow

[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414232#comment-17414232 ] pralabhkumar commented on SPARK-32285: -- [~hyukjin.kwon] just added the initial version for

[jira] [Assigned] (SPARK-32285) Add PySpark support for nested timestamps with arrow

[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32285: Assignee: (was: Apache Spark) > Add PySpark support for nested timestamps with arrow

[jira] [Commented] (SPARK-32285) Add PySpark support for nested timestamps with arrow

[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414215#comment-17414215 ] Apache Spark commented on SPARK-32285: -- User 'pralabhkumar' has created a pull request for this

[jira] [Commented] (SPARK-32285) Add PySpark support for nested timestamps with arrow

[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414216#comment-17414216 ] Apache Spark commented on SPARK-32285: -- User 'pralabhkumar' has created a pull request for this

[jira] [Assigned] (SPARK-32285) Add PySpark support for nested timestamps with arrow

[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32285: Assignee: Apache Spark > Add PySpark support for nested timestamps with arrow >

[jira] [Resolved] (SPARK-36724) Support timestamp_ntz as a type of time column for SessionWindow

[ https://issues.apache.org/jira/browse/SPARK-36724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36724. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33965

[jira] [Commented] (SPARK-36742) Fix ps.to_datetime with plurals of keys like years, months, days

[ https://issues.apache.org/jira/browse/SPARK-36742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414194#comment-17414194 ] dgd_contributor commented on SPARK-36742: - working on this   > Fix ps.to_datetime with plurals

[jira] [Created] (SPARK-36742) Fix ps.to_datetime with plurals of keys like years, months, days

dgd_contributor created SPARK-36742: --- Summary: Fix ps.to_datetime with plurals of keys like years, months, days Key: SPARK-36742 URL: https://issues.apache.org/jira/browse/SPARK-36742 Project:

[jira] [Commented] (SPARK-36728) Can't create datetime object from anything other then year column Pyspark - koalas

[ https://issues.apache.org/jira/browse/SPARK-36728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414188#comment-17414188 ] Hyukjin Kwon commented on SPARK-36728: -- We should actually revisit whole datetime operations. I

[jira] [Resolved] (SPARK-36738) Wrong description on Cot API

[ https://issues.apache.org/jira/browse/SPARK-36738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36738. -- Fix Version/s: 3.0.4 3.1.3 3.2.0 Assignee: Yuto

[jira] [Created] (SPARK-36741) array_distinct should not return duplicated NaN

angerszhu created SPARK-36741: - Summary: array_distinct should not return duplicated NaN Key: SPARK-36741 URL: https://issues.apache.org/jira/browse/SPARK-36741 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36728) Can't create datetime object from anything other then year column Pyspark - koalas

[ https://issues.apache.org/jira/browse/SPARK-36728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414179#comment-17414179 ] dgd_contributor commented on SPARK-36728: - thanks, pandas API on spark does not work with

[jira] [Updated] (SPARK-36702) array_union should not produce duplicated result

[ https://issues.apache.org/jira/browse/SPARK-36702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-36702: -- Parent: SPARK-36740 Issue Type: Sub-task (was: Bug) > array_union should not produce

[jira] [Created] (SPARK-36740) collection operators should handle duplicated NaN

angerszhu created SPARK-36740: - Summary: collection operators should handle duplicated NaN Key: SPARK-36740 URL: https://issues.apache.org/jira/browse/SPARK-36740 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-36728) Can't create datetime object from anything other then year column Pyspark - koalas

[ https://issues.apache.org/jira/browse/SPARK-36728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-36728: Attachment: pyspark_date2.txt > Can't create datetime object from anything other then

[jira] [Commented] (SPARK-36728) Can't create datetime object from anything other then year column Pyspark - koalas

[ https://issues.apache.org/jira/browse/SPARK-36728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414160#comment-17414160 ] Bjørn Jørgensen commented on SPARK-36728: - In

[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414159#comment-17414159 ] H. Vetinari commented on SPARK-33772: - If at all possible (and the currently open changes looks

[jira] [Comment Edited] (SPARK-33772) Build and Run Spark on Java 17

[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414159#comment-17414159 ] H. Vetinari edited comment on SPARK-33772 at 9/13/21, 11:50 AM: If at

[jira] [Commented] (SPARK-34112) Upgrade ORC

[ https://issues.apache.org/jira/browse/SPARK-34112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414158#comment-17414158 ] H. Vetinari commented on SPARK-34112: - ORC 1.7rc0 has been released which contains ORC-716. >

[jira] [Assigned] (SPARK-36739) Add Apache license header to makefiles of python documents

[ https://issues.apache.org/jira/browse/SPARK-36739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36739: Assignee: Apache Spark > Add Apache license header to makefiles of python documents >

[jira] [Assigned] (SPARK-36739) Add Apache license header to makefiles of python documents

[ https://issues.apache.org/jira/browse/SPARK-36739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36739: Assignee: (was: Apache Spark) > Add Apache license header to makefiles of python

[jira] [Commented] (SPARK-36739) Add Apache license header to makefiles of python documents

[ https://issues.apache.org/jira/browse/SPARK-36739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414136#comment-17414136 ] Apache Spark commented on SPARK-36739: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Created] (SPARK-36739) Add Apache license header to makefiles of python documents

Leona Yoda created SPARK-36739: -- Summary: Add Apache license header to makefiles of python documents Key: SPARK-36739 URL: https://issues.apache.org/jira/browse/SPARK-36739 Project: Spark Issue

[jira] [Commented] (SPARK-36738) Wrong description on Cot API

[ https://issues.apache.org/jira/browse/SPARK-36738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414095#comment-17414095 ] Apache Spark commented on SPARK-36738: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36738) Wrong description on Cot API

[ https://issues.apache.org/jira/browse/SPARK-36738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36738: Assignee: Apache Spark > Wrong description on Cot API > > >

[jira] [Assigned] (SPARK-36738) Wrong description on Cot API

[ https://issues.apache.org/jira/browse/SPARK-36738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36738: Assignee: (was: Apache Spark) > Wrong description on Cot API >

[jira] [Created] (SPARK-36738) Wrong description on Cot API

Yuto Akutsu created SPARK-36738: --- Summary: Wrong description on Cot API Key: SPARK-36738 URL: https://issues.apache.org/jira/browse/SPARK-36738 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33832: --- Assignee: XiDuo You > Add an option in AQE to mitigate skew even if it causes an new

[jira] [Resolved] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33832. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 32816

[jira] [Issue Comment Deleted] (SPARK-33772) Build and Run Spark on Java 17

[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-33772: - Comment: was deleted (was: Just in case, tomorrow (2021/09/14) is GA of JDK 17, see

[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414050#comment-17414050 ] Max Gekk commented on SPARK-33772: -- Just in case, tomorrow (2021/09/14) is GA of JDK 17, see

[jira] [Commented] (SPARK-36456) Clean up the depredation usage of o.a.c.io.IOUtils.closeQuietly

[ https://issues.apache.org/jira/browse/SPARK-36456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414022#comment-17414022 ] Apache Spark commented on SPARK-36456: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36737) Upgrade commons-io to 2.11.0 and revert change of SPARK-36456

[ https://issues.apache.org/jira/browse/SPARK-36737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414021#comment-17414021 ] Apache Spark commented on SPARK-36737: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-36737) Upgrade commons-io to 2.11.0 and revert change of SPARK-36456

[ https://issues.apache.org/jira/browse/SPARK-36737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36737: Assignee: Apache Spark > Upgrade commons-io to 2.11.0 and revert change of SPARK-36456 >

[jira] [Assigned] (SPARK-36737) Upgrade commons-io to 2.11.0 and revert change of SPARK-36456

[ https://issues.apache.org/jira/browse/SPARK-36737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36737: Assignee: (was: Apache Spark) > Upgrade commons-io to 2.11.0 and revert change of

[jira] [Updated] (SPARK-36737) Upgrade commons-io to 2.11.0 and revert change of SPARK-36456

[ https://issues.apache.org/jira/browse/SPARK-36737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-36737: - Description: {{`JavaUtils.closeQuietly`}} looks to be slightly different from

[jira] [Created] (SPARK-36737) Upgrade commons-io to 2.11.0 and revert change of SPARK-36456

Yang Jie created SPARK-36737: Summary: Upgrade commons-io to 2.11.0 and revert change of SPARK-36456 Key: SPARK-36737 URL: https://issues.apache.org/jira/browse/SPARK-36737 Project: Spark Issue

[jira] [Updated] (SPARK-33275) ANSI mode: runtime errors instead of returning null on invalid inputs

[ https://issues.apache.org/jira/browse/SPARK-33275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-33275: --- Summary: ANSI mode: runtime errors instead of returning null on invalid inputs (was: ANSI

[jira] [Assigned] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

[ https://issues.apache.org/jira/browse/SPARK-36736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36736: Assignee: Apache Spark (was: Max Gekk) > Support ILIKE (ALL | ANY | SOME) - case

[jira] [Commented] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

[ https://issues.apache.org/jira/browse/SPARK-36736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413974#comment-17413974 ] Apache Spark commented on SPARK-36736: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

[ https://issues.apache.org/jira/browse/SPARK-36736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36736: Assignee: Max Gekk (was: Apache Spark) > Support ILIKE (ALL | ANY | SOME) - case

[jira] [Commented] (SPARK-36714) bugs in MIniLSH

[ https://issues.apache.org/jira/browse/SPARK-36714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413953#comment-17413953 ] shengzhang commented on SPARK-36714: [~hyukjin.kwon]     currently spark 3.x is not  supported  in

[jira] [Updated] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

[ https://issues.apache.org/jira/browse/SPARK-36736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-36736: - Description: Add ILIKE (case insensitive LIKE) to improve user experience with Spark SQL. Other DBMSs

[jira] [Created] (SPARK-36736) Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE

Max Gekk created SPARK-36736: Summary: Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE Key: SPARK-36736 URL: https://issues.apache.org/jira/browse/SPARK-36736 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36683) Support secant and cosecant

[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuto Akutsu updated SPARK-36683: Summary: Support secant and cosecant (was: Support secent and cosecent) > Support secant and

[jira] [Assigned] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36705: Assignee: Apache Spark > Disable push based shuffle when IO encryption is enabled or

[jira] [Assigned] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36705: Assignee: (was: Apache Spark) > Disable push based shuffle when IO encryption is

[jira] [Commented] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413929#comment-17413929 ] Apache Spark commented on SPARK-36705: -- User 'rmcyang' has created a pull request for this issue:

<    1   2