[jira] [Commented] (SPARK-40559) Add applyInArrow to pyspark.sql.GroupedData

2022-11-01 Thread Enrico Minack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627483#comment-17627483 ] Enrico Minack commented on SPARK-40559: --- That would require users to re-implement

[jira] [Assigned] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40991: Assignee: Dongjoon Hyun > Update cloudpickle to v2.2.0 > > >

[jira] [Resolved] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40991. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38474 [https://gi

[jira] [Assigned] (SPARK-40968) Fix some wrong/misleading comments in DAGSchedulerSuite

2022-11-01 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40968: --- Assignee: Jiexing Li > Fix some wrong/misleading comments in DAGSchedulerSu

[jira] [Resolved] (SPARK-40968) Fix some wrong/misleading comments in DAGSchedulerSuite

2022-11-01 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40968. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 383

[jira] [Assigned] (SPARK-40748) Migrate type check failures of conditions onto error classes

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40748: Assignee: BingKun Pan > Migrate type check failures of conditions onto error classes > --

[jira] [Resolved] (SPARK-40748) Migrate type check failures of conditions onto error classes

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40748. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38438 [https://github.com

[jira] [Assigned] (SPARK-40993) Migrate markdown style README to python/docs/development/testing.rst

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40993: Assignee: Apache Spark > Migrate markdown style README to python/docs/development/testing

[jira] [Assigned] (SPARK-40993) Migrate markdown style README to python/docs/development/testing.rst

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40993: Assignee: (was: Apache Spark) > Migrate markdown style README to python/docs/developm

[jira] [Commented] (SPARK-40993) Migrate markdown style README to python/docs/development/testing.rst

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627423#comment-17627423 ] Apache Spark commented on SPARK-40993: -- User 'amaliujia' has created a pull request

[jira] [Created] (SPARK-40993) Migrate markdown style README to python/docs/development/testing.rst

2022-11-01 Thread Rui Wang (Jira)
Rui Wang created SPARK-40993: Summary: Migrate markdown style README to python/docs/development/testing.rst Key: SPARK-40993 URL: https://issues.apache.org/jira/browse/SPARK-40993 Project: Spark

[jira] [Comment Edited] (SPARK-35217) com.google.protobuf.Parser.parseFrom() method Can't use in spark

2022-11-01 Thread ShuDaoNan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333086#comment-17333086 ] ShuDaoNan edited comment on SPARK-35217 at 11/2/22 2:23 AM:

[jira] [Commented] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627415#comment-17627415 ] Douglas Moore commented on SPARK-40990: --- How wide of ndarray before we break somet

[jira] [Updated] (SPARK-40988) Spark3 partition column value is not validated with user provided schema.

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40988: - Fix Version/s: (was: 3.4.0) > Spark3 partition column value is not validated with user provi

[jira] [Updated] (SPARK-40988) Spark3 partition column value is not validated with user provided schema.

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40988: - Target Version/s: (was: 3.4.0) > Spark3 partition column value is not validated with user prov

[jira] [Assigned] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40976: Assignee: (was: Apache Spark) > Upgrade sbt to 1.7.3 > > >

[jira] [Assigned] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40976: Assignee: Apache Spark > Upgrade sbt to 1.7.3 > > >

[jira] [Reopened] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-40976: -- Assignee: (was: Yang Jie) Reverted in https://github.com/apache/spark/commit/a0ab1720ab

[jira] [Updated] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40976: - Fix Version/s: (was: 3.4.0) > Upgrade sbt to 1.7.3 > > >

[jira] [Assigned] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40990: Assignee: Xinrong Meng > DataFrame creation from 2d NumPy array with arbitrary columns >

[jira] [Resolved] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40990. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38473 [https://gi

[jira] [Resolved] (SPARK-40930) Support Collect() in Python client

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40930. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38409 [https://gi

[jira] [Assigned] (SPARK-40930) Support Collect() in Python client

2022-11-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40930: Assignee: Rui Wang > Support Collect() in Python client > ---

[jira] [Commented] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627394#comment-17627394 ] Apache Spark commented on SPARK-40976: -- User 'linhongliu-db' has created a pull req

[jira] [Commented] (SPARK-40976) Upgrade sbt to 1.7.3

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627393#comment-17627393 ] Apache Spark commented on SPARK-40976: -- User 'linhongliu-db' has created a pull req

[jira] [Commented] (SPARK-40992) Support toDF(columnNames) in Connect DSL

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627376#comment-17627376 ] Apache Spark commented on SPARK-40992: -- User 'amaliujia' has created a pull request

[jira] [Assigned] (SPARK-40992) Support toDF(columnNames) in Connect DSL

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40992: Assignee: Apache Spark > Support toDF(columnNames) in Connect DSL > -

[jira] [Commented] (SPARK-40992) Support toDF(columnNames) in Connect DSL

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627375#comment-17627375 ] Apache Spark commented on SPARK-40992: -- User 'amaliujia' has created a pull request

[jira] [Assigned] (SPARK-40992) Support toDF(columnNames) in Connect DSL

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40992: Assignee: (was: Apache Spark) > Support toDF(columnNames) in Connect DSL > --

[jira] [Assigned] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40991: Assignee: (was: Apache Spark) > Update cloudpickle to v2.2.0 > --

[jira] [Assigned] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40991: Assignee: Apache Spark > Update cloudpickle to v2.2.0 > > >

[jira] [Commented] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627370#comment-17627370 ] Apache Spark commented on SPARK-40991: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-40992) Support toDF(columnNames) in Connect DSL

2022-11-01 Thread Rui Wang (Jira)
Rui Wang created SPARK-40992: Summary: Support toDF(columnNames) in Connect DSL Key: SPARK-40992 URL: https://issues.apache.org/jira/browse/SPARK-40992 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-40991) Update cloudpickle to v2.2.0

2022-11-01 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-40991: - Summary: Update cloudpickle to v2.2.0 Key: SPARK-40991 URL: https://issues.apache.org/jira/browse/SPARK-40991 Project: Spark Issue Type: Improvement

[jira] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows

2022-11-01 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35563 ] Jungtaek Lim deleted comment on SPARK-35563: -- was (Author: JIRAUSER294516): Alternately, simply perform the int overflow check. Personally, I don't think it's a problem that Spark doesn't s

[jira] [Commented] (SPARK-39405) NumPy input support in PySpark SQL

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627359#comment-17627359 ] Xinrong Meng commented on SPARK-39405: -- Thanks [~douglas.mo...@databricks.com] , yo

[jira] [Assigned] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40990: Assignee: Apache Spark > DataFrame creation from 2d NumPy array with arbitrary columns >

[jira] [Commented] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627356#comment-17627356 ] Apache Spark commented on SPARK-40990: -- User 'xinrong-meng' has created a pull requ

[jira] [Assigned] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40990: Assignee: (was: Apache Spark) > DataFrame creation from 2d NumPy array with arbitrary

[jira] [Commented] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627357#comment-17627357 ] Xinrong Meng commented on SPARK-37697: -- Thanks [~douglas.mo...@databricks.com] , yo

[jira] [Updated] (SPARK-40990) DataFrame creation from 2d NumPy array with arbitrary columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Summary: DataFrame creation from 2d NumPy array with arbitrary columns (was: Complete support f

[jira] [Updated] (SPARK-40990) Complete support for DataFrame creation from 2d NumPy array

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Description: Currently, DataFrame creation from 2d ndarray works only with 2 columns. We should

[jira] [Updated] (SPARK-40990) Complete support for DataFrame creation from 2d NumPy array

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Summary: Complete support for DataFrame creation from 2d NumPy array (was: Support DataFrame cr

[jira] [Updated] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Description: Currently, DataFrame creation from ndarray works only with <= 2 columns. We should

[jira] [Updated] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Description: Currently, DataFrame creation from ndarray works only with <= 2 columns. We should

[jira] [Updated] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40990: - Description: Currently, DataFrame creation from ndarray works only with <= 2 columns. We should

[jira] [Commented] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627337#comment-17627337 ] Xinrong Meng commented on SPARK-40990: -- I am working on that. > Support DataFrame

[jira] [Commented] (SPARK-39405) NumPy input support in PySpark SQL

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627336#comment-17627336 ] Xinrong Meng commented on SPARK-39405: -- Hi [~douglas.mo...@databricks.com] thanks f

[jira] [Assigned] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-40990: Assignee: (was: Xinrong Meng) > Support DataFrame creation from ndarray with >2 colum

[jira] [Created] (SPARK-40990) Support DataFrame creation from ndarray with >2 columns

2022-11-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-40990: Summary: Support DataFrame creation from ndarray with >2 columns Key: SPARK-40990 URL: https://issues.apache.org/jira/browse/SPARK-40990 Project: Spark Issue

[jira] [Commented] (SPARK-40588) Sorting issue with partitioned-writing and AQE turned on

2022-11-01 Thread Swetha Baskaran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627324#comment-17627324 ] Swetha Baskaran commented on SPARK-40588: - I have also tested the changes agains

[jira] [Commented] (SPARK-40989) Improve `session.sql` testing coverage in Python client

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627308#comment-17627308 ] Apache Spark commented on SPARK-40989: -- User 'amaliujia' has created a pull request

[jira] [Assigned] (SPARK-40989) Improve `session.sql` testing coverage in Python client

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40989: Assignee: Apache Spark > Improve `session.sql` testing coverage in Python client > --

[jira] [Assigned] (SPARK-40989) Improve `session.sql` testing coverage in Python client

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40989: Assignee: (was: Apache Spark) > Improve `session.sql` testing coverage in Python clie

[jira] [Commented] (SPARK-40989) Improve `session.sql` testing coverage in Python client

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627306#comment-17627306 ] Apache Spark commented on SPARK-40989: -- User 'amaliujia' has created a pull request

[jira] [Created] (SPARK-40989) Improve `session.sql` testing coverage in Python client

2022-11-01 Thread Rui Wang (Jira)
Rui Wang created SPARK-40989: Summary: Improve `session.sql` testing coverage in Python client Key: SPARK-40989 URL: https://issues.apache.org/jira/browse/SPARK-40989 Project: Spark Issue Type: S

[jira] [Commented] (SPARK-40883) Support Range in Connect proto

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627303#comment-17627303 ] Apache Spark commented on SPARK-40883: -- User 'amaliujia' has created a pull request

[jira] [Commented] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627257#comment-17627257 ] Douglas Moore commented on SPARK-37697: --- This works:  {code:java} spark.createData

[jira] [Commented] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627254#comment-17627254 ] Douglas Moore commented on SPARK-37697: --- This fails with the same value error: {co

[jira] [Comment Edited] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627251#comment-17627251 ] Douglas Moore edited comment on SPARK-37697 at 11/1/22 5:04 PM: --

[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows

2022-11-01 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627252#comment-17627252 ] Vivek Garg commented on SPARK-35563: Alternately, simply perform the int overflow ch

[jira] [Comment Edited] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627251#comment-17627251 ] Douglas Moore edited comment on SPARK-37697 at 11/1/22 4:59 PM: --

[jira] [Commented] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627251#comment-17627251 ] Douglas Moore commented on SPARK-37697: --- Sorry, I don't know what's going on... T

[jira] [Assigned] (SPARK-40921) Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO command

2022-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40921: --- Assignee: Johan Lasperas > Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO command > --

[jira] [Resolved] (SPARK-40921) Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO command

2022-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40921. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38400 [https://gith

[jira] [Commented] (SPARK-40988) Spark3 partition column value is not validated with user provided schema.

2022-11-01 Thread Ranga Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627197#comment-17627197 ] Ranga Reddy commented on SPARK-40988: - I will work on this issue. > Spark3 partitio

[jira] [Created] (SPARK-40988) Spark3 partition column value is not validated with user provided schema.

2022-11-01 Thread Ranga Reddy (Jira)
Ranga Reddy created SPARK-40988: --- Summary: Spark3 partition column value is not validated with user provided schema. Key: SPARK-40988 URL: https://issues.apache.org/jira/browse/SPARK-40988 Project: Spar

[jira] [Comment Edited] (SPARK-39405) NumPy input support in PySpark SQL

2022-11-01 Thread Douglas Moore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626930#comment-17626930 ] Douglas Moore edited comment on SPARK-39405 at 11/1/22 1:49 PM: --

[jira] [Resolved] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40978. -- Resolution: Fixed Issue resolved by pull request 38454 [https://github.com/apache/spark/pull/38454] >

[jira] [Resolved] (SPARK-40983) Remove Hadoop requirements for zstd mention in Parquet compression codec

2022-11-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40983. - Fix Version/s: 3.3.2 3.2.3 3.4.0 Resolution: Fixed

[jira] [Assigned] (SPARK-40983) Remove Hadoop requirements for zstd mention in Parquet compression codec

2022-11-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-40983: --- Assignee: Cheng Pan > Remove Hadoop requirements for zstd mention in Parquet compression co

[jira] [Resolved] (SPARK-40980) Support session.sql in Connect DSL

2022-11-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40980. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38459 [https://

[jira] [Assigned] (SPARK-40980) Support session.sql in Connect DSL

2022-11-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40980: - Assignee: Rui Wang > Support session.sql in Connect DSL > -

[jira] [Updated] (SPARK-40986) Add aggregate to reduce the data size for bloom filter

2022-11-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-40986: --- Summary: Add aggregate to reduce the data size for bloom filter (was: Using distinct to reduce the

[jira] [Commented] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627116#comment-17627116 ] Apache Spark commented on SPARK-40987: -- User 'cxzl25' has created a pull request fo

[jira] [Commented] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627113#comment-17627113 ] Apache Spark commented on SPARK-40987: -- User 'cxzl25' has created a pull request fo

[jira] [Assigned] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40987: Assignee: Apache Spark > Avoid creating a directory when deleting a block, causing DAGSch

[jira] [Assigned] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40987: Assignee: (was: Apache Spark) > Avoid creating a directory when deleting a block, cau

[jira] [Created] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-01 Thread dzcxzl (Jira)
dzcxzl created SPARK-40987: -- Summary: Avoid creating a directory when deleting a block, causing DAGScheduler to not work Key: SPARK-40987 URL: https://issues.apache.org/jira/browse/SPARK-40987 Project: Spark

[jira] [Comment Edited] (SPARK-34210) Cannot create a record reader because of a previous error when spark accesses the hive on HBase table

2022-11-01 Thread zhangzhanchang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627107#comment-17627107 ] zhangzhanchang edited comment on SPARK-34210 at 11/1/22 11:05 AM:

[jira] [Commented] (SPARK-34210) Cannot create a record reader because of a previous error when spark accesses the hive on HBase table

2022-11-01 Thread zhangzhanchang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627107#comment-17627107 ] zhangzhanchang commented on SPARK-34210: [https://github.com/apache/spark/pull/3

[jira] [Commented] (SPARK-40986) Using distinct to reduce the data size for bloom filter

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627080#comment-17627080 ] Apache Spark commented on SPARK-40986: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-40986) Using distinct to reduce the data size for bloom filter

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40986: Assignee: Apache Spark > Using distinct to reduce the data size for bloom filter > --

[jira] [Commented] (SPARK-40986) Using distinct to reduce the data size for bloom filter

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627078#comment-17627078 ] Apache Spark commented on SPARK-40986: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-40986) Using distinct to reduce the data size for bloom filter

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40986: Assignee: (was: Apache Spark) > Using distinct to reduce the data size for bloom filt

[jira] [Updated] (SPARK-40986) Using distinct to reduce the data size for bloom filter

2022-11-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-40986: --- Summary: Using distinct to reduce the data size for bloom filter (was: Add extra aggregate on join

[jira] [Created] (SPARK-40986) Add extra aggregate on join key for bloom filter

2022-11-01 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-40986: -- Summary: Add extra aggregate on join key for bloom filter Key: SPARK-40986 URL: https://issues.apache.org/jira/browse/SPARK-40986 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-40985) Upgrade RoaringBitmap to 0.9.35

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627038#comment-17627038 ] Apache Spark commented on SPARK-40985: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-40985) Upgrade RoaringBitmap to 0.9.35

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40985: Assignee: (was: Apache Spark) > Upgrade RoaringBitmap to 0.9.35 > ---

[jira] [Commented] (SPARK-40985) Upgrade RoaringBitmap to 0.9.35

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627035#comment-17627035 ] Apache Spark commented on SPARK-40985: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-40985) Upgrade RoaringBitmap to 0.9.35

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40985: Assignee: Apache Spark > Upgrade RoaringBitmap to 0.9.35 > --

[jira] [Commented] (SPARK-32628) Use bloom filter to improve dynamicPartitionPruning

2022-11-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627027#comment-17627027 ] Apache Spark commented on SPARK-32628: -- User 'wangyum' has created a pull request f

[jira] [Created] (SPARK-40985) Upgrade RoaringBitmap to 0.9.35

2022-11-01 Thread Yang Jie (Jira)
Yang Jie created SPARK-40985: Summary: Upgrade RoaringBitmap to 0.9.35 Key: SPARK-40985 URL: https://issues.apache.org/jira/browse/SPARK-40985 Project: Spark Issue Type: Improvement Com

[jira] [Created] (SPARK-40984) Replace `FRAME_LESS_OFFSET_WITHOUT_FOLDABLE` with `NON_FOLDABLE_INPUT`

2022-11-01 Thread Yang Jie (Jira)
Yang Jie created SPARK-40984: Summary: Replace `FRAME_LESS_OFFSET_WITHOUT_FOLDABLE` with `NON_FOLDABLE_INPUT` Key: SPARK-40984 URL: https://issues.apache.org/jira/browse/SPARK-40984 Project: Spark

[jira] [Assigned] (SPARK-40981) Support session.range in Python client

2022-11-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40981: - Assignee: Rui Wang > Support session.range in Python client > -

[jira] [Resolved] (SPARK-40981) Support session.range in Python client

2022-11-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40981. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38460 [https://

[jira] [Resolved] (SPARK-40890) Check error classes in DataSourceV2SQLSuite

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40890. -- Resolution: Fixed Issue resolved by pull request 38439 [https://github.com/apache/spark/pull/38439] >

[jira] [Assigned] (SPARK-40890) Check error classes in DataSourceV2SQLSuite

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40890: Assignee: BingKun Pan > Check error classes in DataSourceV2SQLSuite > ---

[jira] [Resolved] (SPARK-40371) Migrate type check failures of NthValue and NTile onto error classes

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40371. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38457 [https://github.com

[jira] [Assigned] (SPARK-40371) Migrate type check failures of NthValue and NTile onto error classes

2022-11-01 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40371: Assignee: Yang Jie > Migrate type check failures of NthValue and NTile onto error classes > -

  1   2   >