[jira] [Updated] (SPARK-36208) SparkScriptTransformation

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36208: --- Parent: SPARK-27790 Issue Type: Sub-task (was: Bug) > SparkScriptTransformation >

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: (was: Apache Spark) > Export databaseExists in pyspark.sql.catalog > --

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383322#comment-17383322 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: Apache Spark > Export databaseExists in pyspark.sql.catalog > -

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383321#comment-17383321 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull requ

[jira] [Created] (SPARK-36208) SparkScriptTransformation

Kousuke Saruta created SPARK-36208: -- Summary: SparkScriptTransformation Key: SPARK-36208 URL: https://issues.apache.org/jira/browse/SPARK-36208 Project: Spark Issue Type: Bug Comp

[jira] [Created] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

Dominik Gehl created SPARK-36207: Summary: Export databaseExists in pyspark.sql.catalog Key: SPARK-36207 URL: https://issues.apache.org/jira/browse/SPARK-36207 Project: Spark Issue Type: Impr

[jira] [Resolved] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36091. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33341 [https:

[jira] [Assigned] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36091: -- Assignee: jiaan.geng > Support TimestampNTZ type in expression TimeWindow >

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35806: Assignee: Haejoon Lee > Mapping the `mode` argument to pandas in DataFrame.to_csv > -

[jira] [Resolved] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35806. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33414 [https://gi

[jira] [Resolved] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36205. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33412 [https://gi

[jira] [Assigned] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36181: Assignee: Dominik Gehl > Update pyspark sql readwriter documentation to Scala level > ---

[jira] [Resolved] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36181. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33394 [https://gi

[jira] [Resolved] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36178. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33392 [https://gi

[jira] [Assigned] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36178: Assignee: Dominik Gehl > Document PySpark Catalog APIs in docs/source/reference/pyspark.s

[jira] [Assigned] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36205: Assignee: Hyukjin Kwon > Use set-env instead of set-output in GitHub Actions > --

[jira] [Commented] (SPARK-36086) The case of the delta table is inconsistent with parquet

[ https://issues.apache.org/jira/browse/SPARK-36086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383195#comment-17383195 ] Wenchen Fan commented on SPARK-36086: - Seems we should improve the v2 describe table

[jira] [Assigned] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34806: --- Assignee: Enrico Minack > Helper class for batch Dataset.observe() > --

[jira] [Resolved] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34806. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 31905 [https://gith

[jira] [Commented] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383192#comment-17383192 ] tiejiang commented on SPARK-24965: -- I have a similar question, see the link, can anyone

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: (was: Apache Spark) > dropDuplicates does not type check argument > ---

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: Apache Spark > dropDuplicates does not type check argument > --

[jira] [Commented] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383175#comment-17383175 ] Apache Spark commented on SPARK-36161: -- User 'sammyjmoseley' has created a pull req

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: Apache Spark > Mapping the `mode` argument to pandas in DataFrame.to_csv >

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: (was: Apache Spark) > Mapping the `mode` argument to pandas in DataFrame.to

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383172#comment-17383172 ] Apache Spark commented on SPARK-35806: -- User 'itholic' has created a pull request f

[jira] [Assigned] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36163: Assignee: Ivan > Propagate correct JDBC properties in JDBC connector provider and add >

[jira] [Resolved] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36163. -- Fix Version/s: 3.3.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/33

[jira] [Commented] (SPARK-36185) Implement functions in CategoricalAccessor/CategoricalIndex

[ https://issues.apache.org/jira/browse/SPARK-36185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383146#comment-17383146 ] Hyukjin Kwon commented on SPARK-36185: -- I think it's for Spark 3.2. Most of fixes a

[jira] [Commented] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383144#comment-17383144 ] Hyukjin Kwon commented on SPARK-36187: -- For question, let's interact it with Spark

[jira] [Resolved] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36187. -- Resolution: Incomplete > Commit collision avoidance in dynamicPartitionOverwrite for non-Parqu

[jira] [Updated] (SPARK-36192) Better error messages when comparing against list

[ https://issues.apache.org/jira/browse/SPARK-36192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36192: - Description: We shall throw TypeError messages rather than Spark exceptions. > Better error mess

[jira] [Resolved] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36203. -- Resolution: Incomplete > Spark SQL can't use "group by" on the column of map type. > -

[jira] [Commented] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383143#comment-17383143 ] Hyukjin Kwon commented on SPARK-36203: -- Can you show the fullly self-contained repr

[jira] [Resolved] (SPARK-36134) jackson-databind RCE vulnerability

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36134. -- Resolution: Invalid > jackson-databind RCE vulnerability > --

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383131#comment-17383131 ] Hyukjin Kwon commented on SPARK-36088: -- cc [~dongjoon] and [~holdenkarau] FYI > 's

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383130#comment-17383130 ] Hyukjin Kwon commented on SPARK-36088: -- You might have to call https://github.com/

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have an argument named `mode` in the [DataFrame.to_c

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383128#comment-17383128 ] Hyukjin Kwon commented on SPARK-36088: -- does your driver run inside a pod or on a p

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Summary: Mapping the `mode` argument to pandas in DataFrame.to_csv (was: Mapping the `mode` argum

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have a argument named `mode` in the [DataFrame.to_cs

[jira] [Commented] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383125#comment-17383125 ] Apache Spark commented on SPARK-36201: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: Apache Spark > Add check for inner field of schema > --

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: (was: Apache Spark) > Add check for inner field of schema > ---

[jira] [Resolved] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-36197. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33406 [https://github.com

[jira] [Assigned] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-36197: Assignee: Kent Yao > InputFormat of PartitionDesc is not respected >

<    1   2