[jira] [Resolved] (SPARK-42916) JDBCCatalog Keep Char/Varchar meta information on the read-side

2023-04-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42916. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40543

[jira] [Assigned] (SPARK-42916) JDBCCatalog Keep Char/Varchar meta information on the read-side

2023-04-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-42916: Assignee: Kent Yao > JDBCCatalog Keep Char/Varchar meta information on the read-side >

[jira] [Commented] (SPARK-43119) Support Get SQL Keywords Dynamically

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711675#comment-17711675 ] Snoot.io commented on SPARK-43119: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Commented] (SPARK-43119) Support Get SQL Keywords Dynamically

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711676#comment-17711676 ] Snoot.io commented on SPARK-43119: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Created] (SPARK-43119) Support Get SQL Keywords Dynamically

2023-04-12 Thread Kent Yao (Jira)
Kent Yao created SPARK-43119: Summary: Support Get SQL Keywords Dynamically Key: SPARK-43119 URL: https://issues.apache.org/jira/browse/SPARK-43119 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-43021) Shuffle happens when Coalesce Buckets should occur

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711666#comment-17711666 ] Snoot.io commented on SPARK-43021: -- User 'ming95' has created a pull request for this issue:

[jira] [Commented] (SPARK-43118) Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711668#comment-17711668 ] Snoot.io commented on SPARK-43118: -- User 'jerrypeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-43118) Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711667#comment-17711667 ] Snoot.io commented on SPARK-43118: -- User 'jerrypeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-37099) Introduce a rank-based filter to optimize top-k computation

2023-04-12 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711665#comment-17711665 ] Snoot.io commented on SPARK-37099: -- User 'ulysses-you' has created a pull request for this issue:

[jira] [Updated] (SPARK-43118) Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream

2023-04-12 Thread Boyang Jerry Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-43118: -- Description: The assert    {code:java}

[jira] [Created] (SPARK-43118) Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream

2023-04-12 Thread Boyang Jerry Peng (Jira)
Boyang Jerry Peng created SPARK-43118: - Summary: Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream Key: SPARK-43118 URL: https://issues.apache.org/jira/browse/SPARK-43118

[jira] [Created] (SPARK-43117) proto message abbreviation should support repeated and map fields

2023-04-12 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-43117: - Summary: proto message abbreviation should support repeated and map fields Key: SPARK-43117 URL: https://issues.apache.org/jira/browse/SPARK-43117 Project: Spark

[jira] [Resolved] (SPARK-43115) Split pyspark-pandas-connect from pyspark-connect module.

2023-04-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43115. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40764

[jira] [Assigned] (SPARK-43115) Split pyspark-pandas-connect from pyspark-connect module.

2023-04-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43115: - Assignee: Takuya Ueshin > Split pyspark-pandas-connect from pyspark-connect module. >

[jira] [Updated] (SPARK-43116) Fix Cast.forceNullable

2023-04-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43116: Description:

[jira] [Updated] (SPARK-43116) Fix Cast.forceNullable

2023-04-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43116: Description:

[jira] [Created] (SPARK-43116) Fix Cast.forceNullable

2023-04-12 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-43116: --- Summary: Fix Cast.forceNullable Key: SPARK-43116 URL: https://issues.apache.org/jira/browse/SPARK-43116 Project: Spark Issue Type: Bug Components:

[jira] [Assigned] (SPARK-43063) `df.show` handle null should print NULL instead of null

2023-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43063: --- Assignee: yikaifei > `df.show` handle null should print NULL instead of null >

[jira] [Resolved] (SPARK-43063) `df.show` handle null should print NULL instead of null

2023-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43063. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40699

[jira] [Resolved] (SPARK-43110) Move asIntegral to PhysicalDataType

2023-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43110. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40758

[jira] [Updated] (SPARK-43101) Add CREATE/DROP catalog

2023-04-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43101: -- Description: Convenient registration of the catalog, in sts ref:

[jira] [Updated] (SPARK-43101) Dynamic Catalogs

2023-04-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43101: -- Summary: Dynamic Catalogs (was: Add CREATE/DROP catalog ) > Dynamic Catalogs > > >

[jira] [Updated] (SPARK-43101) Add CREATE/DROP catalog

2023-04-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43101: -- Description: Convenient registration of the catalog, in sts ref:

[jira] [Commented] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711614#comment-17711614 ] Bruce Robbins commented on SPARK-43113: --- PR here: https://github.com/apache/spark/pull/40766/files

[jira] [Assigned] (SPARK-43031) Enable tests for Python streaming spark-connect

2023-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43031: Assignee: Wei Liu > Enable tests for Python streaming spark-connect >

[jira] [Resolved] (SPARK-43031) Enable tests for Python streaming spark-connect

2023-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43031. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40691

[jira] [Commented] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711607#comment-17711607 ] Asif commented on SPARK-43112: -- Open a WIP PR [SPARK-43112|https://github.com/apache/spark/pull/40765/]

[jira] [Updated] (SPARK-43107) Coalesce buckets in join applied on broadcast join stream side

2023-04-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43107: Summary: Coalesce buckets in join applied on broadcast join stream side (was: Coalesce applied

[jira] [Commented] (SPARK-43107) Coalesce applied on broadcast join stream side

2023-04-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711605#comment-17711605 ] Yuming Wang commented on SPARK-43107: - https://github.com/apache/spark/pull/40756 > Coalesce

[jira] [Commented] (SPARK-43114) Add interval types to TypeCoercionSuite

2023-04-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711604#comment-17711604 ] Yuming Wang commented on SPARK-43114: - https://github.com/apache/spark/pull/40763 > Add interval

[jira] [Created] (SPARK-43115) Split pyspark-pandas-connect from pyspark-connect module.

2023-04-12 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43115: - Summary: Split pyspark-pandas-connect from pyspark-connect module. Key: SPARK-43115 URL: https://issues.apache.org/jira/browse/SPARK-43115 Project: Spark

[jira] [Created] (SPARK-43114) Add interval types to TypeCoercionSuite

2023-04-12 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-43114: --- Summary: Add interval types to TypeCoercionSuite Key: SPARK-43114 URL: https://issues.apache.org/jira/browse/SPARK-43114 Project: Spark Issue Type:

[jira] [Created] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-12 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43113: - Summary: Codegen error when full outer join's bound condition has multiple references to the same stream-side column Key: SPARK-43113 URL:

[jira] [Updated] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-43112: - Description: The class org.apache.spark.sql.catalyst.catalog.HiveTableRelation has its output method

[jira] [Created] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
Asif created SPARK-43112: Summary: Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables Key: SPARK-43112 URL:

[jira] [Commented] (SPARK-43111) Merge nested if statements into single if statements

2023-04-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-43111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711554#comment-17711554 ] Bjørn Jørgensen commented on SPARK-43111: - https://github.com/apache/spark/pull/40759 > Merge

[jira] [Created] (SPARK-43111) Merge nested if statements into single if statements

2023-04-12 Thread Jira
Bjørn Jørgensen created SPARK-43111: --- Summary: Merge nested if statements into single if statements Key: SPARK-43111 URL: https://issues.apache.org/jira/browse/SPARK-43111 Project: Spark

[jira] [Resolved] (SPARK-42437) Pyspark catalog.cacheTable allow to specify storage level Connect add support Storagelevel

2023-04-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-42437. --- Fix Version/s: 3.5.0 Assignee: Khalid Mammadov Resolution: Fixed Issue

[jira] [Commented] (SPARK-43084) Add Python state API (applyInPandasWithState) and verify UDFs

2023-04-12 Thread Peng Zhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711507#comment-17711507 ] Peng Zhong commented on SPARK-43084: applyInPandasWithState spark connect is added in this PR:

[jira] [Commented] (SPARK-43084) Add Python state API (applyInPandasWithState) and verify UDFs

2023-04-12 Thread Peng Zhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711505#comment-17711505 ] Peng Zhong commented on SPARK-43084: Verified that udf working in streaming spark connect:  

[jira] [Created] (SPARK-43110) Move asIntegral to PhysicalDataType

2023-04-12 Thread Rui Wang (Jira)
Rui Wang created SPARK-43110: Summary: Move asIntegral to PhysicalDataType Key: SPARK-43110 URL: https://issues.apache.org/jira/browse/SPARK-43110 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-16484) Incremental Cardinality estimation operations with Hyperloglog

2023-04-12 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711481#comment-17711481 ] Hudson commented on SPARK-16484: User 'RyanBerti' has created a pull request for this issue:

[jira] [Created] (SPARK-43109) JavaRDD.saveAsTextFile Directory Creation issue using Spark 3.3.2, with hadoop3

2023-04-12 Thread shamim (Jira)
shamim created SPARK-43109: -- Summary: JavaRDD.saveAsTextFile Directory Creation issue using Spark 3.3.2, with hadoop3 Key: SPARK-43109 URL: https://issues.apache.org/jira/browse/SPARK-43109 Project: Spark

[jira] [Updated] (SPARK-43108) org.apache.spark.storage.StorageStatus NotSerializableException when try to access StorageStatus in a MapPartitionsFunction

2023-04-12 Thread surender godara (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] surender godara updated SPARK-43108: Description: When you try to access the *storage status

[jira] [Created] (SPARK-43108) org.apache.spark.storage.StorageStatus NotSerializableException when try to access StorageStatus in a MapPartitionsFunction

2023-04-12 Thread surender godara (Jira)
surender godara created SPARK-43108: --- Summary: org.apache.spark.storage.StorageStatus NotSerializableException when try to access StorageStatus in a MapPartitionsFunction Key: SPARK-43108 URL:

[jira] [Resolved] (SPARK-43038) Support the CBC mode by aes_encrypt()/aes_decrypt()

2023-04-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43038. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40704

[jira] [Created] (SPARK-43107) Coalesce applied on broadcast join stream side

2023-04-12 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-43107: --- Summary: Coalesce applied on broadcast join stream side Key: SPARK-43107 URL: https://issues.apache.org/jira/browse/SPARK-43107 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43103) Move Integral to PhysicalDataType

2023-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43103. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40752

[jira] [Commented] (SPARK-43105) Abbreviate Bytes in proto message's debug string

2023-04-12 Thread GridGain Integration (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711311#comment-17711311 ] GridGain Integration commented on SPARK-43105: -- User 'zhengruifeng' has created a pull

[jira] [Updated] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-12 Thread vaibhav beriwala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vaibhav beriwala updated SPARK-43106: - Description: When we run an INSERT OVERWRITE query for an unpartitioned table on

[jira] [Updated] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-12 Thread vaibhav beriwala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vaibhav beriwala updated SPARK-43106: - Description: When we run an INSERT OVERWRITE query for an unpartitioned table on

[jira] [Updated] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-12 Thread vaibhav beriwala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vaibhav beriwala updated SPARK-43106: - Issue Type: Bug (was: Improvement) > Data lost from the table if the INSERT OVERWRITE

[jira] [Created] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-12 Thread vaibhav beriwala (Jira)
vaibhav beriwala created SPARK-43106: Summary: Data lost from the table if the INSERT OVERWRITE query fails Key: SPARK-43106 URL: https://issues.apache.org/jira/browse/SPARK-43106 Project: Spark

[jira] [Created] (SPARK-43105) Abbreviate Bytes in proto message's debug string

2023-04-12 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-43105: - Summary: Abbreviate Bytes in proto message's debug string Key: SPARK-43105 URL: https://issues.apache.org/jira/browse/SPARK-43105 Project: Spark Issue

[jira] [Assigned] (SPARK-42994) Torch Distributor support Local Mode

2023-04-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-42994: - Assignee: Ruifeng Zheng > Torch Distributor support Local Mode >

[jira] [Resolved] (SPARK-42994) Torch Distributor support Local Mode

2023-04-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42994. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40695

[jira] [Created] (SPARK-43104) Set `shadeTestJar` of protobuf module to false

2023-04-12 Thread Yang Jie (Jira)
Yang Jie created SPARK-43104: Summary: Set `shadeTestJar` of protobuf module to false Key: SPARK-43104 URL: https://issues.apache.org/jira/browse/SPARK-43104 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-42985) Fix createDataFrame from pandas to respect session timezone.

2023-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42985. -- Fix Version/s: 3.5.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in

[jira] [Resolved] (SPARK-43055) createDataFrame should support duplicated nested field names

2023-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43055. -- Fix Version/s: 3.5.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in

[jira] [Created] (SPARK-43103) Move Integral to PhysicalDataType

2023-04-12 Thread Rui Wang (Jira)
Rui Wang created SPARK-43103: Summary: Move Integral to PhysicalDataType Key: SPARK-43103 URL: https://issues.apache.org/jira/browse/SPARK-43103 Project: Spark Issue Type: Sub-task