[jira] [Created] (SPARK-44520) Remove UNSUPPORTED_DATA_SOURCE_FOR_DIRECT_QUERY in favor of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY

2023-07-23 Thread Kent Yao (Jira)
Kent Yao created SPARK-44520: Summary: Remove UNSUPPORTED_DATA_SOURCE_FOR_DIRECT_QUERY in favor of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY Key: SPARK-44520 URL: https://issues.apache.org/jira/browse/SPARK-44520

[jira] [Created] (SPARK-44519) SparkConnectServerUtils generated incorrect parameters for jars

2023-07-23 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-44519: -- Summary: SparkConnectServerUtils generated incorrect parameters for jars Key: SPARK-44519 URL: https://issues.apache.org/jira/browse/SPARK-44519 Project: Spark

[jira] [Updated] (SPARK-44514) Optimize join if maximum number of rows on one side is 1

2023-07-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-44514: Summary: Optimize join if maximum number of rows on one side is 1 (was: Rewrite the join to

[jira] (SPARK-40588) Sorting issue with partitioned-writing and AQE turned on

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40588 ] Yiu-Chung Lee deleted comment on SPARK-40588: --- was (Author: JIRAUSER301473): I discovered similar sort-then-partitionby issue in Spark 3.4.1 in another use case, mentioned in SPARK-44512.

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2023-07-23 Thread He Qi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746191#comment-17746191 ] He Qi commented on SPARK-44518: --- [~LuciferYang] [~yumwang] [~Qin Yao] [~csun] WDYT? > Completely make

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2023-07-23 Thread He Qi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Qi updated SPARK-44518: -- Shepherd: (was: Yang Jie) > Completely make hive as a data source > - >

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2023-07-23 Thread He Qi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Qi updated SPARK-44518: -- Shepherd: Yang Jie (was: yu) > Completely make hive as a data source > -

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2023-07-23 Thread He Qi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Qi updated SPARK-44518: -- Shepherd: yu (was: Yang Jie) > Completely make hive as a data source > -

[jira] [Commented] (SPARK-44514) Rewrite the join to filter if one side maximum number of rows is 1

2023-07-23 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746185#comment-17746185 ] Snoot.io commented on SPARK-44514: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-44518) Completely make hive as a data source

2023-07-23 Thread He Qi (Jira)
He Qi created SPARK-44518: - Summary: Completely make hive as a data source Key: SPARK-44518 URL: https://issues.apache.org/jira/browse/SPARK-44518 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-44493) Extract pushable predicates from disjunctive predicates

2023-07-23 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746175#comment-17746175 ] Snoot.io commented on SPARK-44493: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-44033) Support list-like for binary ops

2023-07-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746167#comment-17746167 ] Haejoon Lee commented on SPARK-44033: - > Do we need to implement other binary ops just like this  

[jira] [Assigned] (SPARK-44513) Upgrade snappy-java to 1.1.10.3

2023-07-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44513: Assignee: BingKun Pan > Upgrade snappy-java to 1.1.10.3 > --- > >

[jira] [Resolved] (SPARK-44513) Upgrade snappy-java to 1.1.10.3

2023-07-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44513. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Created] (SPARK-44517) first operator should respect the nullability of child expression as well as ignoreNulls option

2023-07-23 Thread Nan Zhu (Jira)
Nan Zhu created SPARK-44517: --- Summary: first operator should respect the nullability of child expression as well as ignoreNulls option Key: SPARK-44517 URL: https://issues.apache.org/jira/browse/SPARK-44517

[jira] [Resolved] (SPARK-44510) Update dataTables to 1.13.5 and remove some unreached png files

2023-07-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44510. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44510) Update dataTables to 1.13.5 and remove some unreached png files

2023-07-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44510: Assignee: Kent Yao > Update dataTables to 1.13.5 and remove some unreached png files >

[jira] [Created] (SPARK-44516) Spark Connect Python StreamingQueryListener removeListener method

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44516: --- Summary: Spark Connect Python StreamingQueryListener removeListener method Key: SPARK-44516 URL: https://issues.apache.org/jira/browse/SPARK-44516 Project: Spark

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Description: (In this example the dataset is of type Tuple3, and the columns are named _1,

[jira] [Commented] (SPARK-40588) Sorting issue with partitioned-writing and AQE turned on

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746147#comment-17746147 ] Yiu-Chung Lee commented on SPARK-40588: --- I discovered similar sort-then-partitionby issue in Spark

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Labels: correctness (was: ) > dataset.sort.select.write.partitionBy does not return a sorted

[jira] [Commented] (SPARK-44514) Rewrite the join to filter if one side maximum number of rows is 1

2023-07-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746143#comment-17746143 ] Yuming Wang commented on SPARK-44514: - https://github.com/apache/spark/pull/42114 > Rewrite the

[jira] [Created] (SPARK-44515) Code Improvement: PySpark add util function to set python version

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44515: --- Summary: Code Improvement: PySpark add util function to set python version Key: SPARK-44515 URL: https://issues.apache.org/jira/browse/SPARK-44515 Project: Spark

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2023-07-23 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746127#comment-17746127 ] Szymon Kuryło commented on SPARK-33782: --- [~pratik.malani]  I have a similar problem, and thus a

[jira] [Commented] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

2023-07-23 Thread Jose Santos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746120#comment-17746120 ] Jose Santos commented on SPARK-36277: -- I know this issue has been created 2 Years ago, but if

[jira] [Created] (SPARK-44514) Rewrite the join to filter if one side maximum number of rows is 1

2023-07-23 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-44514: --- Summary: Rewrite the join to filter if one side maximum number of rows is 1 Key: SPARK-44514 URL: https://issues.apache.org/jira/browse/SPARK-44514 Project: Spark

[jira] [Created] (SPARK-44513) Upgrade snappy-java to 1.1.10.3

2023-07-23 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-44513: --- Summary: Upgrade snappy-java to 1.1.10.3 Key: SPARK-44513 URL: https://issues.apache.org/jira/browse/SPARK-44513 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Description: (In this example the dataset is of type Tuple3, and the columns are named _1,

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Description: (In this example the dataset is of type Tuple3, and the columns are named _1,

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Description: (In this example the dataset is of type Tuple3, and the columns are named _1,

[jira] [Commented] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746025#comment-17746025 ] Yiu-Chung Lee commented on SPARK-44512: --- [^Test.java] (Attached the code) To compile: javac

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Environment: (was: Code that replicates the problem:   import java.io.IOException;

[jira] [Created] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
Yiu-Chung Lee created SPARK-44512: - Summary: dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled Key: SPARK-44512 URL: https://issues.apache.org/jira/browse/SPARK-44512

[jira] [Updated] (SPARK-44512) dataset.sort.select.write.partitionBy does not return a sorted output if AQE is enabled

2023-07-23 Thread Yiu-Chung Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiu-Chung Lee updated SPARK-44512: -- Attachment: Test.java > dataset.sort.select.write.partitionBy does not return a sorted output