[jira] [Assigned] (SPARK-42696) Speed up parquet reading with Java Vector API

2023-03-07 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-42696: --- Assignee: Yuming Wang > Speed up parquet reading with Java Vector API > ---

[jira] [Commented] (SPARK-42694) Data duplication and loss occur after executing 'insert overwrite...' in Spark 3.1.1

2023-03-07 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697316#comment-17697316 ] Yuming Wang commented on SPARK-42694: - Could you upgrade to Spark 3.1.3 or Spark 3.3

[jira] [Resolved] (SPARK-42682) When I execute the spark-shell command, "WARN ui.JettyUtils: GET /jobs/ failed: java.util.NoSuchElementException: Failed to get the application information. If you are

2023-03-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42682. - Resolution: Invalid This is because the spark context is not fully started. > When I execute th

[jira] [Resolved] (SPARK-34145) Combine scalar subqueries

2023-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34145. - Resolution: Not A Problem Issue fixed through SPARK-34079. > Combine scalar subqueries > --

[jira] [Resolved] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2023-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34117. - Resolution: Not A Problem Issue fixed through SPARK-34081. > Disable LeftSemi/LeftAnti push dow

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696665#comment-17696665 ] Yuming Wang commented on SPARK-42650: - It seems to be caused by https://github.com/a

[jira] [Resolved] (SPARK-42675) Should clean up temp view after test

2023-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42675. - Fix Version/s: 3.4.0 Assignee: Yang Jie (was: Apache Spark) Resolution: Fixed >

[jira] [Commented] (SPARK-40610) Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause

2023-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696634#comment-17696634 ] Yuming Wang commented on SPARK-40610: - This is because of [https://github.com/apach

[jira] [Resolved] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42473. - Fix Version/s: 3.3.3 Assignee: Runyao.Chen Resolution: Fixed > An explicit cast

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695989#comment-17695989 ] Yuming Wang commented on SPARK-42650: - It seems like a Kyuubi bug? > link issue SPA

[jira] [Updated] (SPARK-42597) UnwrapCastInBinaryComparison support unwrap timestamp type

2023-02-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42597: Summary: UnwrapCastInBinaryComparison support unwrap timestamp type (was: UnwrapCastInBinaryCompa

[jira] [Created] (SPARK-42597) UnwrapCastInBinaryComparison also support timestamp type

2023-02-27 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42597: --- Summary: UnwrapCastInBinaryComparison also support timestamp type Key: SPARK-42597 URL: https://issues.apache.org/jira/browse/SPARK-42597 Project: Spark Issue

[jira] [Resolved] (SPARK-42594) spark can not read lastest view sql when run `create or replace view` by hive

2023-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42594. - Resolution: Not A Bug > spark can not read lastest view sql when run `create or replace view` by

[jira] [Reopened] (SPARK-42594) spark can not read lastest view sql when run `create or replace view` by hive

2023-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-42594: - > spark can not read lastest view sql when run `create or replace view` by hive > --

[jira] [Commented] (SPARK-42594) spark can not read lastest view sql when run `create or replace view` by hive

2023-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693799#comment-17693799 ] Yuming Wang commented on SPARK-42594: - Spark saves information to table properties,

[jira] [Updated] (SPARK-42583) Remove outer join if all aggregate functions are distinct

2023-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42583: Description: To support more cases: https://github.com/pingcap/tidb/blob/master/planner/core/rule_

[jira] [Created] (SPARK-42583) Remove outer join if all aggregate functions are distinct

2023-02-26 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42583: --- Summary: Remove outer join if all aggregate functions are distinct Key: SPARK-42583 URL: https://issues.apache.org/jira/browse/SPARK-42583 Project: Spark Issue

[jira] [Resolved] (SPARK-42525) collapse two adjacent windows with the same partition/order in subquery

2023-02-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42525. - Fix Version/s: 3.5.0 Assignee: zhuml Resolution: Fixed > collapse two adjacent w

[jira] [Resolved] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42550. - Resolution: Cannot Reproduce > table directory will lost on hdfs when `INSERT OVERWRITE` faild >

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693103#comment-17693103 ] Yuming Wang commented on SPARK-42550: - I can't reproduce this issue: {noformat} spar

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693016#comment-17693016 ] Yuming Wang commented on SPARK-42550: - Please refresh table: {code:sql} refresh tabl

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42550: Description: {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during execut

[jira] [Updated] (SPARK-40158) Remove useless configuration & extract common code for parquet read

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40158: Fix Version/s: (was: 3.4.0) > Remove useless configuration & extract common code for parquet r

[jira] [Updated] (SPARK-40355) Improve pushdown for orc & parquet when cast scenario

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40355: Fix Version/s: (was: 3.4.0) > Improve pushdown for orc & parquet when cast scenario >

[jira] [Updated] (SPARK-42513) Push down topK through join

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42513: Summary: Push down topK through join (was: Push topK through join) > Push down topK through join

[jira] [Updated] (SPARK-42513) Push topK through join

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42513: Attachment: before-UI.png > Push topK through join > -- > > Ke

[jira] [Updated] (SPARK-42513) Push topK through join

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42513: Attachment: after-UI.png > Push topK through join > -- > > Key

[jira] [Created] (SPARK-42513) Push topK through join

2023-02-21 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42513: --- Summary: Push topK through join Key: SPARK-42513 URL: https://issues.apache.org/jira/browse/SPARK-42513 Project: Spark Issue Type: Improvement Compon

[jira] [Resolved] (SPARK-42506) Fix Sort's maxRowsPerPartition if maxRows does not exist

2023-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42506. - Resolution: Not A Problem > Fix Sort's maxRowsPerPartition if maxRows does not exist > -

[jira] [Created] (SPARK-42506) Fix Sort's maxRowsPerPartition if maxRows does not exist

2023-02-20 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42506: --- Summary: Fix Sort's maxRowsPerPartition if maxRows does not exist Key: SPARK-42506 URL: https://issues.apache.org/jira/browse/SPARK-42506 Project: Spark Issue

[jira] [Commented] (SPARK-40278) Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691388#comment-17691388 ] Yuming Wang commented on SPARK-40278: - [~LuciferYang] Is this issue still exist? >

[jira] [Commented] (SPARK-40610) Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691387#comment-17691387 ] Yuming Wang commented on SPARK-40610: - [~icyjhl] What's your dt data type? date, str

[jira] [Commented] (SPARK-42503) Spark SQL should do further validation on join condition fields

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691321#comment-17691321 ] Yuming Wang commented on SPARK-42503: - Do other databases also have this validation?

[jira] [Updated] (SPARK-42503) Spark SQL should do further validation on join condition fields

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42503: Fix Version/s: (was: 3.4.0) > Spark SQL should do further validation on join condition fields

[jira] [Updated] (SPARK-42503) Spark SQL should do further validation on join condition fields

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42503: Target Version/s: (was: 3.4.0) > Spark SQL should do further validation on join condition fields

[jira] [Resolved] (SPARK-41741) [SQL] ParquetFilters StringStartsWith push down matching string do not use UTF-8

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-41741. - Resolution: Fixed > [SQL] ParquetFilters StringStartsWith push down matching string do not use

[jira] [Assigned] (SPARK-41741) [SQL] ParquetFilters StringStartsWith push down matching string do not use UTF-8

2023-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-41741: --- Fix Version/s: 3.4.0 3.3.3 Assignee: Yuming Wang > [SQL] Parque

[jira] [Created] (SPARK-42500) ConstantPropagation support more cases

2023-02-20 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42500: --- Summary: ConstantPropagation support more cases Key: SPARK-42500 URL: https://issues.apache.org/jira/browse/SPARK-42500 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-42286) Fix internal error for valid CASE WHEN expression with CAST when inserting into a table

2023-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-42286: --- Fix Version/s: 3.4.0 Assignee: Runyao.Chen > Fix internal error for valid CASE WHE

[jira] [Commented] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691011#comment-17691011 ] Yuming Wang commented on SPARK-42473: - It seems we should backport https://github.co

[jira] [Commented] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690833#comment-17690833 ] Yuming Wang commented on SPARK-42473: - What is your test.spark33_decimal_orc column

[jira] [Assigned] (SPARK-39851) Improve join stats estimation if one side can keep uniqueness

2023-02-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-39851: --- Assignee: Wan Kun > Improve join stats estimation if one side can keep uniqueness > ---

[jira] [Updated] (SPARK-42450) dataset.where() omit quotes if where IN clause has more than 10 operands

2023-02-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42450: Fix Version/s: (was: 3.4.0) (was: 3.3.2) > dataset.where() omit quotes

[jira] [Updated] (SPARK-42450) dataset.where() omit quotes if where IN clause has more than 10 operands

2023-02-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-42450: Target Version/s: (was: 3.3.0) > dataset.where() omit quotes if where IN clause has more than 10

[jira] [Assigned] (SPARK-42451) Remove 3.1 and Java 17 check from filter condition of `testingVersions` in `HiveExternalCatalogVersionsSuite`

2023-02-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-42451: --- Assignee: Yang Jie > Remove 3.1 and Java 17 check from filter condition of `testingVersion

[jira] [Resolved] (SPARK-42451) Remove 3.1 and Java 17 check from filter condition of `testingVersions` in `HiveExternalCatalogVersionsSuite`

2023-02-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42451. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40039 [https://gith

[jira] [Resolved] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42346. - Fix Version/s: 3.3.2 3.4.0 3.5.0 Assignee: Peter T

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684177#comment-17684177 ] Yuming Wang commented on SPARK-42346: - cc [~petertoth] > distinct(count colname) wi

[jira] [Created] (SPARK-42333) Change log level to debug when fetching result set from SparkExecuteStatementOperation

2023-02-03 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42333: --- Summary: Change log level to debug when fetching result set from SparkExecuteStatementOperation Key: SPARK-42333 URL: https://issues.apache.org/jira/browse/SPARK-42333

[jira] [Updated] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2023-02-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-14922: Target Version/s: 3.5.0 (was: 3.2.0) > Alter Table Drop Partition Using Predicate-based Partition

[jira] [Reopened] (SPARK-40303) The performance will be worse after codegen

2023-01-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-40303: - > The performance will be worse after codegen > --- > >

[jira] [Resolved] (SPARK-42063) Register `byte[][]` to KyroSerializer

2023-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-42063. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39569 [https://gith

[jira] [Assigned] (SPARK-42063) Register `byte[][]` to KyroSerializer

2023-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-42063: --- Assignee: Dongjoon Hyun > Register `byte[][]` to KyroSerializer > -

[jira] [Created] (SPARK-42064) Implement bloom filter join hint

2023-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-42064: --- Summary: Implement bloom filter join hint Key: SPARK-42064 URL: https://issues.apache.org/jira/browse/SPARK-42064 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-39217) Makes DPP support the pruning side has Union

2023-01-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-39217: --- Assignee: Wan Kun > Makes DPP support the pruning side has Union >

[jira] [Resolved] (SPARK-39217) Makes DPP support the pruning side has Union

2023-01-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-39217. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39460 [https://gith

[jira] [Created] (SPARK-41986) Introduce shuffle on SinglePartition

2023-01-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-41986: --- Summary: Introduce shuffle on SinglePartition Key: SPARK-41986 URL: https://issues.apache.org/jira/browse/SPARK-41986 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-41741) [SQL] ParquetFilters StringStartsWith push down matching string do not use UTF-8

2023-01-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655893#comment-17655893 ] Yuming Wang commented on SPARK-41741: - What is your env? you can put the env in your

[jira] [Commented] (SPARK-41741) [SQL] ParquetFilters StringStartsWith push down matching string do not use UTF-8

2023-01-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655882#comment-17655882 ] Yuming Wang commented on SPARK-41741: - What is your {{file.encoding}}? > [SQL] Parq

[jira] [Updated] (SPARK-41498) Union does not propagate Metadata output

2022-12-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-41498: Fix Version/s: (was: 3.4.0) > Union does not propagate Metadata output > -

[jira] [Commented] (SPARK-41459) Spark Thrift Server operation log output is empty

2022-12-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645111#comment-17645111 ] Yuming Wang commented on SPARK-41459: - cc [~LuciferYang] > Spark Thrift Server oper

[jira] [Resolved] (SPARK-34987) AQE improve: change shuffle hash join to sort merge join when skewed shuffle hash join exists

2022-12-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34987. - Resolution: Not A Problem > AQE improve: change shuffle hash join to sort merge join when skewed

[jira] [Resolved] (SPARK-41167) Optimize LikeSimplification rule to improve multi like performance

2022-12-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-41167. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38682 [https://gith

[jira] [Assigned] (SPARK-41167) Optimize LikeSimplification rule to improve multi like performance

2022-12-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-41167: --- Assignee: Wan Kun > Optimize LikeSimplification rule to improve multi like performance > --

[jira] [Commented] (SPARK-41336) BroadcastExchange does not support the execute() code path. when AQE enabled

2022-11-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641191#comment-17641191 ] Yuming Wang commented on SPARK-41336: - Could you use Spark 3.3.1? > BroadcastExchan

[jira] [Commented] (SPARK-41324) Follow-up on JDK-8180450

2022-11-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641155#comment-17641155 ] Yuming Wang commented on SPARK-41324: - cc [~LuciferYang] > Follow-up on JDK-8180450

[jira] [Commented] (SPARK-41299) OOM when filter pushdown `last_day` function

2022-11-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640347#comment-17640347 ] Yuming Wang commented on SPARK-41299: - Do you have the query plan? > OOM when filte

[jira] [Commented] (SPARK-41219) Regression in IntegralDivide returning null instead of 0

2022-11-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637102#comment-17637102 ] Yuming Wang commented on SPARK-41219: - cc [~ulysses] > Regression in IntegralDivide

[jira] [Commented] (SPARK-41207) Regression in IntegralDivide

2022-11-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636383#comment-17636383 ] Yuming Wang commented on SPARK-41207: - cc [~ulysses] > Regression in IntegralDivide

[jira] [Updated] (SPARK-41207) Regression in IntegralDivide

2022-11-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-41207: Target Version/s: (was: 3.4.0) > Regression in IntegralDivide > > >

[jira] [Updated] (SPARK-41207) Regression in IntegralDivide

2022-11-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-41207: Fix Version/s: (was: 3.4.0) > Regression in IntegralDivide > > >

[jira] [Resolved] (SPARK-41017) Support column pruning with multiple nondeterministic Filters

2022-11-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-41017. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38511 [https://gith

[jira] [Updated] (SPARK-41141) avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

2022-11-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-41141: Target Version/s: (was: 3.3.1) > avoid introducing a new aggregate expression in the analysis ph

[jira] [Created] (SPARK-41088) Add PartialAggregate and FinalAggregate logic operators

2022-11-09 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-41088: --- Summary: Add PartialAggregate and FinalAggregate logic operators Key: SPARK-41088 URL: https://issues.apache.org/jira/browse/SPARK-41088 Project: Spark Issue T

[jira] [Resolved] (SPARK-41071) Metaspace OOM when Local run dev/make-distribution.sh

2022-11-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-41071. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38577 [https://gith

[jira] [Assigned] (SPARK-41071) Metaspace OOM when Local run dev/make-distribution.sh

2022-11-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-41071: --- Assignee: Yang Jie > Metaspace OOM when Local run dev/make-distribution.sh > -

[jira] [Commented] (SPARK-41013) spark-3.1.2以cluster模式提交作业报 Could not initialize class com.github.luben.zstd.ZstdOutputStream

2022-11-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17629240#comment-17629240 ] Yuming Wang commented on SPARK-41013: - Could you test the Spark 3.3.1? > spark-3.1.

[jira] [Updated] (SPARK-40999) Hints on subqueries are not properly propagated

2022-11-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40999: Fix Version/s: (was: 3.4.0) > Hints on subqueries are not properly propagated > --

[jira] [Resolved] (SPARK-40248) Use larger number of bits to build bloom filter

2022-11-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40248. - Fix Version/s: 3.4.0 Assignee: Yuming Wang Resolution: Fixed This is resolved vi

[jira] [Resolved] (SPARK-40983) Remove Hadoop requirements for zstd mention in Parquet compression codec

2022-11-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40983. - Fix Version/s: 3.3.2 3.2.3 3.4.0 Resolution: Fixed

[jira] [Assigned] (SPARK-40983) Remove Hadoop requirements for zstd mention in Parquet compression codec

2022-11-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-40983: --- Assignee: Cheng Pan > Remove Hadoop requirements for zstd mention in Parquet compression co

[jira] [Commented] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626490#comment-17626490 ] Yuming Wang commented on SPARK-40972: - cc [~michaelzhang-db] > OptimizeLocalShuffle

[jira] [Resolved] (SPARK-35904) Collapse above RebalancePartitions

2022-10-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35904. - Resolution: Not A Problem > Collapse above RebalancePartitions > ---

[jira] [Resolved] (SPARK-40929) Add official image dockerfile for Spark v3.3.1

2022-10-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40929. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 20 [https://github.

[jira] [Assigned] (SPARK-40929) Add official image dockerfile for Spark v3.3.1

2022-10-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-40929: --- Assignee: Yikun Jiang > Add official image dockerfile for Spark v3.3.1 > --

[jira] [Assigned] (SPARK-36057) SPIP: Support Customized Kubernetes Schedulers

2022-10-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-36057: --- Assignee: Yikun Jiang > SPIP: Support Customized Kubernetes Schedulers > --

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40874: Fix Version/s: 3.3.2 (was: 3.3.1) > Fix broadcasts in Python UDFs when encr

[jira] [Commented] (SPARK-36057) SPIP: Support Customized Kubernetes Schedulers

2022-10-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623505#comment-17623505 ] Yuming Wang commented on SPARK-36057: - Thank you [~dongjoon]. > SPIP: Support Custo

[jira] [Comment Edited] (SPARK-34966) Avoid shuffle if join type do not match

2022-10-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315591#comment-17315591 ] Yuming Wang edited comment on SPARK-34966 at 10/24/22 11:37 AM: --

[jira] [Updated] (SPARK-40885) Spark will filter out data field sorting when dynamic partitions and data fields are sorted at the same time

2022-10-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40885: Fix Version/s: (was: 3.4.0) > Spark will filter out data field sorting when dynamic partitions

[jira] [Commented] (SPARK-40303) The performance will be worse after codegen

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619978#comment-17619978 ] Yuming Wang commented on SPARK-40303: - How to run benchmark code: # Download lates

[jira] [Resolved] (SPARK-40736) Spark 3.3.0 doesn't works with Hive 3.1.2

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40736. - Resolution: Invalid > Spark 3.3.0 doesn't works with Hive 3.1.2 > --

[jira] [Commented] (SPARK-40736) Spark 3.3.0 doesn't works with Hive 3.1.2

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619501#comment-17619501 ] Yuming Wang commented on SPARK-40736: - Please do not copy Hive related jars to ${SPA

[jira] [Commented] (SPARK-40736) Spark 3.3.0 doesn't works with Hive 3.1.2

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619433#comment-17619433 ] Yuming Wang commented on SPARK-40736: - Do you copy hive related jars to ${SPARK_HOME

[jira] [Comment Edited] (SPARK-34966) Avoid shuffle if join type do not match

2022-10-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315591#comment-17315591 ] Yuming Wang edited comment on SPARK-34966 at 10/18/22 7:37 AM: ---

[jira] [Commented] (SPARK-40563) Error at where clause, when sql case executes by else branch

2022-10-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619189#comment-17619189 ] Yuming Wang commented on SPARK-40563: - Thank you [~Zing] > Error at where clause, w

[jira] [Assigned] (SPARK-39951) Support columnar batches with nested fields in Parquet V2

2022-10-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-39951: --- Assignee: Adam Binford (was: Apache Spark) > Support columnar batches with nested fields i

[jira] [Commented] (SPARK-40563) Error at where clause, when sql case executes by else branch

2022-10-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17618234#comment-17618234 ] Yuming Wang commented on SPARK-40563: - [~Zing] Does branch-3.3 also fixed this issue

[jira] [Updated] (SPARK-39200) Stream is corrupted Exception while fetching the blocks from fallback storage system

2022-10-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-39200: Fix Version/s: 3.3.1 (was: 3.3.2) > Stream is corrupted Exception while fet

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-10-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40535: Fix Version/s: 3.3.1 (was: 3.3.2) > NPE from observe of collect_list >

<    1   2   3   4   5   6   7   8   9   10   >