[jira] [Commented] (SPARK-46295) TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join

[ https://issues.apache.org/jira/browse/SPARK-46295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794081#comment-17794081 ] Kazuyuki Tanimura commented on SPARK-46295: --- I realized that I am using the de

[jira] [Updated] (SPARK-46295) TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join

[ https://issues.apache.org/jira/browse/SPARK-46295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-46295: -- Affects Version/s: 3.4.2 (was: 3.4.1) > TPCDS q39a and a39b

[jira] [Updated] (SPARK-46295) TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join

[ https://issues.apache.org/jira/browse/SPARK-46295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-46295: -- Labels: correctness (was: ) > TPCDS q39a and a39b have correctness issues with broadc

[jira] [Created] (SPARK-46295) TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join

Kazuyuki Tanimura created SPARK-46295: - Summary: TPCDS q39a and a39b have correctness issues with broadcast hash join and shuffled hash join Key: SPARK-46295 URL: https://issues.apache.org/jira/browse/SPARK-46

[jira] [Updated] (SPARK-45786) Inaccurate Decimal multiplication and division results

[ https://issues.apache.org/jira/browse/SPARK-45786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-45786: -- Affects Version/s: 4.0.0 > Inaccurate Decimal multiplication and division results > --

[jira] [Created] (SPARK-45786) Inaccurate Decimal multiplication and division results

Kazuyuki Tanimura created SPARK-45786: - Summary: Inaccurate Decimal multiplication and division results Key: SPARK-45786 URL: https://issues.apache.org/jira/browse/SPARK-45786 Project: Spark

[jira] [Created] (SPARK-42833) Refactor `applyExtensions` in `SparkSession`

Kazuyuki Tanimura created SPARK-42833: - Summary: Refactor `applyExtensions` in `SparkSession` Key: SPARK-42833 URL: https://issues.apache.org/jira/browse/SPARK-42833 Project: Spark Issue

[jira] [Updated] (SPARK-42256) SPIP: Lazy Materialization for Parquet Read Performance Improvement

[ https://issues.apache.org/jira/browse/SPARK-42256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-42256: -- Description: Spark-SQL filter operation is a common workload in order to select sp

[jira] [Created] (SPARK-42256) SPIP: Lazy Materialization for Parquet Read Performance Improvement

Kazuyuki Tanimura created SPARK-42256: - Summary: SPIP: Lazy Materialization for Parquet Read Performance Improvement Key: SPARK-42256 URL: https://issues.apache.org/jira/browse/SPARK-42256 Project

[jira] [Created] (SPARK-41096) Support reading parquet FIXED_LEN_BYTE_ARRAY type

Kazuyuki Tanimura created SPARK-41096: - Summary: Support reading parquet FIXED_LEN_BYTE_ARRAY type Key: SPARK-41096 URL: https://issues.apache.org/jira/browse/SPARK-41096 Project: Spark I

[jira] [Resolved] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

[ https://issues.apache.org/jira/browse/SPARK-40477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura resolved SPARK-40477. --- Resolution: Won't Fix gave another thought and decided to close this one not to be f

[jira] [Created] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

Kazuyuki Tanimura created SPARK-40477: - Summary: Support `NullType` in `ColumnarBatchRow` Key: SPARK-40477 URL: https://issues.apache.org/jira/browse/SPARK-40477 Project: Spark Issue Type

[jira] [Resolved] (SPARK-40195) Add PrunedScanWithAQESuite

[ https://issues.apache.org/jira/browse/SPARK-40195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura resolved SPARK-40195. --- Resolution: Invalid I just realized the suite is not for AQE, so closing > Add Prun

[jira] [Created] (SPARK-40195) Add PrunedScanWithAQESuite

Kazuyuki Tanimura created SPARK-40195: - Summary: Add PrunedScanWithAQESuite Key: SPARK-40195 URL: https://issues.apache.org/jira/browse/SPARK-40195 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-40110) Add JDBCWithAQESuite

Kazuyuki Tanimura created SPARK-40110: - Summary: Add JDBCWithAQESuite Key: SPARK-40110 URL: https://issues.apache.org/jira/browse/SPARK-40110 Project: Spark Issue Type: Test Com

[jira] [Created] (SPARK-40088) Add SparkPlanWIthAQESuite

Kazuyuki Tanimura created SPARK-40088: - Summary: Add SparkPlanWIthAQESuite Key: SPARK-40088 URL: https://issues.apache.org/jira/browse/SPARK-40088 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-40049) Add adaptive plan case in ReplaceNullWithFalseInPredicateEndToEndSuite

Kazuyuki Tanimura created SPARK-40049: - Summary: Add adaptive plan case in ReplaceNullWithFalseInPredicateEndToEndSuite Key: SPARK-40049 URL: https://issues.apache.org/jira/browse/SPARK-40049 Proj

[jira] [Updated] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

Title: Message Title Kazuyuki Tanimura upd

[jira] [Commented] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

Title: Message Title Kazuyuki Tanimura com

[jira] [Updated] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

[ https://issues.apache.org/jira/browse/SPARK-39584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-39584: -- Description: GenTPCDSData uses the schema defined in `TPCDSSchema` that contains varc

[jira] [Updated] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

[ https://issues.apache.org/jira/browse/SPARK-39584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-39584: -- Description: GenTPCDSData uses the schema defined in `TPCDSSchema` that contains varc

[jira] [Updated] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

[ https://issues.apache.org/jira/browse/SPARK-39584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-39584: -- Description: GenTPCDSData uses the schema defined in `TPCDSSchema` that contains varc

[jira] [Commented] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

[ https://issues.apache.org/jira/browse/SPARK-39584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558678#comment-17558678 ] Kazuyuki Tanimura commented on SPARK-39584: --- Hi [~maropu] , pinging you since

[jira] [Created] (SPARK-39584) Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results

Kazuyuki Tanimura created SPARK-39584: - Summary: Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results Key: SPARK-39584 URL: https://issues.apache.org/jira/browse/SPARK-39584 Projec

[jira] [Updated] (SPARK-38573) Support Auto Partition Statistics Collection

[ https://issues.apache.org/jira/browse/SPARK-38573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-38573: -- Summary: Support Auto Partition Statistics Collection (was: Support Auto Partition Le

[jira] [Created] (SPARK-38786) Test Bug in StatisticsSuite "change stats after add/drop partition command"

Kazuyuki Tanimura created SPARK-38786: - Summary: Test Bug in StatisticsSuite "change stats after add/drop partition command" Key: SPARK-38786 URL: https://issues.apache.org/jira/browse/SPARK-38786

[jira] [Updated] (SPARK-38573) Support Auto Partition Level Statistics Collection

[ https://issues.apache.org/jira/browse/SPARK-38573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-38573: -- Summary: Support Auto Partition Level Statistics Collection (was: Support Partition L

[jira] [Updated] (SPARK-38573) Support Partition Level Statistics Collection

[ https://issues.apache.org/jira/browse/SPARK-38573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-38573: -- Affects Version/s: 3.4.0 (was: 3.3.0) > Support Partition L

[jira] [Created] (SPARK-38573) Support Partition Level Statistics Collection

Kazuyuki Tanimura created SPARK-38573: - Summary: Support Partition Level Statistics Collection Key: SPARK-38573 URL: https://issues.apache.org/jira/browse/SPARK-38573 Project: Spark Issue

[jira] [Created] (SPARK-38142) Move ArrowColumnVectorSuite to org.apache.spark.sql.vectorized

Kazuyuki Tanimura created SPARK-38142: - Summary: Move ArrowColumnVectorSuite to org.apache.spark.sql.vectorized Key: SPARK-38142 URL: https://issues.apache.org/jira/browse/SPARK-38142 Project: Spa

[jira] [Commented] (SPARK-38142) Move ArrowColumnVectorSuite to org.apache.spark.sql.vectorized

[ https://issues.apache.org/jira/browse/SPARK-38142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489101#comment-17489101 ] Kazuyuki Tanimura commented on SPARK-38142: --- on it > Move ArrowColumnVectorSu

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488506#comment-17488506 ] Kazuyuki Tanimura commented on SPARK-36665: --- [~aokolnychyi] issue resolved. >

[jira] [Created] (SPARK-38132) Remove NotPropagation

Kazuyuki Tanimura created SPARK-38132: - Summary: Remove NotPropagation Key: SPARK-38132 URL: https://issues.apache.org/jira/browse/SPARK-38132 Project: Spark Issue Type: Bug Com

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487222#comment-17487222 ] Kazuyuki Tanimura commented on SPARK-36665: --- Understood, thank you [~aokolnych

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486848#comment-17486848 ] Kazuyuki Tanimura commented on SPARK-36665: --- I saw the test case at [https://g

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486829#comment-17486829 ] Kazuyuki Tanimura commented on SPARK-36665: --- [~aokolnychyi] Thank you for brin

[jira] [Commented] (SPARK-38086) Make ArrowColumnVector Extendable

[ https://issues.apache.org/jira/browse/SPARK-38086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485530#comment-17485530 ] Kazuyuki Tanimura commented on SPARK-38086: --- I am working on this > Make Arro

[jira] [Created] (SPARK-38086) Make ArrowColumnVector Extendable

Kazuyuki Tanimura created SPARK-38086: - Summary: Make ArrowColumnVector Extendable Key: SPARK-38086 URL: https://issues.apache.org/jira/browse/SPARK-38086 Project: Spark Issue Type: Impro

[jira] [Commented] (SPARK-35867) Enable vectorized read for VectorizedPlainValuesReader.readBooleans

[ https://issues.apache.org/jira/browse/SPARK-35867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440813#comment-17440813 ] Kazuyuki Tanimura commented on SPARK-35867: --- I am working on this > Enable ve

[jira] [Commented] (SPARK-36721) Simplify boolean equalities if one side is literal

[ https://issues.apache.org/jira/browse/SPARK-36721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413404#comment-17413404 ] Kazuyuki Tanimura commented on SPARK-36721: --- I am working on this > Simplify

[jira] [Created] (SPARK-36721) Simplify boolean equalities if one side is literal

Kazuyuki Tanimura created SPARK-36721: - Summary: Simplify boolean equalities if one side is literal Key: SPARK-36721 URL: https://issues.apache.org/jira/browse/SPARK-36721 Project: Spark

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409746#comment-17409746 ] Kazuyuki Tanimura commented on SPARK-36665: --- I am working on this > Add more

[jira] [Created] (SPARK-36665) Add more Not operator optimizations

Kazuyuki Tanimura created SPARK-36665: - Summary: Add more Not operator optimizations Key: SPARK-36665 URL: https://issues.apache.org/jira/browse/SPARK-36665 Project: Spark Issue Type: Imp

[jira] [Created] (SPARK-36644) Push down boolean column filter

Kazuyuki Tanimura created SPARK-36644: - Summary: Push down boolean column filter Key: SPARK-36644 URL: https://issues.apache.org/jira/browse/SPARK-36644 Project: Spark Issue Type: Improve

[jira] [Commented] (SPARK-36644) Push down boolean column filter

[ https://issues.apache.org/jira/browse/SPARK-36644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408348#comment-17408348 ] Kazuyuki Tanimura commented on SPARK-36644: --- I am working on this issue > Pus

[jira] [Created] (SPARK-36607) Support BooleanType in UnwrapCastInBinaryComparison

Kazuyuki Tanimura created SPARK-36607: - Summary: Support BooleanType in UnwrapCastInBinaryComparison Key: SPARK-36607 URL: https://issues.apache.org/jira/browse/SPARK-36607 Project: Spark

[jira] [Updated] (SPARK-32210) Failed to serialize large MapStatuses

[ https://issues.apache.org/jira/browse/SPARK-32210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-32210: -- Affects Version/s: 3.3.0 2.4.8 3.0.3 > F

[jira] [Updated] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-36464: -- Description: The `size` method of `ChunkedByteBufferOutputStream` returns a `Long` val

[jira] [Updated] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-36464: -- Description: The `size` method of `ChunkedByteBufferOutputStream` returns a `Long` val

[jira] [Updated] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuyuki Tanimura updated SPARK-36464: -- Description: The `size` method of `ChunkedByteBufferOutputStream` returns a `Long` val

[jira] [Created] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

Kazuyuki Tanimura created SPARK-36464: - Summary: Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data Key: SPARK-36464 URL: https://issues.apache.org/jira/brow