This is an automated email from the ASF dual-hosted git repository.
richox pushed a change to branch dev-spark-auron-conf
in repository https://gitbox.apache.org/repos/asf/auron.git
discard efa72d5f Refactor SparkAuronConfiguration and remove deprecated
AuronConf classes
discard 778d8f7f deprecate AuronConf
omit 29ae9b33 update spark-3.5 golden files because aggregate buffer schema
is changed
omit 3413f9e4 Fix cargo clippy error
omit 88bd19f8 Reformat codes
omit acd22ce0 Implement columnar aggregate buffers
add 6a833df1 [AURON #1944] fix(clippy): fix panic error in
spark_partition_id test (#1945)
add 87de6be3 [AURON #1917] Clean up Clippy format/string lints (#1948)
add e6944f73 [AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet
related tests (#1954)
add 5ea773cc [AURON #1937][CI] Enable TPCDS test q14b (#1946)
add 9bbf9627 [AURON #1920] Clean up Clippy condition/logic lints (#1960)
add bab42102 [AURON #1942] Implement columnar aggregate buffers (#1943)
add 814d4ad9 [AURON #1778] Add
spark.auron.expression.singleChildFallback.enabled config (#1779)
add 29766fa0 [AURON #1769][CI] Improve auron-it build (#1932)
add b899b4da [AURON #1875] Minor refactor: move legacy spark version
compatibility methods to Shims.scala (#1876)
add 37509c71 [AURON #1739] Support LIMIT with OFFSET (#1740)
add 886e1ee3 [AURON #1757] Add cleanup logic to prevent flaky tests caused
by leftover locations (#1758)
add 295f41e0 [AURON #1940] Extract Rust test logic to standalone CI
workflow (#1941)
add 4964053c [AURON #1870][BUILD] Add `-Xfatal-warnings` to
scala-maven-plugin and fix related compilation failures (#1872)
new 83ccab8a Refactor SparkAuronConfiguration and remove deprecated
AuronConf classes
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (efa72d5f)
\
N -- N -- N refs/heads/dev-spark-auron-conf (83ccab8a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/{paimon.yml => rust-test.yml} | 57 +++++-----
.github/workflows/tpcds-reusable.yml | 48 ++++-----
Cargo.toml | 13 ---
.../org/apache/spark/sql/SparkQueryTestsBase.scala | 2 +-
.../org/apache/spark/sql/AuronDataFrameSuite.scala | 8 +-
.../AuronParquetAvroCompatibilitySuite.scala | 10 +-
.../parquet/AuronParquetColumnIndexSuite.scala | 8 +-
.../parquet/AuronParquetCompatibilityTest.scala | 8 +-
...onParquetCompressionCodecPrecedenceSuite.scala} | 14 +--
.../parquet/AuronParquetEncodingSuite.scala | 8 +-
.../parquet/AuronParquetFieldIdIOSuite.scala | 8 +-
.../parquet/AuronParquetFieldIdSchemaSuite.scala | 10 +-
.../parquet/AuronParquetFileFormatSuite.scala | 8 +-
.../parquet/AuronParquetFileFormatV1Suite.scala | 8 +-
.../parquet/AuronParquetFileFormatV2Suite.scala | 8 +-
.../datasources/parquet/AuronParquetIOSuite.scala} | 6 +-
.../AuronParquetInteroperabilitySuite.scala | 10 +-
.../AuronParquetPartitionDiscoverySuite.scala | 10 +-
.../AuronParquetProtobufCompatibilitySuite.scala | 10 +-
.../parquet/AuronParquetQuerySuite.scala} | 6 +-
.../parquet/AuronParquetRebaseDatetimeSuite.scala | 10 +-
.../AuronParquetRebaseDatetimeV1Suite.scala | 10 +-
.../AuronParquetRebaseDatetimeV2Suite.scala | 10 +-
.../parquet/AuronParquetSchemaInferenceSuite.scala | 10 +-
.../parquet/AuronParquetSchemaPruningSuite.scala | 8 +-
.../parquet/AuronParquetSchemaSuite.scala | 8 +-
.../datasources/parquet/AuronParquetTest.scala} | 6 +-
.../AuronParquetThriftCompatibilitySuite.scala | 10 +-
.../parquet/AuronParquetV1FilterSuite.scala | 8 +-
.../AuronParquetV1PartitionDiscoverySuite.scala | 10 +-
.../parquet/AuronParquetV1QuerySuite.scala} | 6 +-
.../parquet/AuronParquetV1SchemaPruningSuite.scala | 10 +-
.../parquet/AuronParquetV2FilterSuite.scala | 8 +-
.../AuronParquetV2PartitionDiscoverySuite.scala | 10 +-
.../parquet/AuronParquetV2QuerySuite.scala} | 6 +-
.../parquet/AuronParquetV2SchemaPruningSuite.scala | 10 +-
.../parquet/AuronParquetVectorizedSuite.scala | 8 +-
.../execution/ui/AuronSQLAppStatusListener.scala | 3 +
dev/auron-it/pom.xml | 2 +-
dev/auron-it/run-it.sh | 2 +-
.../tpcds-plan-stability/spark-3.5/q1.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q10.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q14a.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q14b.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q15.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q18.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q21.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q22.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q23b.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q26.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q27.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q3.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q30.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q33.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q35.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q36.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q40.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q41.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q42.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q43.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q44.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q45.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q47.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q49.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q5.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q50.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q52.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q55.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q56.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q57.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q59.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q6.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q60.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q62.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q66.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q67.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q69.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q7.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q70.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q72.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q76.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q77.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q78.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q79.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q8.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q80.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q81.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q82.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q83.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q84.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q85.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q86.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q93.txt | 2 +-
.../tpcds-plan-stability/spark-3.5/q99.txt | 2 +-
.../comparison/PlanStabilityChecker.scala | 1 +
dev/mvn-build-helper/build-native.sh | 10 +-
native-engine/auron-planner/proto/auron.proto | 6 +-
native-engine/auron-planner/src/planner.rs | 22 ++--
.../datafusion-ext-functions/src/spark_bround.rs | 14 +--
.../datafusion-ext-functions/src/spark_strings.rs | 45 ++++----
.../datafusion-ext-plans/src/agg/agg_table.rs | 17 +--
native-engine/datafusion-ext-plans/src/agg_exec.rs | 4 +-
.../src/common/cached_exprs_evaluator.rs | 10 +-
.../src/common/execution_context.rs | 17 ++-
.../src/common/row_null_checker.rs | 14 +--
.../src/joins/bhj/semi_join.rs | 8 +-
.../datafusion-ext-plans/src/limit_exec.rs | 99 ++++++++++++++---
native-engine/datafusion-ext-plans/src/orc_exec.rs | 107 ++++++++----------
.../datafusion-ext-plans/src/parquet_sink_exec.rs | 4 +-
.../datafusion-ext-plans/src/shuffle/mod.rs | 2 +-
.../src/shuffle/single_repartitioner.rs | 2 +-
.../datafusion-ext-plans/src/sort_exec.rs | 101 ++++++++++++++---
pom.xml | 17 +++
.../sql/auron/InterceptedValidateSparkPlan.scala | 3 +
.../org/apache/spark/sql/auron/ShimsImpl.scala | 72 ++++++++++---
.../auron/plan/NativeCollectLimitExec.scala | 4 +-
.../auron/plan/NativeGlobalLimitExec.scala | 4 +-
.../auron/plan/NativeLocalLimitExec.scala | 2 +-
.../auron/plan/NativePartialTakeOrderedExec.scala | 2 +-
.../auron/plan/NativeTakeOrderedExec.scala | 5 +-
.../shuffle/AuronBlockStoreShuffleReader.scala | 3 +
.../auron/shuffle/AuronRssShuffleManagerBase.scala | 3 +
.../plan/NativeShuffledHashJoinExecProvider.scala | 4 +
.../org/apache/auron/AuronFunctionSuite.scala | 4 -
.../scala/org/apache/auron/AuronQuerySuite.scala | 15 ---
.../scala/org/apache/auron/BaseAuronSQLSuite.scala | 29 ++++-
.../org/apache/auron/exec/AuronExecSuite.scala | 95 +++++++++++++++-
.../sql/execution/BuildInfoInSparkUISuite.scala | 4 +-
.../configuration/SparkAuronConfiguration.java | 7 +-
.../SparkAuronConfigurationDocGenerator.java | 21 ++--
.../apache/spark/sql/auron/AuronConverters.scala | 120 +++++++--------------
.../sql/auron/AuronSparkSessionExtension.scala | 4 +-
.../apache/spark/sql/auron/NativeConverters.scala | 62 ++++++-----
.../org/apache/spark/sql/auron/NativeHelper.scala | 4 +-
.../scala/org/apache/spark/sql/auron/Shims.scala | 38 +++++--
.../spark/sql/auron/util/TaskContextHelper.scala | 2 +-
.../auron/columnar/AuronArrowColumnVector.scala | 3 +
.../auron/plan/NativeCollectLimitBase.scala | 6 +-
.../auron/plan/NativeGlobalLimitBase.scala | 3 +-
.../auron/plan/NativeLocalLimitBase.scala | 2 +-
.../NativeParquetInsertIntoHiveTableBase.scala | 7 +-
.../auron/plan/NativeParquetSinkBase.scala | 6 +-
.../auron/plan/NativeShuffleExchangeBase.scala | 2 +
.../auron/plan/NativeShuffledHashJoinBase.scala | 2 +
.../auron/plan/NativeTakeOrderedBase.scala | 59 +++++-----
.../sql/execution/auron/plan/NativeUnionBase.scala | 4 +-
.../shuffle/AuronBlockStoreShuffleReaderBase.scala | 2 +-
.../src/main/scala/org/apache/auron/sparkver.scala | 6 +-
.../uniffle/AuronUniffleShuffleReader.scala | 3 +
149 files changed, 949 insertions(+), 686 deletions(-)
copy .github/workflows/{paimon.yml => rust-test.yml} (53%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala
(79%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala
(79%)
copy
auron-spark-tests/{common/src/test/scala/org/apache/auron/utils/SQLQueryTestSettings.scala
=>
spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala}
(75%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala
(80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala
(80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala
(77%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala
(80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala
(79%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala
(79%)
copy
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/{AuronStringFunctionsSuite.scala
=> execution/datasources/parquet/AuronParquetIOSuite.scala} (81%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala
(77%)
copy
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/{AuronStringFunctionsSuite.scala
=> execution/datasources/parquet/AuronParquetQuerySuite.scala} (80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala
(77%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala
(79%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala
(79%)
copy
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/{AuronStringFunctionsSuite.scala
=> execution/datasources/parquet/AuronParquetTest.scala} (80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala
(80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala
(77%)
copy
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/{AuronStringFunctionsSuite.scala
=> execution/datasources/parquet/AuronParquetV1QuerySuite.scala} (80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala
(80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala
(77%)
copy
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/{AuronStringFunctionsSuite.scala
=> execution/datasources/parquet/AuronParquetV2QuerySuite.scala} (80%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala
(78%)
copy common/src/main/templates/org/apache/auron/common/ProjectConstants.java
=>
auron-spark-tests/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala
(80%)
