This is an automated email from the ASF dual-hosted git repository.
rui-mo pushed a change to branch ts_ntz_dev
in repository https://gitbox.apache.org/repos/asf/gluten.git
omit e497b43804 [GLUTEN-11622][VL] Add basic TIMESTAMP_NTZ type support
(#11939)
add 02bf50bb4c [VL] Optimize bhj hash table built performance (#11927)
add 0d0890ff45 [MINOR] Refactor: move ClickHouse-specific extension tests
to backend module (#11926)
add edcb509331 [CORE] Refactor common expression mappings out of Spark
shims (#11941)
add c46dc08f3f [MINOR][DOCS] Fix several nit issues in docs (#11936)
add 835172a952 [VL] Find the aggregate function name by matching result
type (#11931)
add ffbf953d54 [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_15)
(#11945)
add 41a6fd83c1 [GLUTEN-10215][VL] Delta 4 UT: sync test infra from Delta
3.3 suites (#11938)
add f384947c88 [VL] Delta: Unify code style with Gluten codebase (#11596)
add 6c305d5ff7 [GLUTEN-11906][VL] Support dynamic build for ppc64le
(#11907)
add dc2f9f2379 Bump pillow in /tools/workload/benchmark_velox/analysis
(#11940)
add f9c090abdf [VL][UT] Enable two tests previously failing due to Velox
Expand operator bug (#11947)
add 7e35388d3f [VL] Ignore null values when copy velox string buffer to
arrow (#11944)
add 139c332edc [GLUTEN-9456][VL] Override clone in
GlutenDirectBufferedInput (#11933)
add a7058c7110 [GLUTEN-11862][VL] Work around GMT session timezone
validation failure on macOS (#11869)
add 6988ab6d10 [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_16)
(#11949)
add 69b319dc65 [MINOR] Split Iceberg test resources into separate Maven
entries (#11952)
add 584d1b52a5 [VL] Fix shallow CreateNamedStruct detection in
CollapseProjectExecTransformer (#11953)
add f805f13733 [VL] shuffle: TypeAwareCompress(tac) for column-wise data
compression like (U)INT64 (#11894)
add 7b83b92964 [MINOR][DOCS] Fix typos in VeloxConfig.scala doc strings
(#11950)
add b8a6e906c2 [CORE] Add .claude/ to .gitignore (#11962)
add 52dd67cf16 Add storageReads metric to track number of storage read
operations (#11964)
add f74ca6f0d8 [MINOR] Rename Spark resource and dependency installation
scripts (#11937)
add b9fe9efda8 [GLUTEN-11958] Flatten pom for modules containing
spark-shim (#11959)
add 650f0c3026 [VL] Experimental eager aggregation (#11942)
add 98d816874b [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_17)
(#11956)
add fb23d83fa2 [GLUTEN-11895][VL] Fix SIGSEGV on IOThreadPool threads
during HDFS scan (#11896)
add 989ab773f9 [MINOR][VL] Fix install-spark-resources.sh: Spark 4.1 case
references wrong directory (#11973)
add 1ca995c17a [VL] Remove unused libprotobuf.so.32 from shared library
loaders and build script (#11974)
add 50632379b0 [CH] Fix package script issue (#11886)
add e23eba4091 [GLUTEN-10134][VL] Add ANSI mode CI baseline with
expression offload tracking (#11975)
add cb2f46b877 [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_23)
(#11979)
add d1a2a7fb6b [GLUTEN-10215][VL] Follow-up - Remove throw statement for
FetchFailedException (#11978)
add 6cdcfe88ed [GLUTEN-11969] Enable Java 17 release flag for Spark 4+
build (#11971)
add 7bf4467ec2 [GLUTEN-10215][VL] Delta 4 native write support for Spark
4.0 (#11924)
add bd79573cbd Move glog initialization before first LOG() call in
VeloxBackend::init (#11983)
add f0a3df8944 [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_24)
(#11982)
add 23d7ca19ee [VL] Support case-insensitive celeborn shuffle manager
detection (#11992)
add a568a1e9e2 [INFRA] Upgrade scalafmt to 3.10.1 (#11999)
add 4a018f6f12 [GLUTEN-11488][CH] Update Clickhouse Version
(Stable-25.12.10.7_20260425) (#11734)
add 33f2cc8649 [GLUTEN-11911][VL][Spark4] Enable Structured Streaming
suites with GlutenPlugin fallback (#11957)
add 93de3c8b41 [GLUTEN-12010][CH] Pass the correct values to
ParquetInputFormat (#12011)
add b1906e5f7b [GLUTEN-11635][FOLLOW-UP] Respect partial fallback configs
when checking node support (#11988)
add b6c278e1ad [VL][DELTA] Reconcile struct field names in CaseWhen/If for
Velox SWITCH compatibility (#11948)
add 20c77fcecb [VL] Added iceberg write configs (#11776)
add cf8bcb98b9 [GLUTEN-6887][VL] Daily Update Velox Version (2026_04_30)
(#12014)
add ffea382c93 [VL] Fix nightly upload action (#12006)
add 8726bf5232 [DOC] Update Velox Delta guide for Spark 4.0 / Delta 4.0
support (#12003)
add e1e6e6840a [CORE] Upgrade Paimon to 1.4.1 (#11996)
add 6b8c0dcaea [GLUTEN-8304][CORE] Fix incorrect path merging in
`CollapseGetJsonObjectExpressionRule` for non-GJO expressions (#12007)
add c8200ec5ee [VL] Rework option
`spark.gluten.sql.columnar.backend.velox.asyncTimeoutOnTaskStopping` (#11882)
add 28b0071ffa [GLUTEN-11885][VL] Respect custom VELOX_HOME in build info
generation (#11905)
add d8135abfe7 [GLUTEN-12018] Support parquet metadata cache for the
Parquet Reader V3 (#12019)
add 571e4792e5 [VL][CI] Fix clang-tidy CI failure (#12021)
add 35784e2155 [CI] Correct job names to align with tested Spark versions
(#11994)
add 752f58ad02 [GLUTEN-12032][CH] Reopen some queries test for the Spark
3.5 (#12033)
add 040d7894d2 [MINOR] Add Hadoop 3.4 profile (#11995)
add d904cb4f45 [VL] Fix str_to_map MAP_KEY_DEDUP_POLICY comparison for
Spark 4.1 compatibility (#12035)
add feb8278c91 [CORE][UI] Skip GlutenPlanFallbackEvent posting when gluten
is disabled (#12027)
add b22975d2b5 [GLUTEN-6887][VL] Daily Update Velox Version
(dft-2026_05_02) (#12023)
add dd16714827 [VL][Delta] Add roaring bitmap facilities for Delta DV
support (#12001)
add 1c4a27ef32 [MINOR][VL] Improve validation log messages for aggregate
function signature binding (#12031)
add ffbb64770d [MINOR] Ignore .idea/vcs.xml from git tracking (#12041)
add 74c9638ef6 [GLUTEN-12044][CH] Fix build on arm (#12046)
add 6d6a1afae3 [GLUTEN-1433][VL] Fix TimestampNTZ schema validation
(#12029)
add 82644d3ca3 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_06)
(#12043)
add 8d76cb81c9 [GLUTEN-11920] Keep columnar children for dual-mode parents
with row output (#12004)
add aa0bcc8d4d [GLUTEN-11918][VL] Fall back Cast when per-expression
timezone differs from session timezone (#12048)
add 4fa7bdc712 [VL] Delta: Offload Delta OPTIMIZE compaction command
transactions (#12024)
add 985186dbc7 [VL][MINOR] Remove unused bloom filter configs (#12002)
add c3165c1c4c [VL] Fix json_tuple rewrite producing incompatible JSON
path in fallback scenarios (#12038)
add eae60a5b4d [CORE] Preserve fallback tag for nodes without logicalLink
in RemoveFallbackTagRule (#12028)
add 7e04fae42e [GLUTEN-12053][CELEBORN] Bump Celeborn version to 0.6.3
(#12057)
add 5b7955e9ef Bump gitpython in /tools/workload/benchmark_velox/analysis
(#12037)
add 6298aebe0e [GLUTEN-11782][CORE] Optimize parquet metadata validation
by sampling root paths (#12042)
add 0342cdb222 [CORE] Deduplicate fallback reason when merging Appendable
tags (#12049)
add 1f58e544e9 [VL] Fix parallel BHJ build after #11882 (#12070)
add ea4d893fa3 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_11)
(#12068)
add 7669052792 [GLUTEN-12075][CH] Add a metric to indicate whether it is
Parquet Reader V3 for the CH Backend (#12076)
add 296fd3cb2a [MINOR][DOC] Add AGENTS.md with guidelines for AI coding
agents (#12081)
add 6a23fe5c51 [GLUTEN-6887][VL] Daily Update Velox Version
(dft-2026_05_13) (#12085)
add 1cc35351c7 [GLUTEN-11867] Add JDK 25 build support (#12012)
add d35552e625 [VL] Fix SCOPED_TIMER macro destroying timer immediately
(#12083)
add 3bf094200b [GLUTEN-12071][VL] Respect HadoopConf write options in
Velox native Parquet writer (#12072)
add f725166d1a Bump mistune in /tools/workload/benchmark_velox/analysis
(#12093)
add 356fda012d [GLUTEN-10072][UT] Enable hive execution tests (#12005)
add 18390c3bff [GLUTEN-6887][VL] Daily Update Velox Version
(dft-2026_05_15) (#12096)
add 97632d8435 [VL] Add min/max partition stats to columnar
InMemoryRelation cache for partition pruning (#12092)
add ad03c2d991 [VL] Restore hash shuffle reader payload merging (#12097)
add d2c6f382f6 [VL] Add per-batch input-encoding counter to
VeloxHashShuffleWriter (#12107)
add faad82cdeb [CORE] Fix Delta 4.0 Spark 4.1 package build (#12078)
add 0b1e5115e5 [VL] Reduce Velox hash shuffle partition buffer memory by
evicting large partitions after split (#12089)
add 71302c6392 [VL] Add Velox batch resizer copyRanges fast path (#12101)
add 9e6e8bf03c [VL] Gate non-binary collation StringType in cached batch
stats dispatch (#12112)
add 5286b8cf1e [MINOR][DOC] Fix and expand build/test instructions in
AGENTS.md (#12121)
add c69a7c28e1 [VL] Add cross-config / cross-build-cycle invariant tests
for ColumnarCachedBatchSerializer (#12124)
add e733332184 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_18)
(#12102)
add 7bb00d1632 [VL] Delta: Add native Delta DV reader support (#12040)
add d2b48f1ce9 [GLUTEN-11998][CORE] Fix incorrect modifiability status for
GlutenConfig entries (#12036)
add 60b9f568c8 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_22)
(#12128)
add 3bfac4d60f [GLUTEN-11991][VL] CI: Switch to gcc 13 images (#12118)
add d206c5e20e [MINOR][VL] Remove dead Arrow-CSV / Arrow-Dataset JVM code
paths (#12130)
add b8c5711d67 [VL] Add metric abandonedPartialAggregation (#12106)
add 01bae2b4c8 [BUILD] Upgrade Maven to 3.9.16 (#12059)
add cb9e5d7b24 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_25)
(#12133)
add 5efa2d65fc Bump org.apache.flink:flink-table-planner_2.12 in
/backends-clickhouse (#12134)
add f71350979b [VL] Add devcontainer configuration for Gluten Velox
Backend (#12125)
add 8b8d413c48 [Velox][Build][MIRROR] Skip sudo on macOS when staging
Velox Arrow patches (#12129)
add 2563a40ed5 [GLUTEN-3456][VL] Enable columnar table cache by default
and extend benchmark coverage (#12138)
add 14f268cf28 Bump lxml in /tools/workload/benchmark_velox/analysis
(#12139)
add b83ed55da1 Bump gitpython in /tools/workload/benchmark_velox/analysis
(#12140)
add b4425283ec [MINOR][TEST] Remove SparkFunctionStatistics (#12136)
add 6f508d2962 [MINOR][GLUTEN-12137] Remove unmaintained gluten-te folder
(#12149)
add d175c63220 [VL] Narrow down FlushableHashAggregateRule to only protect
against Spark code `AggUtils.planAggregateWithOneDistinct` (#12098)
add 3b59dc964b [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_27)
(#12150)
add ba8c13ebf4 Bump pip from 26.0 to 26.1 in
/tools/workload/benchmark_velox/analysis (#12144)
add cad164adc2 Bump nbconvert in /tools/workload/benchmark_velox/analysis
(#12145)
add 9bc830e693 [GLUTEN-12058][VL] Skip stage ResourceProfile adjustment
rule when off-heap memory is disabled (#12060)
add 7bfd45ca98 [VL] Stop using Input.available() to probe trailing markers
in CachedColumnarBatch (#12147)
add e6e0403ba4 [MINOR][DOC] Update default of
spark.gluten.sql.columnar.tableCache to true (#12153)
add b0ca60a5c8 [GLUTEN-8232][VL] Allow to enable dynamic openssl link in
VCPKG packaging (#11444)
add 4df6e9cd10 [VL][FOLLOWUP] Reduce Velox hash shuffle partition buffer
memory by evicting large partitions after split (#12156)
add 53dc27d7d2 [VL] Hoist per-partition constants out of
ColumnarCachedBatchSerializer.serialize hot path (#12166)
add 55eb0d348f [Velox] Fix Iceberg writer aggregate-init build failure on
macOS C++20 libc++ (#12115)
add 0cc376a359 [GLUTEN-11917][VL] Respect allowPrecisionLoss from
expression context in Spark 4.1 (#12110)
add 82b70aab98 [INFRA] Use hadoop 2.10.2 for uniffle test (#12111)
add 256ab3c0a9 [GLUTEN-6887][VL] Daily Update Velox Version (2026_05_28)
(#12162)
add 952c197da8 Bump pyopenssl in /tools/workload/benchmark_velox/analysis
(#12159)
add c1d7a309aa Bump idna from 3.10 to 3.15 in
/tools/workload/benchmark_velox/analysis (#12167)
add 33c61b9065 [VL] Compute real nullCount for flat-encoded unsupported
types in cached-batch partition stats (#12176)
add 09a629ace7 [GLUTEN-11980][CORE][TESTS] Add test cases for decimal-key
joins if either side of join exists native scan fallback to vanilla. (#12061)
add a80309f5d6 [GLUTEN-12181][CH] Fix issues for the Spark 3.5 + Delta
Delete Vector with the CH backend (#12182)
add b9865b2d4a [VL] Bump Iceberg/Hudi for Spark-4.1 package (#12168)
add a897401d86 [VL] Catch corrupt stats frames per-batch in
ColumnarCachedBatchSerializer (#12183)
add 0aa1620e9b [GLUTEN-12185][CH] Guard missing prepared set in parquet
index filter (#12186)
add d87c2b0c47 [GLUTEN-12172][CH] Fix group limit first array result
offset (#12173)
add ed8af63539 [GLUTEN-12169][CH] Fix str_to_map nullable input null-map
access (#12170)
add b8bb3b20dc [GLUTEN-12174][CH] Fix flatten nullable inner array row
handling (#12175)
add 2db6253b3c [VL] Simplify the logic in
AppendBatchResizeForShuffleInputAndOutput (#12087)
add ce6e16fe8f [GLUTEN-6887][VL] Daily Update Velox Version
(dft-2026_05_29) (#12189)
add 33be6fb8bf [VL][TEST] Pin framedSerializeWithStats cross-language wire
format (#12196)
add fb23c3ea51 [MINOR] Make ci-velox-buildstatic-centos-9.sh executable
(#12200)
add e8cb66b79c [GLUTEN-8232][VL] Follow-up: update docs for the vcpkg
dynamic OpenSSL feature (#12178)
add 0dabb55630 [VL] Tighten cache-stats wire-edge bound checks (#12224)
add b6cab2f484 [VL] Isolate /usr/local from native builds on macOS (#12105)
add 97f06b41d0 [VL][Delta] Add DV scan info extraction utility (#12197)
add 9c34b45b2b [GLUTEN-6887][VL] Daily Update Velox Version (2026_06_02)
(#12223)
new d5bbbf7448 [GLUTEN-11622][VL] Add basic TIMESTAMP_NTZ type support
(#11939)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e497b43804)
\
N -- N -- N refs/heads/ts_ntz_dev (d5bbbf7448)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.devcontainer/devcontainer.json | 5 +
.github/skills/ansi-analysis/SKILL.md | 95 +
.github/skills/ansi-analysis/analyze-ansi.py | 602 ++++
.github/skills/ansi-analysis/shared.md | 115 +
.github/workflows/build_bundle_package.yml | 3 +-
.github/workflows/build_release.yml | 4 +-
.github/workflows/cpp_clang_tidy.yml | 4 +
.github/workflows/docker_image.yml | 3 +-
.github/workflows/flink.yml | 2 +-
...ll-flink-resources.sh => install-flink-deps.sh} | 0
...{install-resources.sh => install-spark-deps.sh} | 112 +-
...all-resources.sh => install-spark-resources.sh} | 138 +-
.github/workflows/velox_backend_ansi.yml | 612 ++++
.github/workflows/velox_backend_arm.yml | 9 +-
.github/workflows/velox_backend_cache.yml | 22 +-
.github/workflows/velox_backend_enhanced.yml | 29 +-
.github/workflows/velox_backend_x86.yml | 138 +-
.github/workflows/velox_nightly.yml | 25 +-
.gitignore | 4 +-
.scalafmt.conf | 12 +-
AGENTS.md | 123 +
CONTRIBUTING.md | 2 -
backends-clickhouse/pom.xml | 2 +-
.../GlutenDeltaMergeTreeDeletionVectorSuite.scala | 293 --
.../GlutenDeltaParquetDeletionVectorSuite.scala | 39 +-
.../execution/iceberg/ClickHouseIcebergSuite.scala | 4 +-
.../kafka/ClickhouseGlutenKafkaScanSuite.scala | 3 +-
.../org/apache/gluten/metrics/MetricsStep.java | 34 +
.../gluten/backendsapi/clickhouse/CHBackend.scala | 1 +
.../backendsapi/clickhouse/CHListenerApi.scala | 3 +-
.../backendsapi/clickhouse/CHMetricsApi.scala | 11 +-
.../gluten/backendsapi/clickhouse/CHRuleApi.scala | 1 +
.../gluten/execution/CHRangeExecTransformer.scala | 2 +-
.../extension/AddPreProjectionForHashJoin.scala | 2 +-
.../extension/BasicExpressionRewriteRule.scala | 2 +-
.../CommonSubexpressionEliminateRule.scala | 2 +-
...PushdownAggregatePreProjectionAheadExpand.scala | 2 +-
.../metrics/FileSourceScanMetricsUpdater.scala | 6 +
.../sql/execution/CHColumnarWriteFilesExec.scala | 4 +-
.../gluten/benchmarks/GenTPCDSTableScripts.scala | 2 +-
...nClickHouseColumnarMemorySortShuffleSuite.scala | 3 +
.../GlutenClickHouseColumnarShuffleAQESuite.scala | 3 +
.../GlutenClickHouseDSV2ColumnarShuffleSuite.scala | 3 +
.../execution/GlutenClickHouseDSV2Suite.scala | 3 +
.../GlutenClickHouseDeltaParquetWriteSuite.scala | 106 +-
.../GlutenClickHouseTPCHAbstractSuite.scala | 3 +
.../GlutenClickHouseTPCHBucketSuite.scala | 3 +
...GlutenClickHouseTPCHNotNullSkipIndexSuite.scala | 3 +
...lickHouseTPCHNullableColumnarShuffleSuite.scala | 3 +
...lutenClickHouseTPCHNullableSkipIndexSuite.scala | 3 +
.../GlutenClickHouseTPCHNullableSuite.scala | 3 +
.../execution/GlutenClickHouseTPCHSuite.scala | 7 +-
.../execution/GlutenFunctionValidateSuite.scala | 167 +-
.../gluten/execution/GlutenNothingValueCheck.scala | 8 +-
.../GlutenClickhouseFunctionSuite.scala | 38 +-
.../hive/GlutenClickHouseHiveTableSuite.scala | 32 +-
.../GlutenClickHouseNativeWriteTableSuite.scala | 5 +-
.../GlutenClickHouseMergeTreeCacheDataSuite.scala | 3 +
.../GlutenClickHouseMergeTreeOptimizeSuite.scala | 38 +-
...tenClickHouseMergeTreePathBasedWriteSuite.scala | 2 +
.../GlutenClickHouseMergeTreeTPCHSuite.scala | 4 +
...GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala | 3 +
...eMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala | 3 +
.../GlutenClickHouseMergeTreeWriteOnS3Suite.scala | 3 +
.../GlutenClickHouseMergeTreeWriteStatsSuite.scala | 3 +
.../GlutenClickHouseMergeTreeWriteSuite.scala | 2 +
...useMergeTreeWriteTaskNotSerializableSuite.scala | 3 +
...lutenClickhouseMergetreeSoftAffinitySuite.scala | 2 +
.../GlutenClickHouseMergeTreeMetricsSuite.scala | 3 +
.../GlutenClickHouseTPCDSMetricsSuite.scala | 4 +-
...ckHouseTPCHColumnarShuffleParquetAQESuite.scala | 29 +
...enClickHouseTPCHParquetAQEConcurrentSuite.scala | 3 +-
.../GlutenClickHouseTPCHParquetBucketSuite.scala | 4 +-
...nClickHouseTPCHSaltNullNativeParquetSuite.scala | 4 -
.../GlutenClickHouseTPCHSaltNullParquetSuite.scala | 42 +-
.../scala/org/apache/gluten/s3/S3AuthSuite.scala | 2 +-
.../extension/GlutenCustomerExtensionSuite.scala | 152 +
.../ColumnarTableCacheBenchmark-results.txt | 101 +-
backends-velox/pom.xml | 2 +-
.../VeloxCelebornColumnarBatchSerializer.scala | 8 +-
.../VeloxCelebornColumnarShuffleWriter.scala | 1 +
.../gluten/delta/DeltaDeletionVectorScanInfo.scala | 207 ++
.../sql/delta/GlutenDeltaParquetFileFormat.scala | 452 +--
.../sql/delta/GlutenOptimisticTransaction.scala | 26 +-
.../delta/files/GlutenDeltaFileFormatWriter.scala | 207 +-
.../perf/GlutenDeltaOptimizedWriterExec.scala | 1 -
.../delta/stats/GlutenDeltaJobStatsTracker.scala | 2 +-
.../datasources/v2/DeltaWriteOperators.scala | 19 +-
.../datasources/v2/OffloadDeltaCommand.scala | 22 +-
.../delta/DeltaDeletionVectorScanInfoSuite.scala | 154 +
.../apache/spark/sql/delta/DeleteSQLSuite.scala | 64 +-
.../apache/spark/sql/delta/DeleteSuiteBase.scala | 429 +--
.../spark/sql/delta/DeletionVectorsTestUtils.scala | 153 +-
.../sql/delta/DeltaColumnMappingTestUtils.scala | 337 +-
.../org/apache/spark/sql/delta/DeltaDDLSuite.scala | 3 +-
...DeltaExcludedBySparkVersionTestMixinShims.scala | 15 +-
.../spark/sql/delta/DeltaNativeWriteSuite.scala | 287 ++
.../apache/spark/sql/delta/DeltaTestUtils.scala | 234 +-
.../apache/spark/sql/delta/UpdateSuiteBase.scala | 3 +-
.../test/DeltaColumnMappingSelectedTestMixin.scala | 37 +-
.../sql/delta/test/DeltaExcludedTestMixin.scala | 7 +-
.../spark/sql/delta/test/DeltaSQLCommandTest.scala | 12 +-
.../spark/sql/delta/test/DeltaSQLTestUtils.scala | 18 +-
.../spark/sql/delta/test/DeltaTestImplicits.scala | 55 +-
...che.gluten.component.VeloxDelta40WriteComponent | 0
.../component/VeloxDelta40WriteComponent.scala | 43 +
.../gluten/delta/DeltaDeletionVectorScanInfo.scala | 217 ++
.../sql/delta/GlutenDeltaParquetFileFormat.scala | 13 +-
.../sql/delta/GlutenOptimisticTransaction.scala | 28 +-
.../spark/sql/delta/GlutenParquetFileFormat.scala | 1 +
.../delta/files/GlutenDeltaFileFormatWriter.scala | 17 +-
.../perf/GlutenDeltaOptimizedWriterExec.scala | 1 +
.../stats/GlutenDeltaJobStatsFallbackTracker.scala | 1 +
.../GlutenDeltaJobStatsRowCountingTracker.scala | 1 +
.../delta/stats/GlutenDeltaJobStatsTracker.scala | 18 +-
.../datasources/v2/DeltaWriteOperators.scala | 1 +
.../datasources/v2/OffloadDeltaCommand.scala | 1 +
.../delta/DeltaDeletionVectorScanInfoSuite.scala | 154 +
.../apache/spark/sql/delta/DeleteSQLSuite.scala | 0
.../apache/spark/sql/delta/DeleteSuiteBase.scala | 0
.../spark/sql/delta/DeletionVectorsTestUtils.scala | 0
.../sql/delta/DeltaColumnMappingTestUtils.scala | 0
.../org/apache/spark/sql/delta/DeltaDDLSuite.scala | 1 +
...DeltaExcludedBySparkVersionTestMixinShims.scala | 0
.../sql/delta/DeltaInsertIntoTableSuite.scala | 1 +
.../delta/DeltaInsertIntoTableSuiteShims.scala} | 24 +-
.../apache/spark/sql/delta/DeltaMetricsUtils.scala | 1 +
.../spark/sql/delta/DeltaNativeWriteSuite.scala | 517 +++
.../org/apache/spark/sql/delta/DeltaSuite.scala | 3503 ++++++++++++++++++++
.../apache/spark/sql/delta/DeltaSuiteShims.scala | 1 +
.../apache/spark/sql/delta/DeltaTestUtils.scala | 0
.../apache/spark/sql/delta/FakeFileSystem.scala | 1 +
.../apache/spark/sql/delta/UpdateSQLSuite.scala | 1 +
.../apache/spark/sql/delta/UpdateSuiteBase.scala | 1 +
.../clustering/ClusteredTableClusteringSuite.scala | 1 +
.../CoordinatedCommitsTestUtils.scala | 1 +
.../perf/OptimizeMetadataOnlyDeltaQuerySuite.scala | 1 +
.../sql/delta/perf/OptimizedWritesSuite.scala | 1 +
.../delta/skipping/ClusteredTableTestUtils.scala | 1 +
.../test/DeltaColumnMappingSelectedTestMixin.scala | 0
.../sql/delta/test/DeltaExcludedTestMixin.scala | 0
.../spark/sql/delta/test/DeltaSQLCommandTest.scala | 3 +
.../spark/sql/delta/test/DeltaSQLTestUtils.scala | 0
.../spark/sql/delta/test/DeltaTestImplicits.scala | 0
.../spark/sql/delta/test/ScanReportHelper.scala | 1 +
.../connector/write/IcebergDataWriteFactory.scala | 15 +-
.../execution/AbstractIcebergWriteExec.scala | 18 +
.../writer/VeloxUniffleColumnarShuffleWriter.java | 1 +
.../java/org/apache/gluten/metrics/Metrics.java | 8 +
.../org/apache/gluten/metrics/OperatorMetrics.java | 6 +
.../org/apache/gluten/utils/VeloxBatchResizer.java | 2 +
.../gluten/utils/VeloxBatchResizerJniWrapper.java | 1 +
.../apache/gluten/vectorized/HashJoinBuilder.java | 4 +-
.../gluten/backendsapi/velox/VeloxBackend.scala | 22 +-
.../backendsapi/velox/VeloxIteratorApi.scala | 2 +-
.../gluten/backendsapi/velox/VeloxMetricsApi.scala | 6 +
.../gluten/backendsapi/velox/VeloxRuleApi.scala | 5 +-
.../backendsapi/velox/VeloxSparkPlanExecApi.scala | 12 +-
.../backendsapi/velox/VeloxTransformerApi.scala | 13 +-
.../backendsapi/velox/VeloxValidatorApi.scala | 4 +-
.../org/apache/gluten/config/VeloxConfig.scala | 76 +-
.../gluten/datasource/ArrowCSVFileFormat.scala | 379 ---
.../datasource/ArrowCSVOptionConverter.scala | 62 -
.../v2/ArrowCSVPartitionReaderFactory.scala | 176 -
.../apache/gluten/datasource/v2/ArrowCSVScan.scala | 76 -
.../gluten/datasource/v2/ArrowCSVScanBuilder.scala | 44 -
.../gluten/datasource/v2/ArrowCSVTable.scala | 80 -
.../execution/ColumnarPartialGenerateExec.scala | 4 +-
.../execution/ColumnarPartialProjectExec.scala | 4 +-
.../gluten/execution/GenerateExecTransformer.scala | 8 +-
.../gluten/execution/HashJoinExecTransformer.scala | 20 +
.../gluten/execution/VeloxResizeBatchesExec.scala | 8 +-
.../datasource/v2/ArrowBatchScanExec.scala | 46 -
...AppendBatchResizeForShuffleInputAndOutput.scala | 108 +-
.../gluten/extension/ArrowConvertorRule.scala | 120 -
.../gluten/extension/ArrowScanReplaceRule.scala | 39 -
.../extension/FlushableHashAggregateRule.scala | 136 +-
.../apache/gluten/extension/PartialFallback.scala | 9 +-
.../gluten/metrics/BatchScanMetricsUpdater.scala | 1 +
.../metrics/FileSourceScanMetricsUpdater.scala | 2 +
.../metrics/HashAggregateMetricsUpdater.scala | 2 +
.../metrics/HiveTableScanMetricsUpdater.scala | 2 +
.../org/apache/gluten/metrics/MetricsUtil.scala | 14 +-
.../gluten/spi/SharedLibraryLoaderCentos7.scala | 1 -
.../gluten/spi/SharedLibraryLoaderCentos8.scala | 1 -
.../gluten/spi/SharedLibraryLoaderCentos9.scala | 1 -
.../gluten/spi/SharedLibraryLoaderDebian11.scala | 1 -
.../gluten/spi/SharedLibraryLoaderDebian12.scala | 1 -
.../spi/SharedLibraryLoaderOpenEuler2403.scala | 1 -
.../gluten/spi/SharedLibraryLoaderUbuntu2004.scala | 1 -
.../gluten/spi/SharedLibraryLoaderUbuntu2204.scala | 1 -
.../apache/gluten/utils/ParquetMetadataUtils.scala | 36 +-
.../vectorized/ColumnarBatchSerializer.scala | 7 +-
.../spark/shuffle/ColumnarShuffleWriter.scala | 4 +-
.../sql/execution/ArrowFileSourceScanExec.scala | 61 -
.../sql/execution/ColumnarBuildSideRelation.scala | 6 +-
.../execution/ColumnarCachedBatchSerializer.scala | 841 ++++-
.../execution/SparkWriteFilesCommitProtocol.scala | 11 +-
.../unsafe/UnsafeColumnarBuildSideRelation.scala | 6 +-
.../apache/spark/sql/expression/UDFResolver.scala | 2 +-
.../gluten/config/AllVeloxConfiguration.scala | 25 +-
.../gluten/execution/ArrowCsvScanSuite.scala | 238 --
.../apache/gluten/execution/FallbackSuite.scala | 280 +-
.../execution/JsonTuplePathRewriteSuite.scala | 73 +
.../gluten/execution/MiscOperatorSuite.scala | 9 +-
.../execution/StarSchemaJoinAggregateSuite.scala | 1062 ++++++
.../execution/VeloxAggregateFunctionsSuite.scala | 20 +-
.../gluten/execution/VeloxMetricsSuite.scala | 27 +-
.../VeloxParquetDataTypeValidationSuite.scala | 16 +-
.../apache/gluten/execution/VeloxTPCHSuite.scala | 11 +
.../functions/MathFunctionsValidateSuite.scala | 26 +
.../functions/ScalarFunctionsValidateSuite.scala | 26 +
.../utils/ParquetEncryptionDetectionSuite.scala | 82 +-
.../ColumnarCacheShipBlockerMarshalSuite.scala | 133 +
.../ColumnarCachedBatchBuildFilterPruneSuite.scala | 330 ++
.../ColumnarCachedBatchBuildFilterSuite.scala | 77 +
.../execution/ColumnarCachedBatchE2ESuite.scala | 512 +++
.../ColumnarCachedBatchFramedBytesSuite.scala | 472 +++
.../ColumnarCachedBatchIntFamilyMarshalSuite.scala | 185 ++
...umnarCachedBatchKryoBoundaryProbeBugSuite.scala | 111 +
.../execution/ColumnarCachedBatchKryoSuite.scala | 178 +
.../ColumnarCachedBatchSerializerHelperSuite.scala | 154 +
.../ColumnarCachedBatchStatsBlobSuite.scala | 257 ++
.../spark/sql/execution/GlutenHiveUDFSuite.scala | 35 +-
.../sql/execution/VeloxParquetWriteSuite.scala | 54 +
.../benchmark/ColumnarTableCacheBenchmark.scala | 114 +-
...ColumnarTableCachePartitionStatsBenchmark.scala | 122 +
.../org/apache/spark/utils/GlutenSuiteUtils.scala | 25 +
...arTableCachePartitionStatsBenchmark-results.txt | 28 +
cpp-ch/CMakeLists.txt | 9 +-
cpp-ch/clickhouse.version | 4 +-
.../AggregateFunctions/GroupLimitFunctions.cpp | 3 +-
cpp-ch/local-engine/CMakeLists.txt | 4 +-
cpp-ch/local-engine/Common/AggregateUtil.cpp | 6 +-
cpp-ch/local-engine/Common/ArrayJoinHelper.cpp | 4 +-
cpp-ch/local-engine/Common/BlockTypeUtils.cpp | 12 +
cpp-ch/local-engine/Common/BlockTypeUtils.h | 1 +
cpp-ch/local-engine/Common/CHUtil.cpp | 21 +-
cpp-ch/local-engine/Common/CHUtil.h | 2 +
cpp-ch/local-engine/Common/DebugUtils.cpp | 5 +-
cpp-ch/local-engine/Common/PlanUtil.cpp | 23 +-
cpp-ch/local-engine/Common/PlanUtil.h | 2 +-
cpp-ch/local-engine/Common/QueryContext.cpp | 13 +-
.../CompactObjectStorageDiskTransaction.cpp | 28 +-
.../CompactObjectStorageDiskTransaction.h | 20 +-
.../Disks/ObjectStorages/GlutenDiskHDFS.h | 4 +-
.../Disks/ObjectStorages/GlutenDiskS3.cpp | 2 +-
.../Disks/ObjectStorages/GlutenDiskS3.h | 2 +-
.../ObjectStorages/GlutenHDFSObjectStorage.cpp | 8 -
.../Disks/ObjectStorages/GlutenHDFSObjectStorage.h | 6 +-
.../ObjectStorages/MetadataStorageFromRocksDB.cpp | 39 +-
.../ObjectStorages/MetadataStorageFromRocksDB.h | 18 +-
...dataStorageFromRocksDBTransactionOperations.cpp | 24 +-
...tadataStorageFromRocksDBTransactionOperations.h | 28 +-
.../registerGlutenDiskObjectStorage.cpp | 9 +-
cpp-ch/local-engine/Disks/registerGlutenDisks.cpp | 6 +-
.../Functions/AggregateFunctionSparkArrayFold.cpp | 297 ++
.../local-engine/Functions/FunctionGetDateData.h | 4 +-
.../local-engine/Functions/FunctionsBloomFilter.h | 6 +-
.../Functions/LocalDigitsToAsciiDigitForDate.cpp | 29 +-
.../local-engine/Functions/SparkArrayFlatten.cpp | 23 +-
.../Functions/SparkFunctionArrayDistinct.cpp | 6 +-
.../Functions/SparkFunctionArrayJoin.cpp | 19 +-
.../local-engine/Functions/SparkFunctionConv.cpp | 3 +-
.../Functions/SparkFunctionGetJsonObject.cpp | 20 +-
.../Functions/SparkFunctionGetJsonObject.h | 10 +-
.../Functions/SparkFunctionHashingExtended.h | 8 +-
.../Functions/SparkFunctionMonthsBetween.cpp | 4 +-
.../Functions/SparkFunctionReinterpretAsString.cpp | 3 +-
.../Functions/SparkFunctionStrToMap.cpp | 8 +-
.../Functions/SparkFunctionToDateTime.h | 4 +-
.../local-engine/Functions/SparkFunctionTrim.cpp | 16 +-
cpp-ch/local-engine/Functions/SparkParseURL.cpp | 3 +-
.../Functions/SparkPartitionEscape.cpp | 3 +-
cpp-ch/local-engine/IO/SplittableBzip2ReadBuffer.h | 1 -
.../FillingDeltaInternalRowDeletedStep.cpp | 3 +-
.../Operator/GraceAggregatingTransform.cpp | 4 +-
.../Parser/AggregateFunctionParser.cpp | 5 +-
cpp-ch/local-engine/Parser/CHColumnToSparkRow.cpp | 25 +-
cpp-ch/local-engine/Parser/CHColumnToSparkRow.h | 4 +-
cpp-ch/local-engine/Parser/ExpressionParser.cpp | 4 +-
cpp-ch/local-engine/Parser/FunctionParser.cpp | 3 +-
cpp-ch/local-engine/Parser/RelMetric.cpp | 16 +-
.../Parser/RelParsers/AggregateRelParser.cpp | 3 +-
.../Parser/RelParsers/CrossRelParser.cpp | 6 +-
.../Parser/RelParsers/FilterRelParser.cpp | 2 +-
.../Parser/RelParsers/GroupLimitRelParser.cpp | 11 +-
.../Parser/RelParsers/JoinRelParser.cpp | 7 +-
.../Parser/RelParsers/MergeTreeRelParser.cpp | 6 +-
.../Parser/RelParsers/ReadRelParser.cpp | 13 +-
.../Parser/RelParsers/SortRelParser.cpp | 3 +-
.../Parser/RelParsers/WindowRelParser.cpp | 4 +-
.../Parser/RelParsers/WriteRelParser.cpp | 12 +-
.../local-engine/Parser/SerializedPlanParser.cpp | 9 +-
cpp-ch/local-engine/Parser/SerializedPlanParser.h | 2 +-
cpp-ch/local-engine/Parser/SparkRowToCHColumn.cpp | 10 +-
cpp-ch/local-engine/Parser/SparkRowToCHColumn.h | 13 +-
.../BloomFilterAggParser.cpp | 2 +-
.../aggregate_function_parser/CountParser.cpp | 3 +-
.../aggregate_function_parser/LeadLagParser.cpp | 6 +-
.../PercentileParserBase.cpp | 2 +-
.../SimpleStatisticsFunctions.cpp | 3 +-
.../arrayHighOrderFunctions.cpp | 46 +-
.../scalar_function_parser/arrayPosition.cpp | 2 +-
.../Parser/scalar_function_parser/elt.cpp | 2 +-
.../Parser/scalar_function_parser/findInset.cpp | 2 +-
.../Parser/scalar_function_parser/getTimestamp.h | 2 +-
.../Parser/scalar_function_parser/locate.cpp | 2 +-
.../mapHighOrderFunctions.cpp | 2 +-
.../Parser/scalar_function_parser/slice.cpp | 2 +-
.../local-engine/Storages/Cache/CacheManager.cpp | 4 +-
.../Storages/IO/AggregateSerializationUtils.cpp | 2 +-
cpp-ch/local-engine/Storages/IO/NativeReader.cpp | 20 +-
cpp-ch/local-engine/Storages/IO/NativeWriter.cpp | 2 +-
.../Storages/Kafka/GlutenKafkaSource.cpp | 3 +-
.../Storages/MergeTree/MergeSparkMergeTreeTask.cpp | 3 +-
.../Storages/MergeTree/MetaDataHelper.cpp | 18 +-
.../Storages/MergeTree/SparkMergeTreeSink.cpp | 3 +-
.../Storages/MergeTree/SparkMergeTreeWriter.cpp | 8 +-
.../Storages/MergeTree/SparkMergeTreeWriter.h | 3 +-
.../Storages/MergeTree/SparkStorageMergeTree.cpp | 44 +-
.../Storages/Parquet/ColumnIndexFilter.cpp | 112 +-
.../Storages/Parquet/ColumnIndexFilter.h | 39 +-
.../Storages/Parquet/ParquetConverter.h | 4 +-
.../Delta/Bitmap/DeltaDVRoaringBitmapArray.cpp | 2 +-
.../Delta/Bitmap/DeltaDVRoaringBitmapArray.h | 2 +-
.../Storages/SubstraitSource/Delta/DeltaWriter.cpp | 7 +-
.../Iceberg/SimpleParquetReader.cpp | 2 +-
.../Storages/SubstraitSource/ORCFormatFile.cpp | 4 +-
.../Storages/SubstraitSource/ParquetFormatFile.cpp | 76 +-
.../Storages/SubstraitSource/ParquetFormatFile.h | 2 -
.../Storages/SubstraitSource/ReadBufferBuilder.cpp | 9 +-
.../Storages/SubstraitSource/ReadBufferBuilder.h | 2 +-
cpp-ch/local-engine/local_engine_jni.cpp | 18 +-
cpp/CMake/BuildRoaring.cmake | 32 +
cpp/CMake/FindRoaring.cmake | 178 +
cpp/core/CMakeLists.txt | 2 +
cpp/core/compute/Runtime.h | 5 +
cpp/core/config/GlutenConfig.cc | 11 +
cpp/core/config/GlutenConfig.h | 3 +
cpp/core/jni/JniCommon.cc | 8 +-
cpp/core/jni/JniCommon.h | 2 +-
cpp/core/jni/JniWrapper.cc | 58 +-
.../operators/serializer/ColumnarBatchSerializer.h | 9 +
cpp/core/shuffle/LocalPartitionWriter.h | 6 +
cpp/core/shuffle/Options.h | 26 +-
cpp/core/shuffle/PartitionWriter.h | 4 +
cpp/core/shuffle/Payload.cc | 130 +-
cpp/core/shuffle/Payload.h | 15 +-
cpp/core/tests/CMakeLists.txt | 1 +
cpp/core/tests/FForCodecTest.cc | 645 ++++
cpp/core/utils/Macros.h | 3 +-
cpp/core/utils/Metrics.h | 2 +
cpp/core/utils/tac/FForCodec.cc | 60 +
cpp/core/utils/tac/FForCodec.h | 45 +
cpp/core/utils/tac/TypeAwareCompressCodec.cc | 82 +
cpp/core/utils/tac/TypeAwareCompressCodec.h | 68 +
cpp/core/utils/tac/ffor.hpp | 501 +++
cpp/velox/CMakeLists.txt | 27 +
cpp/velox/benchmarks/CMakeLists.txt | 2 +
cpp/velox/benchmarks/VeloxBatchResizerBenchmark.cc | 693 ++++
cpp/velox/compute/VeloxBackend.cc | 101 +-
cpp/velox/compute/VeloxBackend.h | 33 +-
.../velox/compute/VeloxConnectorIds.h | 20 +-
cpp/velox/compute/VeloxPlanConverter.cc | 10 +-
cpp/velox/compute/VeloxPlanConverter.h | 2 +
cpp/velox/compute/VeloxRuntime.cc | 253 +-
cpp/velox/compute/VeloxRuntime.h | 32 +
cpp/velox/compute/WholeStageResultIterator.cc | 63 +-
cpp/velox/compute/WholeStageResultIterator.h | 20 +-
cpp/velox/compute/delta/DeltaConnector.cpp | 48 +
cpp/velox/compute/delta/DeltaConnector.h | 70 +
cpp/velox/compute/delta/DeltaDataSource.cpp | 98 +
cpp/velox/compute/delta/DeltaDataSource.h | 78 +
.../compute/delta/DeltaDeletionVectorReader.cpp | 209 ++
.../compute/delta/DeltaDeletionVectorReader.h | 110 +
cpp/velox/compute/delta/DeltaSplit.cpp | 75 +
cpp/velox/compute/delta/DeltaSplit.h | 122 +
cpp/velox/compute/delta/DeltaSplitReader.cpp | 208 ++
cpp/velox/compute/delta/DeltaSplitReader.h | 118 +
cpp/velox/compute/delta/RoaringBitmapArray.cpp | 86 +
cpp/velox/compute/delta/RoaringBitmapArray.h | 60 +
.../velox/compute/delta/tests/CMakeLists.txt | 49 +-
.../compute/delta/tests/DeltaConnectorTest.cpp | 180 +
.../delta/tests/DeltaDeletionVectorReaderTest.cpp | 219 ++
cpp/velox/compute/delta/tests/DeltaSplitTest.cpp | 95 +
.../compute/delta/tests/RoaringBitmapArrayTest.cpp | 53 +
cpp/velox/compute/iceberg/IcebergWriter.cc | 6 +-
cpp/velox/config/VeloxConfig.h | 16 +-
cpp/velox/cudf/CudfPlanValidator.cc | 2 +-
cpp/velox/jni/JniHashTable.cc | 29 +-
cpp/velox/jni/JniHashTable.h | 6 +
cpp/velox/jni/VeloxJniWrapper.cc | 91 +-
cpp/velox/memory/GlutenDirectBufferedInput.h | 26 +
cpp/velox/memory/VeloxMemoryManager.cc | 25 +-
cpp/velox/operators/hashjoin/HashTableBuilder.cc | 194 +-
cpp/velox/operators/hashjoin/HashTableBuilder.h | 34 +-
.../serializer/VeloxColumnarBatchSerializer.cc | 471 +++
.../serializer/VeloxColumnarBatchSerializer.h | 22 +
cpp/velox/shuffle/VeloxHashShuffleWriter.cc | 133 +-
cpp/velox/shuffle/VeloxHashShuffleWriter.h | 83 +-
cpp/velox/shuffle/VeloxShuffleReader.cc | 173 +-
cpp/velox/shuffle/VeloxShuffleReader.h | 16 +-
.../velox/shuffle/VeloxTypeAwareCompress.h | 25 +-
cpp/velox/substrait/SubstraitToVeloxPlan.cc | 82 +-
cpp/velox/substrait/SubstraitToVeloxPlan.h | 5 +
.../substrait/SubstraitToVeloxPlanValidator.cc | 27 +-
.../substrait/SubstraitToVeloxPlanValidator.h | 16 +-
cpp/velox/tests/CMakeLists.txt | 12 +-
cpp/velox/tests/FunctionTest.cc | 3 +-
cpp/velox/tests/ScopedTimerTest.cc | 88 +
.../tests/Substrait2VeloxPlanConversionTest.cc | 7 +-
.../Substrait2VeloxValuesNodeConversionTest.cc | 8 +-
cpp/velox/tests/VeloxBatchResizerTest.cc | 422 +++
.../tests/VeloxColumnarBatchSerializerTest.cc | 948 ++++++
.../VeloxHashShuffleWriterInputEncodingTest.cc | 201 ++
cpp/velox/tests/VeloxShuffleWriterTest.cc | 348 +-
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc | 9 +-
cpp/velox/utils/Common.h | 28 +-
cpp/velox/utils/ConfigExtractor.cc | 6 +-
cpp/velox/utils/VeloxBatchResizer.cc | 141 +-
cpp/velox/utils/VeloxBatchResizer.h | 21 +-
cpp/velox/utils/VeloxWriterUtils.cc | 4 +-
dev/build-helper-functions.sh | 19 +
dev/build-thirdparty.sh | 12 +-
dev/buildbundle-veloxbe.sh | 5 +-
dev/builddeps-veloxbe.sh | 19 +-
dev/ci-velox-buildshared-centos-8.sh | 6 +
.../cp.sh => dev/ci-velox-buildshared-centos-9.sh | 19 +-
...velox-buildstatic-centos-7-enhanced-features.sh | 2 -
dev/ci-velox-buildstatic-centos-7.sh | 2 -
dev/ci-velox-buildstatic-centos-8.sh | 1 -
dev/ci-velox-buildstatic-centos-9.sh | 0
dev/docker/Dockerfile.centos8-dynamic-build | 21 +-
dev/docker/Dockerfile.centos9-dynamic-build | 21 +-
dev/gluten-build-info.sh | 43 +-
dev/package.sh | 2 +
dev/release/build-release.sh | 5 +-
dev/run-scala-test.sh | 7 +
dev/vcpkg/init.sh | 11 +-
dev/vcpkg/setup-build-depends.sh | 2 +-
dev/vcpkg/triplets/arm64-linux-neon.cmake | 4 +
dev/vcpkg/triplets/x64-linux-avx.cmake | 4 +
dev/vcpkg/vcpkg.json | 9 +
dev/verify-ansi-expressions.sh | 271 ++
docs/Configuration.md | 283 +-
docs/developers/HowTo.md | 2 +-
docs/developers/NewToGluten.md | 12 +-
docs/developers/UsingGperftoolsInCH.md | 2 +-
docs/developers/UsingJemallocWithCH.md | 2 +-
docs/developers/dev-container.md | 95 +
docs/get-started/ClickHouse.md | 6 +-
docs/get-started/Velox.md | 17 +-
docs/get-started/VeloxDelta.md | 30 +-
docs/get-started/VeloxDynamicOpenSSL.md | 54 +
docs/get-started/VeloxGCS.md | 4 +-
docs/get-started/VeloxIceberg.md | 4 +-
docs/get-started/VeloxStageResourceAdj.md | 6 +-
docs/get-started/build-guide.md | 8 +
docs/velox-configuration.md | 155 +-
docs/velox-spark-configuration.md | 2 +-
ep/build-clickhouse/src/package.sh | 4 +-
ep/build-velox/src/build-velox.sh | 39 +-
ep/build-velox/src/get-velox.sh | 17 +-
gluten-arrow/pom.xml | 4 +
.../ColumnarBatchSerializerJniWrapper.java | 6 +
.../vectorized/LocalPartitionWriterJniWrapper.java | 3 +-
.../gluten/vectorized/ShuffleReaderJniWrapper.java | 3 +-
.../gluten/vectorized/ShuffleWriterJniWrapper.java | 1 +
.../org/apache/gluten/utils/ArrowAbiUtil.scala | 2 +-
.../scala/org/apache/gluten/utils/ArrowUtil.scala | 168 +-
.../apache/spark/sql/utils/SparkVectorUtil.scala | 4 +-
.../shuffle/gluten/celeborn/CelebornUtils.java | 50 +
.../shuffle/CelebornColumnarShuffleWriter.scala | 19 +-
.../org/apache/gluten/config/ConfigBuilder.scala | 2 +-
.../org/apache/gluten/config/ConfigRegistry.scala | 32 +-
.../apache/gluten/config/GlutenCoreConfig.scala | 10 +-
.../org/apache/gluten/execution/GlutenPlan.scala | 11 +-
.../gluten/extension/columnar/FallbackTag.scala | 32 +-
.../columnar/offload/OffloadSingleNode.scala | 2 +-
.../columnar/transition/ConventionFunc.scala | 8 +
.../extension/injector/InjectorControl.scala | 4 +-
.../spark/memory/GlobalOffHeapMemoryTarget.scala | 4 +-
.../apache/gluten/component/ComponentSuite.scala | 2 +-
.../extension/columnar/FallbackTagSuite.scala | 49 +
.../gluten/extension/DeltaPostTransformRules.scala | 99 +-
.../org/apache/gluten/execution/DeltaSuite.scala | 176 +
gluten-hudi/pom.xml | 6 +
.../apache/gluten/execution/IcebergWriteExec.scala | 22 +-
.../org/apache/gluten/execution/IcebergSuite.scala | 4 +-
.../execution/MicroBatchScanExecTransformer.scala | 2 +-
gluten-paimon/pom.xml | 2 +-
gluten-substrait/pom.xml | 4 +
.../gluten/backendsapi/BackendSettingsApi.scala | 2 -
.../gluten/backendsapi/SparkPlanExecApi.scala | 5 +-
.../gluten/backendsapi/SubstraitBackend.scala | 5 +-
.../org/apache/gluten/config/GlutenConfig.scala | 101 +-
.../execution/FileSourceScanExecTransformer.scala | 3 +-
.../apache/gluten/expression/ConverterUtils.scala | 4 +-
.../gluten/expression/ExpressionConverter.scala | 35 +-
.../gluten/expression/ExpressionMappings.scala | 18 +-
.../CollapseGetJsonObjectExpressionRule.scala | 6 +-
.../columnar/CollapseProjectExecTransformer.scala | 17 +-
.../columnar/heuristic/ExpandFallbackPolicy.scala | 1 +
.../columnar/rewrite/ProjectColumnPruning.scala | 9 +-
.../extension/joinagg/ImplementJoinAggregate.scala | 308 ++
.../joinagg/JoinAggregateFunctionWrapper.scala | 257 ++
.../joinagg/PushAggregateThroughJoin.scala | 598 ++++
.../gluten/utils/DecimalArithmeticUtil.scala | 3 +-
.../GlutenAutoAdjustStageResourceProfile.scala | 7 +-
.../spark/sql/execution/GlutenImplicits.scala | 12 +-
.../execution/GlutenQueryExecutionListener.scala | 42 +-
.../gluten/config/AllGlutenConfiguration.scala | 61 +-
.../execution/GlutenQueryComparisonTest.scala | 2 +-
.../execution/PushAggregateThroughJoinSuite.scala | 456 +++
.../CollapseGetJsonObjectExpressionRuleSuite.scala | 210 ++
.../columnar/transition/TransitionSuite.scala | 14 +
.../columnar/transition/TransitionSuiteBase.scala | 18 +
.../apache/spark/sql/WithQueryPlanListener.scala | 4 +-
gluten-ui/pom.xml | 4 +
.../spark/sql/GlutenExpressionOffloadTracker.scala | 174 +
.../apache/spark/sql/GlutenTestsCommonTrait.scala | 26 +-
.../org/apache/spark/sql/GlutenTestsTrait.scala | 119 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 8 +-
.../gluten/utils/velox/VeloxTestSettings.scala | 42 +-
.../apache/spark/sql/GlutenDataFrameSuite.scala | 3 +-
.../apache/spark/sql/GlutenSQLQueryTestSuite.scala | 5 +-
.../sql/execution/FallbackStrategiesSuite.scala | 6 +-
.../ClickHouseAdaptiveQueryExecSuite.scala | 3 +-
.../velox/VeloxAdaptiveQueryExecSuite.scala | 3 +-
.../benchmarks/ParquetReadBenchmark.scala | 2 +-
.../datasources/orc/GlutenOrcSourceSuite.scala | 3 +-
.../sql/extension/CustomerColumnarPreRules.scala | 47 -
...GlutenCollapseProjectExecTransformerSuite.scala | 34 +
.../extension/GlutenCustomerExtensionSuite.scala | 56 -
.../TestFileSourceScanExecTransformer.scala | 75 -
.../execution/GlutenAggregationQuerySuite.scala} | 15 +-
.../hive/execution/GlutenHiveCommandSuite.scala | 171 +
.../GlutenHiveComparisonTestSupport.scala | 324 ++
.../sql/hive/execution/GlutenHiveDDLSuite.scala | 121 +
.../hive/execution/GlutenHiveExplainSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHivePlanTest.scala} | 8 +-
.../sql/hive/execution/GlutenHiveQuerySuite.scala | 111 +
.../execution/GlutenHiveResolutionSuite.scala} | 13 +-
.../execution/GlutenHiveResourcePathSupport.scala | 22 +-
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 4 +-
.../hive/execution/GlutenHiveSQLViewSuite.scala} | 8 +-
.../GlutenHiveScriptTransformationSuite.scala} | 12 +-
.../execution/GlutenHiveSerDeReadWriteSuite.scala} | 12 +-
.../sql/hive/execution/GlutenHiveSerDeSuite.scala | 64 +
.../hive/execution/GlutenHiveTableScanSuite.scala | 51 +
.../execution/GlutenHiveTypeCoercionSuite.scala} | 13 +-
.../sql/hive/execution/GlutenHiveUDAFSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHiveUDFSuite.scala | 73 +
.../GlutenObjectHashAggregateSuite.scala} | 12 +-
.../GlutenPruneHiveTablePartitionsSuite.scala} | 12 +-
.../sql/hive/execution/GlutenPruningSuite.scala | 20 +-
.../hive/execution/GlutenSQLMetricsSuite.scala} | 8 +-
.../sql/hive/execution/GlutenSQLQuerySuite.scala | 141 +
.../sql/hive/execution/GlutenTestHiveTables.scala | 235 ++
.../sql/hive/execution/GlutenUDAQuerySuite.scala} | 13 +-
.../hive/execution/GlutenWindowQuerySuite.scala | 70 +
.../sql/statistics/SparkFunctionStatistics.scala | 220 --
.../utils/clickhouse/ClickHouseTestSettings.scala | 7 +-
.../gluten/utils/velox/VeloxTestSettings.scala | 42 +-
.../apache/spark/sql/GlutenDataFrameSuite.scala | 3 +-
.../apache/spark/sql/GlutenSQLQueryTestSuite.scala | 5 +-
.../GlutenKeyGroupedPartitioningSuite.scala | 15 +-
.../sql/execution/FallbackStrategiesSuite.scala | 6 +-
.../ClickHouseAdaptiveQueryExecSuite.scala | 3 +-
.../velox/VeloxAdaptiveQueryExecSuite.scala | 3 +-
.../benchmarks/ParquetReadBenchmark.scala | 2 +-
.../datasources/orc/GlutenOrcSourceSuite.scala | 3 +-
.../sql/extension/CustomerColumnarPreRules.scala | 47 -
...GlutenCollapseProjectExecTransformerSuite.scala | 34 +
.../extension/GlutenCustomerExtensionSuite.scala | 56 -
.../TestFileSourceScanExecTransformer.scala | 75 -
.../execution/GlutenAggregationQuerySuite.scala} | 15 +-
.../hive/execution/GlutenHiveCommandSuite.scala | 171 +
.../GlutenHiveComparisonTestSupport.scala | 324 ++
.../sql/hive/execution/GlutenHiveDDLSuite.scala | 124 +
.../hive/execution/GlutenHiveExplainSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHivePlanTest.scala} | 8 +-
.../sql/hive/execution/GlutenHiveQuerySuite.scala | 111 +
.../execution/GlutenHiveResolutionSuite.scala} | 13 +-
.../execution/GlutenHiveResourcePathSupport.scala | 22 +-
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 2 +-
.../hive/execution/GlutenHiveSQLViewSuite.scala} | 8 +-
.../GlutenHiveScriptTransformationSuite.scala} | 12 +-
.../execution/GlutenHiveSerDeReadWriteSuite.scala} | 12 +-
.../sql/hive/execution/GlutenHiveSerDeSuite.scala | 64 +
.../hive/execution/GlutenHiveTableScanSuite.scala | 51 +
.../execution/GlutenHiveTypeCoercionSuite.scala} | 13 +-
.../sql/hive/execution/GlutenHiveUDAFSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHiveUDFSuite.scala | 73 +
.../GlutenObjectHashAggregateSuite.scala} | 12 +-
.../GlutenPruneHiveTablePartitionsSuite.scala} | 12 +-
.../sql/hive/execution/GlutenPruningSuite.scala | 20 +-
.../hive/execution/GlutenSQLMetricsSuite.scala} | 8 +-
.../sql/hive/execution/GlutenSQLQuerySuite.scala | 142 +
.../sql/hive/execution/GlutenTestHiveTables.scala | 235 ++
.../sql/hive/execution/GlutenUDAQuerySuite.scala} | 13 +-
.../hive/execution/GlutenWindowQuerySuite.scala | 70 +
.../sql/statistics/SparkFunctionStatistics.scala | 218 --
.../ClickHouseSQLQueryTestSettings.scala | 20 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 10 +-
.../gluten/utils/velox/VeloxTestSettings.scala | 45 +-
.../apache/spark/sql/GlutenDataFrameSuite.scala | 3 +-
.../GlutenKeyGroupedPartitioningSuite.scala | 15 +-
.../sql/execution/FallbackStrategiesSuite.scala | 6 +-
.../ClickHouseAdaptiveQueryExecSuite.scala | 3 +-
.../velox/VeloxAdaptiveQueryExecSuite.scala | 3 +-
.../benchmarks/ParquetReadBenchmark.scala | 2 +-
.../datasources/GlutenReadSchemaSuite.scala | 19 +-
.../execution/datasources/csv/GlutenCSVSuite.scala | 8 +-
.../datasources/orc/GlutenOrcSourceSuite.scala | 3 +-
.../sql/extension/CustomerColumnarPreRules.scala | 47 -
...GlutenCollapseProjectExecTransformerSuite.scala | 35 +-
.../extension/GlutenCustomerExtensionSuite.scala | 56 -
.../TestFileSourceScanExecTransformer.scala | 75 -
.../execution/GlutenAggregationQuerySuite.scala} | 15 +-
.../hive/execution/GlutenHiveCommandSuite.scala | 171 +
.../GlutenHiveComparisonTestSupport.scala | 324 ++
.../sql/hive/execution/GlutenHiveDDLSuite.scala | 124 +
.../hive/execution/GlutenHiveExplainSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHivePlanTest.scala} | 8 +-
.../sql/hive/execution/GlutenHiveQuerySuite.scala | 110 +
.../execution/GlutenHiveResolutionSuite.scala} | 13 +-
.../execution/GlutenHiveResourcePathSupport.scala | 22 +-
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 2 +-
.../hive/execution/GlutenHiveSQLViewSuite.scala} | 8 +-
.../GlutenHiveScriptTransformationSuite.scala} | 12 +-
.../execution/GlutenHiveSerDeReadWriteSuite.scala} | 12 +-
.../sql/hive/execution/GlutenHiveSerDeSuite.scala | 64 +
.../hive/execution/GlutenHiveTableScanSuite.scala | 51 +
.../execution/GlutenHiveTypeCoercionSuite.scala} | 13 +-
.../sql/hive/execution/GlutenHiveUDAFSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHiveUDFSuite.scala | 21 +-
.../GlutenObjectHashAggregateSuite.scala} | 12 +-
.../GlutenPruneHiveTablePartitionsSuite.scala} | 12 +-
.../sql/hive/execution/GlutenPruningSuite.scala | 20 +-
.../hive/execution/GlutenSQLMetricsSuite.scala} | 8 +-
.../sql/hive/execution/GlutenSQLQuerySuite.scala | 142 +
.../sql/hive/execution/GlutenTestHiveTables.scala | 235 ++
.../sql/hive/execution/GlutenUDAQuerySuite.scala} | 13 +-
.../hive/execution/GlutenWindowQuerySuite.scala | 70 +
.../sql/statistics/SparkFunctionStatistics.scala | 218 --
.../q14a.sf100/explain.txt | 4 +-
.../q14a.sf100/simplified.txt | 2 +-
.../q14b.sf100/explain.txt | 4 +-
.../q14b.sf100/simplified.txt | 2 +-
.../q38.sf100/explain.txt | 12 +-
.../q38.sf100/simplified.txt | 6 +-
.../q8.sf100/explain.txt | 4 +-
.../q8.sf100/simplified.txt | 2 +-
.../q87.sf100/explain.txt | 12 +-
.../q87.sf100/simplified.txt | 6 +-
.../q14.sf100/explain.txt | 4 +-
.../q14.sf100/simplified.txt | 2 +-
.../q14a.sf100/explain.txt | 4 +-
.../q14a.sf100/simplified.txt | 2 +-
.../q36a.sf100/explain.txt | 4 +-
.../q36a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q36a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q36a/simplified.txt | 2 +-
.../q70a.sf100/explain.txt | 4 +-
.../q70a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q70a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q70a/simplified.txt | 2 +-
.../q86a.sf100/explain.txt | 4 +-
.../q86a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q86a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q86a/simplified.txt | 2 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 5 +-
.../gluten/utils/velox/VeloxTestSettings.scala | 106 +-
.../apache/spark/sql/GlutenDataFrameSuite.scala | 3 +-
.../spark/sql/GlutenPlanStabilitySuite.scala | 3 +-
...parkSessionJobTaggingAndCancellationSuite.scala | 6 +-
.../spark/sql/GlutenStreamingTestsTraits.scala | 134 +
.../GlutenArithmeticExpressionSuite.scala | 21 +-
.../expressions/GlutenCastWithAnsiOffSuite.scala | 110 +-
.../expressions/GlutenCastWithAnsiOnSuite.scala | 227 +-
.../GlutenCollectionExpressionsSuite.scala | 8 +-
.../expressions/GlutenDateExpressionsSuite.scala | 5 +-
.../expressions/GlutenDecimalExpressionSuite.scala | 8 +-
.../GlutenIntervalExpressionsSuite.scala | 8 +-
.../expressions/GlutenMathExpressionsSuite.scala | 5 +-
.../expressions/GlutenStringExpressionsSuite.scala | 8 +-
.../catalyst/expressions/GlutenTryCastSuite.scala | 43 +-
.../catalyst/expressions/GlutenTryEvalSuite.scala | 19 +-
.../GlutenKeyGroupedPartitioningSuite.scala | 20 +-
.../sql/execution/GlutenSparkSqlParserSuite.scala | 15 +-
.../datasources/GlutenReadSchemaSuite.scala | 19 +-
.../execution/datasources/csv/GlutenCSVSuite.scala | 8 +-
.../datasources/orc/GlutenOrcSourceSuite.scala | 3 +-
.../parquet/GlutenParquetTypeWideningSuite.scala | 3 +-
.../sql/extension/CustomerColumnarPreRules.scala | 47 -
...GlutenCollapseProjectExecTransformerSuite.scala | 35 +-
.../extension/GlutenCustomerExtensionSuite.scala | 56 -
.../TestFileSourceScanExecTransformer.scala | 74 -
.../sql/gluten/GlutenFallbackStrategiesSuite.scala | 6 +-
.../execution/GlutenAggregationQuerySuite.scala} | 15 +-
.../hive/execution/GlutenHiveCommandSuite.scala | 171 +
.../GlutenHiveComparisonTestSupport.scala | 323 ++
.../sql/hive/execution/GlutenHiveDDLSuite.scala | 95 +
.../execution/GlutenHiveExplainSuite.scala} | 8 +-
.../execution/GlutenHivePlanTest.scala} | 8 +-
.../sql/hive/execution/GlutenHiveQuerySuite.scala | 73 +
.../execution/GlutenHiveResolutionSuite.scala} | 13 +-
.../execution/GlutenHiveResourcePathSupport.scala | 22 +-
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 2 +-
.../execution/GlutenHiveSQLViewSuite.scala} | 8 +-
.../GlutenHiveScriptTransformationSuite.scala} | 12 +-
.../execution/GlutenHiveSerDeReadWriteSuite.scala} | 12 +-
.../sql/hive/execution/GlutenHiveSerDeSuite.scala | 64 +
.../hive/execution/GlutenHiveTableScanSuite.scala | 51 +
.../execution/GlutenHiveTypeCoercionSuite.scala} | 13 +-
.../execution/GlutenHiveUDAFSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHiveUDFSuite.scala | 21 +-
.../GlutenObjectHashAggregateSuite.scala} | 12 +-
.../GlutenPruneHiveTablePartitionsSuite.scala} | 12 +-
.../sql/hive/execution/GlutenPruningSuite.scala | 20 +-
.../execution/GlutenSQLMetricsSuite.scala} | 8 +-
.../sql/hive/execution/GlutenSQLQuerySuite.scala | 62 +
.../sql/hive/execution/GlutenTestHiveTables.scala | 235 ++
.../execution/GlutenUDAQuerySuite.scala} | 13 +-
.../hive/execution/GlutenWindowQuerySuite.scala | 70 +
.../sql/statistics/SparkFunctionStatistics.scala | 218 --
.../GlutenAcceptsLatestSeenOffsetSuite.scala | 4 +-
.../streaming/GlutenEventTimeWatermarkSuite.scala | 6 +-
.../sql/streaming/GlutenFileStreamSinkSuite.scala | 6 +-
.../streaming/GlutenFileStreamSourceSuite.scala | 6 +-
.../streaming/GlutenFileStreamStressSuite.scala | 4 +-
...pGroupsInPandasWithStateDistributionSuite.scala | 4 +-
...GlutenFlatMapGroupsInPandasWithStateSuite.scala | 4 +-
...enFlatMapGroupsWithStateDistributionSuite.scala | 4 +-
.../GlutenFlatMapGroupsWithStateSuite.scala | 6 +-
...atMapGroupsWithStateWithInitialStateSuite.scala | 4 +-
.../streaming/GlutenMemorySourceStressSuite.scala | 6 +-
.../GlutenMultiStatefulOperatorsSuite.scala | 4 +-
.../streaming/GlutenReportSinkMetricsSuite.scala | 6 +-
.../spark/sql/streaming/GlutenStreamSuite.scala | 4 +-
...utenStreamingAggregationDistributionSuite.scala | 4 +-
.../GlutenStreamingAggregationSuite.scala | 8 +-
...enStreamingDeduplicationDistributionSuite.scala | 4 +-
.../GlutenStreamingDeduplicationSuite.scala | 6 +-
...treamingDeduplicationWithinWatermarkSuite.scala | 4 +-
.../sql/streaming/GlutenStreamingJoinSuite.scala | 14 +-
...tenStreamingQueryHashPartitionVerifySuite.scala | 105 +-
.../GlutenStreamingQueryListenerSuite.scala | 4 +-
.../GlutenStreamingQueryListenersConfSuite.scala | 4 +-
.../GlutenStreamingQueryManagerSuite.scala | 4 +-
...treamingQueryOptimizationCorrectnessSuite.scala | 4 +-
...lutenStreamingQueryStatusAndProgressSuite.scala | 4 +-
.../sql/streaming/GlutenStreamingQuerySuite.scala | 15 +-
.../streaming/GlutenStreamingSelfUnionSuite.scala | 6 +-
...enStreamingSessionWindowDistributionSuite.scala | 4 +-
.../GlutenStreamingSessionWindowSuite.scala | 4 +-
...reamingStateStoreFormatCompatibilitySuite.scala | 4 +-
...utenStreamingSymmetricHashJoinHelperSuite.scala | 4 +-
.../GlutenTransformWithListStateSuite.scala | 4 +-
.../GlutenTransformWithListStateTTLSuite.scala | 4 +-
.../GlutenTransformWithMapStateSuite.scala | 4 +-
.../GlutenTransformWithMapStateTTLSuite.scala | 4 +-
.../GlutenTransformWithStateAvroSuite.scala | 4 +-
.../GlutenTransformWithStateChainingSuite.scala | 4 +-
.../GlutenTransformWithStateClusterSuite.scala | 4 +-
...GlutenTransformWithStateInitialStateSuite.scala | 4 +-
.../streaming/GlutenTransformWithStateSuite.scala | 4 +-
.../GlutenTransformWithStateUnsafeRowSuite.scala | 4 +-
.../GlutenTransformWithValueStateTTLSuite.scala | 4 +-
.../streaming/GlutenTriggerAvailableNowSuite.scala | 6 +-
.../q14a.sf100/explain.txt | 4 +-
.../q14a.sf100/simplified.txt | 2 +-
.../q14b.sf100/explain.txt | 4 +-
.../q14b.sf100/simplified.txt | 2 +-
.../q38.sf100/explain.txt | 12 +-
.../q38.sf100/simplified.txt | 6 +-
.../q8.sf100/explain.txt | 4 +-
.../q8.sf100/simplified.txt | 2 +-
.../q87.sf100/explain.txt | 12 +-
.../q87.sf100/simplified.txt | 6 +-
.../q14.sf100/explain.txt | 4 +-
.../q14.sf100/simplified.txt | 2 +-
.../q14a.sf100/explain.txt | 4 +-
.../q14a.sf100/simplified.txt | 2 +-
.../q36a.sf100/explain.txt | 4 +-
.../q36a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q36a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q36a/simplified.txt | 2 +-
.../q70a.sf100/explain.txt | 4 +-
.../q70a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q70a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q70a/simplified.txt | 2 +-
.../q86a.sf100/explain.txt | 4 +-
.../q86a.sf100/simplified.txt | 2 +-
.../gluten-approved-plans-v2_7/q86a/explain.txt | 4 +-
.../gluten-approved-plans-v2_7/q86a/simplified.txt | 2 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 5 +-
.../gluten/utils/velox/VeloxTestSettings.scala | 110 +-
.../apache/spark/sql/GlutenDataFrameSuite.scala | 3 +-
.../spark/sql/GlutenPlanStabilitySuite.scala | 3 +-
...parkSessionJobTaggingAndCancellationSuite.scala | 6 +-
.../spark/sql/GlutenStreamingTestsTraits.scala | 134 +
.../GlutenArithmeticExpressionSuite.scala | 23 +-
.../expressions/GlutenCastWithAnsiOffSuite.scala | 111 +-
.../expressions/GlutenCastWithAnsiOnSuite.scala | 229 +-
.../GlutenCollectionExpressionsSuite.scala | 9 +-
.../expressions/GlutenDateExpressionsSuite.scala | 9 +-
.../expressions/GlutenDecimalExpressionSuite.scala | 10 +-
.../GlutenIntervalExpressionsSuite.scala | 10 +-
.../expressions/GlutenMathExpressionsSuite.scala | 9 +-
.../expressions/GlutenStringExpressionsSuite.scala | 10 +-
.../catalyst/expressions/GlutenTryCastSuite.scala | 40 +-
.../catalyst/expressions/GlutenTryEvalSuite.scala | 23 +-
.../GlutenKeyGroupedPartitioningSuite.scala | 20 +-
.../sql/execution/GlutenSparkSqlParserSuite.scala | 15 +-
.../datasources/GlutenReadSchemaSuite.scala | 19 +-
.../execution/datasources/csv/GlutenCSVSuite.scala | 8 +-
.../datasources/orc/GlutenOrcSourceSuite.scala | 3 +-
.../parquet/GlutenParquetTypeWideningSuite.scala | 3 +-
.../sql/extension/CustomerColumnarPreRules.scala | 47 -
...GlutenCollapseProjectExecTransformerSuite.scala | 35 +-
.../extension/GlutenCustomerExtensionSuite.scala | 56 -
.../TestFileSourceScanExecTransformer.scala | 74 -
.../sql/gluten/GlutenFallbackStrategiesSuite.scala | 6 +-
.../execution/GlutenAggregationQuerySuite.scala} | 15 +-
.../hive/execution/GlutenHiveCommandSuite.scala | 171 +
.../GlutenHiveComparisonTestSupport.scala | 323 ++
.../sql/hive/execution/GlutenHiveDDLSuite.scala | 95 +
.../hive/execution/GlutenHiveExplainSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHivePlanTest.scala} | 8 +-
.../sql/hive/execution/GlutenHiveQuerySuite.scala | 73 +
.../execution/GlutenHiveResolutionSuite.scala} | 13 +-
.../execution/GlutenHiveResourcePathSupport.scala | 22 +-
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 2 +-
.../hive/execution/GlutenHiveSQLViewSuite.scala} | 8 +-
.../GlutenHiveScriptTransformationSuite.scala} | 12 +-
.../execution/GlutenHiveSerDeReadWriteSuite.scala} | 12 +-
.../sql/hive/execution/GlutenHiveSerDeSuite.scala | 64 +
.../hive/execution/GlutenHiveTableScanSuite.scala | 51 +
.../execution/GlutenHiveTypeCoercionSuite.scala} | 13 +-
.../sql/hive/execution/GlutenHiveUDAFSuite.scala} | 8 +-
.../sql/hive/execution/GlutenHiveUDFSuite.scala | 21 +-
.../GlutenObjectHashAggregateSuite.scala} | 12 +-
.../GlutenPruneHiveTablePartitionsSuite.scala} | 12 +-
.../sql/hive/execution/GlutenPruningSuite.scala | 20 +-
.../hive/execution/GlutenSQLMetricsSuite.scala} | 8 +-
.../sql/hive/execution/GlutenSQLQuerySuite.scala | 62 +
.../sql/hive/execution/GlutenTestHiveTables.scala | 235 ++
.../sql/hive/execution/GlutenUDAQuerySuite.scala} | 13 +-
.../hive/execution/GlutenWindowQuerySuite.scala | 70 +
.../apache/spark/sql/shim/GlutenTestsTrait.scala | 21 +-
.../sql/statistics/SparkFunctionStatistics.scala | 218 --
.../GlutenAcceptsLatestSeenOffsetSuite.scala | 4 +-
.../streaming/GlutenEventTimeWatermarkSuite.scala | 6 +-
.../sql/streaming/GlutenFileStreamSinkSuite.scala | 6 +-
.../streaming/GlutenFileStreamSourceSuite.scala | 6 +-
.../streaming/GlutenFileStreamStressSuite.scala | 4 +-
...pGroupsInPandasWithStateDistributionSuite.scala | 4 +-
...GlutenFlatMapGroupsInPandasWithStateSuite.scala | 4 +-
...enFlatMapGroupsWithStateDistributionSuite.scala | 4 +-
.../GlutenFlatMapGroupsWithStateSuite.scala | 6 +-
...atMapGroupsWithStateWithInitialStateSuite.scala | 4 +-
.../streaming/GlutenMemorySourceStressSuite.scala | 6 +-
.../GlutenMultiStatefulOperatorsSuite.scala | 4 +-
.../streaming/GlutenReportSinkMetricsSuite.scala | 6 +-
.../GlutenStreamRealTimeModeAllowlistSuite.scala | 4 +-
.../GlutenStreamRealTimeModeE2ESuite.scala | 4 +-
.../streaming/GlutenStreamRealTimeModeSuite.scala | 6 +-
.../spark/sql/streaming/GlutenStreamSuite.scala | 4 +-
...utenStreamingAggregationDistributionSuite.scala | 4 +-
.../GlutenStreamingAggregationSuite.scala | 8 +-
...enStreamingDeduplicationDistributionSuite.scala | 4 +-
.../GlutenStreamingDeduplicationSuite.scala | 6 +-
...treamingDeduplicationWithinWatermarkSuite.scala | 4 +-
.../sql/streaming/GlutenStreamingJoinSuite.scala | 14 +-
...tenStreamingQueryHashPartitionVerifySuite.scala | 105 +-
.../GlutenStreamingQueryListenerSuite.scala | 4 +-
.../GlutenStreamingQueryListenersConfSuite.scala | 4 +-
.../GlutenStreamingQueryManagerSuite.scala | 4 +-
...treamingQueryOptimizationCorrectnessSuite.scala | 4 +-
...lutenStreamingQueryStatusAndProgressSuite.scala | 4 +-
.../sql/streaming/GlutenStreamingQuerySuite.scala | 15 +-
.../streaming/GlutenStreamingSelfUnionSuite.scala | 6 +-
...enStreamingSessionWindowDistributionSuite.scala | 4 +-
.../GlutenStreamingSessionWindowSuite.scala | 4 +-
...reamingStateStoreFormatCompatibilitySuite.scala | 4 +-
...utenStreamingSymmetricHashJoinHelperSuite.scala | 4 +-
.../GlutenTransformWithListStateSuite.scala | 4 +-
.../GlutenTransformWithListStateTTLSuite.scala | 4 +-
.../GlutenTransformWithMapStateSuite.scala | 4 +-
.../GlutenTransformWithMapStateTTLSuite.scala | 4 +-
.../GlutenTransformWithStateAvroSuite.scala | 4 +-
.../GlutenTransformWithStateChainingSuite.scala | 4 +-
.../GlutenTransformWithStateClusterSuite.scala | 4 +-
...GlutenTransformWithStateInitialStateSuite.scala | 4 +-
.../streaming/GlutenTransformWithStateSuite.scala | 4 +-
.../GlutenTransformWithStateUnsafeRowSuite.scala | 4 +-
.../GlutenTransformWithValueStateTTLSuite.scala | 4 +-
.../streaming/GlutenTriggerAvailableNowSuite.scala | 6 +-
.../gluten/config/GlutenRuntimeConfigSuite.scala | 50 +
pom.xml | 95 +-
.../scala/org/apache/gluten/GlutenBuildInfo.scala | 1 +
.../org/apache/gluten/sql/shims/SparkShims.scala | 27 +-
.../gluten/sql/shims/spark33/Spark33Shims.scala | 30 +-
.../datasources/parquet/ParquetFileFormat.scala | 20 +-
.../datasources/v2/BatchScanExecShim.scala | 9 -
.../gluten/sql/shims/spark34/Spark34Shims.scala | 15 +-
.../datasources/v2/BatchScanExecShim.scala | 21 -
.../sql/extension/RewriteCreateTableAsSelect.scala | 4 +-
.../gluten/sql/shims/spark35/Spark35Shims.scala | 14 +-
.../datasources/v2/BatchScanExecShim.scala | 19 -
.../gluten/sql/shims/spark40/Spark40Shims.scala | 19 +-
.../datasources/v2/BatchScanExecShim.scala | 21 -
.../gluten/sql/shims/spark41/Spark41Shims.scala | 30 +-
.../datasources/v2/BatchScanExecShim.scala | 21 -
.../sql/execution/streaming/MemoryStream.scala | 19 +-
.../gluten/integration/command/Parameterized.java | 4 +-
.../gluten/integration/action/Parameterized.scala | 2 +-
.../apache/gluten/integration/stat/RamStat.scala | 4 +-
tools/gluten-it/pom.xml | 21 +-
tools/gluten-te/.fixme | 1 -
tools/gluten-te/centos/build.sh | 93 -
tools/gluten-te/centos/buildenv.sh | 81 -
tools/gluten-te/centos/cbash-build.sh | 71 -
tools/gluten-te/centos/cbash-mount.sh | 60 -
tools/gluten-te/centos/centos-7-deps.sh | 105 -
tools/gluten-te/centos/centos-8-deps.sh | 36 -
tools/gluten-te/centos/defaults.conf | 66 -
tools/gluten-te/centos/dockerfile-build | 106 -
tools/gluten-te/centos/dockerfile-buildenv | 89 -
.../buildhere-veloxbe-portable-libs/README.md | 33 -
.../buildhere-veloxbe-portable-libs/run-default.sh | 21 -
.../buildhere-veloxbe-portable-libs/run.sh | 29 -
.../gluten-te/centos/gha/gha-checkout/checkout.sh | 49 -
tools/gluten-te/centos/gha/gha-checkout/clean.sh | 25 -
tools/gluten-te/centos/gha/gha-checkout/exec.sh | 28 -
.../centos/gha/gha-checkout/scripts/init.sh | 25 -
tools/gluten-te/centos/shared.sh | 48 -
tools/gluten-te/ubuntu/README.md | 106 -
tools/gluten-te/ubuntu/build.sh | 92 -
tools/gluten-te/ubuntu/buildenv.sh | 80 -
tools/gluten-te/ubuntu/cbash-build.sh | 71 -
tools/gluten-te/ubuntu/cbash-mount.sh | 60 -
tools/gluten-te/ubuntu/defaults.conf | 94 -
tools/gluten-te/ubuntu/docker-compose.yml | 16 -
tools/gluten-te/ubuntu/dockerfile-build | 106 -
tools/gluten-te/ubuntu/dockerfile-buildenv | 162 -
tools/gluten-te/ubuntu/dockerfile-tpc | 62 -
.../ubuntu/examples/buildhere-veloxbe-dev/run.sh | 27 -
.../examples/buildhere-veloxbe-dev/scripts/all.sh | 42 -
.../buildhere-veloxbe-portable-libs/README.md | 33 -
.../buildhere-veloxbe-portable-libs/run-default.sh | 21 -
.../buildhere-veloxbe-portable-libs/run.sh | 29 -
.../buildhere-veloxbe-portable-libs/scripts/all.sh | 51 -
.../ubuntu/examples/buildhere-veloxbe/run.sh | 29 -
.../gluten-te/ubuntu/gha/gha-checkout/checkout.sh | 49 -
tools/gluten-te/ubuntu/gha/gha-checkout/clean.sh | 25 -
tools/gluten-te/ubuntu/gha/gha-checkout/cp.sh | 29 -
tools/gluten-te/ubuntu/gha/gha-checkout/exec.sh | 28 -
.../ubuntu/gha/gha-checkout/scripts/init.sh | 25 -
tools/gluten-te/ubuntu/scripts/cmd.sh | 18 -
tools/gluten-te/ubuntu/scripts/set-login-env.sh | 18 -
tools/gluten-te/ubuntu/shared.sh | 48 -
tools/gluten-te/ubuntu/tpc.sh | 154 -
tools/qualification-tool/pom.xml | 2 +-
.../org/apache/gluten/qt/QualificationTool.scala | 95 +-
.../benchmark_velox/analysis/requirements.txt | 16 +-
969 files changed, 36921 insertions(+), 10745 deletions(-)
create mode 100644 .devcontainer/devcontainer.json
create mode 100644 .github/skills/ansi-analysis/SKILL.md
create mode 100644 .github/skills/ansi-analysis/analyze-ansi.py
create mode 100644 .github/skills/ansi-analysis/shared.md
rename .github/workflows/util/{install-flink-resources.sh =>
install-flink-deps.sh} (100%)
copy .github/workflows/util/{install-resources.sh => install-spark-deps.sh}
(56%)
rename .github/workflows/util/{install-resources.sh =>
install-spark-resources.sh} (51%)
create mode 100644 .github/workflows/velox_backend_ansi.yml
create mode 100644 AGENTS.md
delete mode 100644
backends-clickhouse/src-delta33/test/scala/org/apache/spark/gluten/delta/GlutenDeltaMergeTreeDeletionVectorSuite.scala
create mode 100644
backends-clickhouse/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
create mode 100644
backends-velox/src-delta33/main/scala/org/apache/gluten/delta/DeltaDeletionVectorScanInfo.scala
create mode 100644
backends-velox/src-delta33/test/scala/org/apache/gluten/delta/DeltaDeletionVectorScanInfoSuite.scala
create mode 100644
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaNativeWriteSuite.scala
copy .gitmodules =>
backends-velox/src-delta40/main/resources/META-INF/gluten-components/org.apache.gluten.component.VeloxDelta40WriteComponent
(100%)
create mode 100644
backends-velox/src-delta40/main/scala/org/apache/gluten/component/VeloxDelta40WriteComponent.scala
create mode 100644
backends-velox/src-delta40/main/scala/org/apache/gluten/delta/DeltaDeletionVectorScanInfo.scala
copy backends-velox/{src-delta33 =>
src-delta40}/main/scala/org/apache/spark/sql/delta/GlutenDeltaParquetFileFormat.scala
(98%)
copy backends-velox/{src-delta33 =>
src-delta40}/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala
(93%)
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/GlutenParquetFileFormat.scala
copy backends-velox/{src-delta33 =>
src-delta40}/main/scala/org/apache/spark/sql/delta/files/GlutenDeltaFileFormatWriter.scala
(97%)
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/perf/GlutenDeltaOptimizedWriterExec.scala
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsFallbackTracker.scala
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsRowCountingTracker.scala
copy backends-velox/{src-delta33 =>
src-delta40}/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala
(94%)
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/execution/datasources/v2/DeltaWriteOperators.scala
create mode 120000
backends-velox/src-delta40/main/scala/org/apache/spark/sql/execution/datasources/v2/OffloadDeltaCommand.scala
create mode 100644
backends-velox/src-delta40/test/scala/org/apache/gluten/delta/DeltaDeletionVectorScanInfoSuite.scala
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeleteSQLSuite.scala (100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeleteSuiteBase.scala (100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeletionVectorsTestUtils.scala
(100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeltaColumnMappingTestUtils.scala
(100%)
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaDDLSuite.scala
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeltaExcludedBySparkVersionTestMixinShims.scala
(100%)
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaInsertIntoTableSuite.scala
copy
backends-velox/{src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
src-delta40/test/scala/org/apache/spark/sql/delta/DeltaInsertIntoTableSuiteShims.scala}
(62%)
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaMetricsUtils.scala
create mode 100644
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaNativeWriteSuite.scala
create mode 100644
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/DeltaSuiteShims.scala
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/DeltaTestUtils.scala (100%)
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/FakeFileSystem.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/UpdateSQLSuite.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/UpdateSuiteBase.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/clustering/ClusteredTableClusteringSuite.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/coordinatedcommits/CoordinatedCommitsTestUtils.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/perf/OptimizeMetadataOnlyDeltaQuerySuite.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/perf/OptimizedWritesSuite.scala
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/skipping/ClusteredTableTestUtils.scala
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/test/DeltaColumnMappingSelectedTestMixin.scala
(100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/test/DeltaExcludedTestMixin.scala
(100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala
(93%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/test/DeltaSQLTestUtils.scala
(100%)
copy backends-velox/{src-delta33 =>
src-delta40}/test/scala/org/apache/spark/sql/delta/test/DeltaTestImplicits.scala
(100%)
create mode 120000
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/ScanReportHelper.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/ArrowCSVFileFormat.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/ArrowCSVOptionConverter.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/v2/ArrowCSVPartitionReaderFactory.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/v2/ArrowCSVScan.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/v2/ArrowCSVScanBuilder.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/datasource/v2/ArrowCSVTable.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/execution/datasource/v2/ArrowBatchScanExec.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/extension/ArrowConvertorRule.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/gluten/extension/ArrowScanReplaceRule.scala
delete mode 100644
backends-velox/src/main/scala/org/apache/spark/sql/execution/ArrowFileSourceScanExec.scala
delete mode 100644
backends-velox/src/test/scala/org/apache/gluten/execution/ArrowCsvScanSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/gluten/execution/JsonTuplePathRewriteSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/gluten/execution/StarSchemaJoinAggregateSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCacheShipBlockerMarshalSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchBuildFilterPruneSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchBuildFilterSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchE2ESuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchFramedBytesSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchIntFamilyMarshalSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchKryoBoundaryProbeBugSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchKryoSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchSerializerHelperSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/ColumnarCachedBatchStatsBlobSuite.scala
create mode 100644
backends-velox/src/test/scala/org/apache/spark/sql/execution/benchmark/ColumnarTableCachePartitionStatsBenchmark.scala
create mode 100644
benchmarks/ColumnarTableCachePartitionStatsBenchmark-results.txt
create mode 100644
cpp-ch/local-engine/Functions/AggregateFunctionSparkArrayFold.cpp
create mode 100644 cpp/CMake/BuildRoaring.cmake
create mode 100644 cpp/CMake/FindRoaring.cmake
create mode 100644 cpp/core/tests/FForCodecTest.cc
create mode 100644 cpp/core/utils/tac/FForCodec.cc
create mode 100644 cpp/core/utils/tac/FForCodec.h
create mode 100644 cpp/core/utils/tac/TypeAwareCompressCodec.cc
create mode 100644 cpp/core/utils/tac/TypeAwareCompressCodec.h
create mode 100644 cpp/core/utils/tac/ffor.hpp
create mode 100644 cpp/velox/benchmarks/VeloxBatchResizerBenchmark.cc
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=> cpp/velox/compute/VeloxConnectorIds.h (71%)
create mode 100644 cpp/velox/compute/delta/DeltaConnector.cpp
create mode 100644 cpp/velox/compute/delta/DeltaConnector.h
create mode 100644 cpp/velox/compute/delta/DeltaDataSource.cpp
create mode 100644 cpp/velox/compute/delta/DeltaDataSource.h
create mode 100644 cpp/velox/compute/delta/DeltaDeletionVectorReader.cpp
create mode 100644 cpp/velox/compute/delta/DeltaDeletionVectorReader.h
create mode 100644 cpp/velox/compute/delta/DeltaSplit.cpp
create mode 100644 cpp/velox/compute/delta/DeltaSplit.h
create mode 100644 cpp/velox/compute/delta/DeltaSplitReader.cpp
create mode 100644 cpp/velox/compute/delta/DeltaSplitReader.h
create mode 100644 cpp/velox/compute/delta/RoaringBitmapArray.cpp
create mode 100644 cpp/velox/compute/delta/RoaringBitmapArray.h
rename
tools/gluten-te/centos/examples/buildhere-veloxbe-portable-libs/scripts/all.sh
=> cpp/velox/compute/delta/tests/CMakeLists.txt (51%)
mode change 100755 => 100644
create mode 100644 cpp/velox/compute/delta/tests/DeltaConnectorTest.cpp
create mode 100644
cpp/velox/compute/delta/tests/DeltaDeletionVectorReaderTest.cpp
create mode 100644 cpp/velox/compute/delta/tests/DeltaSplitTest.cpp
create mode 100644 cpp/velox/compute/delta/tests/RoaringBitmapArrayTest.cpp
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=> cpp/velox/shuffle/VeloxTypeAwareCompress.h (62%)
create mode 100644 cpp/velox/tests/ScopedTimerTest.cc
create mode 100644 cpp/velox/tests/VeloxHashShuffleWriterInputEncodingTest.cc
rename tools/gluten-te/centos/gha/gha-checkout/cp.sh =>
dev/ci-velox-buildshared-centos-9.sh (73%)
mode change 100644 => 100755 dev/ci-velox-buildstatic-centos-9.sh
create mode 100755 dev/verify-ansi-expressions.sh
create mode 100644 docs/developers/dev-container.md
create mode 100644 docs/get-started/VeloxDynamicOpenSSL.md
create mode 100644
gluten-core/src/test/scala/org/apache/gluten/extension/columnar/FallbackTagSuite.scala
create mode 100644
gluten-substrait/src/main/scala/org/apache/gluten/extension/joinagg/ImplementJoinAggregate.scala
create mode 100644
gluten-substrait/src/main/scala/org/apache/gluten/extension/joinagg/JoinAggregateFunctionWrapper.scala
create mode 100644
gluten-substrait/src/main/scala/org/apache/gluten/extension/joinagg/PushAggregateThroughJoin.scala
create mode 100644
gluten-substrait/src/test/scala/org/apache/gluten/execution/PushAggregateThroughJoinSuite.scala
create mode 100644
gluten-substrait/src/test/scala/org/apache/gluten/extension/columnar/CollapseGetJsonObjectExpressionRuleSuite.scala
create mode 100644
gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenExpressionOffloadTracker.scala
delete mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/extension/CustomerColumnarPreRules.scala
delete mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
delete mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/extension/TestFileSourceScanExecTransformer.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenAggregationQuerySuite.scala}
(63%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveCommandSuite.scala
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveComparisonTestSupport.scala
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveDDLSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveExplainSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHivePlanTest.scala}
(75%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveQuerySuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResolutionSuite.scala}
(72%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResourcePathSupport.scala
(61%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLViewSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveScriptTransformationSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeReadWriteSuite.scala}
(74%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeSuite.scala
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTypeCoercionSuite.scala}
(71%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDAFSuite.scala}
(75%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDFSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenObjectHashAggregateSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruneHiveTablePartitionsSuite.scala}
(73%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruningSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLMetricsSuite.scala}
(75%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLQuerySuite.scala
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenTestHiveTables.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenUDAQuerySuite.scala}
(65%)
create mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenWindowQuerySuite.scala
delete mode 100644
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/statistics/SparkFunctionStatistics.scala
delete mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/extension/CustomerColumnarPreRules.scala
delete mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
delete mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/extension/TestFileSourceScanExecTransformer.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenAggregationQuerySuite.scala}
(63%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveCommandSuite.scala
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveComparisonTestSupport.scala
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveDDLSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveExplainSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHivePlanTest.scala}
(75%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveQuerySuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResolutionSuite.scala}
(72%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResourcePathSupport.scala
(61%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLViewSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveScriptTransformationSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeReadWriteSuite.scala}
(74%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeSuite.scala
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTypeCoercionSuite.scala}
(71%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDAFSuite.scala}
(75%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDFSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenObjectHashAggregateSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruneHiveTablePartitionsSuite.scala}
(73%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruningSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLMetricsSuite.scala}
(75%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLQuerySuite.scala
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenTestHiveTables.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenUDAQuerySuite.scala}
(65%)
create mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenWindowQuerySuite.scala
delete mode 100644
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/statistics/SparkFunctionStatistics.scala
delete mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/extension/CustomerColumnarPreRules.scala
delete mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
delete mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/extension/TestFileSourceScanExecTransformer.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenAggregationQuerySuite.scala}
(63%)
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveCommandSuite.scala
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveComparisonTestSupport.scala
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveDDLSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveExplainSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHivePlanTest.scala}
(75%)
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveQuerySuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResolutionSuite.scala}
(72%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResourcePathSupport.scala
(61%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLViewSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveScriptTransformationSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeReadWriteSuite.scala}
(74%)
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeSuite.scala
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTypeCoercionSuite.scala}
(71%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDAFSuite.scala}
(75%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDFSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenObjectHashAggregateSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruneHiveTablePartitionsSuite.scala}
(73%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruningSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLMetricsSuite.scala}
(75%)
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLQuerySuite.scala
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenTestHiveTables.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenUDAQuerySuite.scala}
(65%)
create mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenWindowQuerySuite.scala
delete mode 100644
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/statistics/SparkFunctionStatistics.scala
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenStreamingTestsTraits.scala
delete mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/extension/CustomerColumnarPreRules.scala
delete mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
delete mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/extension/TestFileSourceScanExecTransformer.scala
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenAggregationQuerySuite.scala} (63%)
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveCommandSuite.scala
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveComparisonTestSupport.scala
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveDDLSuite.scala
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=> hive/execution/GlutenHiveExplainSuite.scala} (75%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=> hive/execution/GlutenHivePlanTest.scala} (75%)
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveQuerySuite.scala
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenHiveResolutionSuite.scala} (72%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResourcePathSupport.scala
(61%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=> hive/execution/GlutenHiveSQLViewSuite.scala} (75%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenHiveScriptTransformationSuite.scala} (73%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenHiveSerDeReadWriteSuite.scala} (74%)
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeSuite.scala
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenHiveTypeCoercionSuite.scala} (71%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=> hive/execution/GlutenHiveUDAFSuite.scala} (75%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDFSuite.scala
(65%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenObjectHashAggregateSuite.scala} (73%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenPruneHiveTablePartitionsSuite.scala} (73%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruningSuite.scala
(65%)
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=> hive/execution/GlutenSQLMetricsSuite.scala} (75%)
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLQuerySuite.scala
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenTestHiveTables.scala
copy
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/{streaming/GlutenFileStreamSinkSuite.scala
=> hive/execution/GlutenUDAQuerySuite.scala} (65%)
create mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenWindowQuerySuite.scala
delete mode 100644
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/statistics/SparkFunctionStatistics.scala
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenStreamingTestsTraits.scala
delete mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/extension/CustomerColumnarPreRules.scala
delete mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/extension/GlutenCustomerExtensionSuite.scala
delete mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/extension/TestFileSourceScanExecTransformer.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenAggregationQuerySuite.scala}
(63%)
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveCommandSuite.scala
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveComparisonTestSupport.scala
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveDDLSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveExplainSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHivePlanTest.scala}
(75%)
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveQuerySuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResolutionSuite.scala}
(72%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveResourcePathSupport.scala
(61%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLViewSuite.scala}
(75%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveScriptTransformationSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeReadWriteSuite.scala}
(74%)
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSerDeSuite.scala
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTypeCoercionSuite.scala}
(71%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDAFSuite.scala}
(75%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveUDFSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenObjectHashAggregateSuite.scala}
(73%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruneHiveTablePartitionsSuite.scala}
(73%)
copy
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenPruningSuite.scala
(65%)
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOnSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLMetricsSuite.scala}
(75%)
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenSQLQuerySuite.scala
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenTestHiveTables.scala
copy
gluten-ut/{spark40/src/test/scala/org/apache/spark/sql/streaming/GlutenFileStreamSinkSuite.scala
=>
spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenUDAQuerySuite.scala}
(65%)
create mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenWindowQuerySuite.scala
delete mode 100644
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/statistics/SparkFunctionStatistics.scala
create mode 100644
gluten-ut/test/src/test/scala/org/apache/gluten/config/GlutenRuntimeConfigSuite.scala
rename
backends-velox/src/main/scala/org/apache/spark/sql/execution/BaseArrowScanExec.scala
=>
shims/spark41/src/main/scala/org/apache/spark/sql/execution/streaming/MemoryStream.scala
(57%)
delete mode 100644 tools/gluten-te/.fixme
delete mode 100755 tools/gluten-te/centos/build.sh
delete mode 100755 tools/gluten-te/centos/buildenv.sh
delete mode 100755 tools/gluten-te/centos/cbash-build.sh
delete mode 100755 tools/gluten-te/centos/cbash-mount.sh
delete mode 100755 tools/gluten-te/centos/centos-7-deps.sh
delete mode 100755 tools/gluten-te/centos/centos-8-deps.sh
delete mode 100755 tools/gluten-te/centos/defaults.conf
delete mode 100644 tools/gluten-te/centos/dockerfile-build
delete mode 100755 tools/gluten-te/centos/dockerfile-buildenv
delete mode 100644
tools/gluten-te/centos/examples/buildhere-veloxbe-portable-libs/README.md
delete mode 100755
tools/gluten-te/centos/examples/buildhere-veloxbe-portable-libs/run-default.sh
delete mode 100755
tools/gluten-te/centos/examples/buildhere-veloxbe-portable-libs/run.sh
delete mode 100755 tools/gluten-te/centos/gha/gha-checkout/checkout.sh
delete mode 100755 tools/gluten-te/centos/gha/gha-checkout/clean.sh
delete mode 100755 tools/gluten-te/centos/gha/gha-checkout/exec.sh
delete mode 100755 tools/gluten-te/centos/gha/gha-checkout/scripts/init.sh
delete mode 100755 tools/gluten-te/centos/shared.sh
delete mode 100644 tools/gluten-te/ubuntu/README.md
delete mode 100755 tools/gluten-te/ubuntu/build.sh
delete mode 100755 tools/gluten-te/ubuntu/buildenv.sh
delete mode 100755 tools/gluten-te/ubuntu/cbash-build.sh
delete mode 100755 tools/gluten-te/ubuntu/cbash-mount.sh
delete mode 100644 tools/gluten-te/ubuntu/defaults.conf
delete mode 100644 tools/gluten-te/ubuntu/docker-compose.yml
delete mode 100644 tools/gluten-te/ubuntu/dockerfile-build
delete mode 100644 tools/gluten-te/ubuntu/dockerfile-buildenv
delete mode 100644 tools/gluten-te/ubuntu/dockerfile-tpc
delete mode 100755 tools/gluten-te/ubuntu/examples/buildhere-veloxbe-dev/run.sh
delete mode 100755
tools/gluten-te/ubuntu/examples/buildhere-veloxbe-dev/scripts/all.sh
delete mode 100644
tools/gluten-te/ubuntu/examples/buildhere-veloxbe-portable-libs/README.md
delete mode 100755
tools/gluten-te/ubuntu/examples/buildhere-veloxbe-portable-libs/run-default.sh
delete mode 100755
tools/gluten-te/ubuntu/examples/buildhere-veloxbe-portable-libs/run.sh
delete mode 100755
tools/gluten-te/ubuntu/examples/buildhere-veloxbe-portable-libs/scripts/all.sh
delete mode 100755 tools/gluten-te/ubuntu/examples/buildhere-veloxbe/run.sh
delete mode 100755 tools/gluten-te/ubuntu/gha/gha-checkout/checkout.sh
delete mode 100755 tools/gluten-te/ubuntu/gha/gha-checkout/clean.sh
delete mode 100755 tools/gluten-te/ubuntu/gha/gha-checkout/cp.sh
delete mode 100755 tools/gluten-te/ubuntu/gha/gha-checkout/exec.sh
delete mode 100755 tools/gluten-te/ubuntu/gha/gha-checkout/scripts/init.sh
delete mode 100755 tools/gluten-te/ubuntu/scripts/cmd.sh
delete mode 100755 tools/gluten-te/ubuntu/scripts/set-login-env.sh
delete mode 100755 tools/gluten-te/ubuntu/shared.sh
delete mode 100755 tools/gluten-te/ubuntu/tpc.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
