This is an automated email from the ASF dual-hosted git repository.
chengchengjin pushed a change to branch cudf
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
discard ca73ea23bb fix docker
discard 1d590461e9 update velox branch
omit 828b0bbb0d update CMakeLists.txt
omit d9736fc01e update Dockerfile
omit b260799a07 Revert "update the velox branch to gpu branch"
omit 622f03725f update docker branch
omit 73f12d1c3d update the velox branch to gpu branch
omit 42014f14cc support cudf
add a0e27af54c [GLUTEN-6887][VL] Daily Update Velox Version (2025_04_04)
(#9226)
add 637bc990cb [GLUTEN-8821][VL] Update aggregate/generator/window support
doc and script (#8971)
add 4814d9fcc6 [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250405)
(#9232)
add 106fb3f2a6 [GLUTEN-6887][VL] Daily Update Velox Version (2025_04_05)
(#9233)
add 5eecd8d5d0 [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250406)
(#9234)
add 55309a2105 [VL] Change to use Velox's wget_and_untar in
setup-centos7.sh (#9207)
add 90e2d36e61 [GLUTEN-9196][CH] Use wide-table aggregation to eliminate
multi-table joins (#9155)
add 744d854c06 [GLUTEN-9149][CORE] Remove Spark-specific code from
JniLibLoader & JniWorkspace (#9150)
add b7761ebb75 [GLUTEN-6887][VL] Daily Update Velox Version (2025_04_06)
(#9235)
add a5ad4a4f1e [VL][CI] Change to use JDK-17 for Spark 3.3/3.4/3.5 tests
(#9209)
add ee9146df22 [CORE][VL] Hide child nodes from implementations of
`OffloadSingleNode` (#9220)
add aa89601296 [GLUTEN-9008][VL] Support json_object_keys function (#9009)
add 0be312ac07 [GLUTEN-9239][CH] Support JDK17 for the CH backend (#9242)
add d27959c1d2 [GLUTEN-9152][CORE] Avoid unnecessary serialization of
hadoop conf (#9153)
add d5472d1a2b [GLUTEN-9240][VL] Write NULL value into relation in gluten
unit tests (#9241)
add cf731edb43 [GLUTEN-6887][VL] Daily Update Velox Version (2025_04_08)
(#9251)
add 1825e39ca0 [VL][CI] Bump to use ubuntu-22.04 runner (#9262)
add 9bb48b2367 Fix diff on parse_url and refactor SparkParseURL (#9179)
add ab4c4b81d2 [CORE] Decrease offheap memory size in resource profile for
whole stage fallback case (#8911)
add a4b230bbbf [GLUTEN-9205][CH] Support deletion vector native write
(#9248)
add 690b689cb4 [VL] Delete global reference to a class object in JNI unload
add 262278348f [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250409)
(#9264)
add fdd8371406 [GLUTEN-9245][VL] Fix partial project expression contains
subquery (#9259)
new 558f4a4718 support cudf
new ae81a95266 update the velox branch to gpu branch
new 81890293c0 update docker branch
new cf1462d126 Revert "update the velox branch to gpu branch"
new 64327b26c1 update Dockerfile
new 59d4f41f8a update CMakeLists.txt
new e4717a700e update velox branch
new 24ea32be9f fix docker
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ca73ea23bb)
\
N -- N -- N refs/heads/cudf (24ea32be9f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build_bundle_package.yml | 4 +-
.github/workflows/velox_backend.yml | 320 +++----
.github/workflows/velox_backend_cache.yml | 8 +-
.github/workflows/velox_nightly.yml | 2 +-
.gitignore | 3 +-
backends-clickhouse/pom.xml | 4 +
.../gluten/vectorized/DeltaWriterJNIWrapper.java | 44 +
.../gluten/sql/shims/delta32/Delta32Shims.scala | 7 +
.../commands/DMLWithDeletionVectorsHelper.scala | 701 +++++++++++++++
.../execution/DeletionVectorWriteTransformer.scala | 175 ++++
.../GlutenDeltaParquetDeletionVectorSuite.scala | 118 ++-
.../apache/gluten/component/CHDeltaComponent.scala | 7 +
.../org/apache/gluten/sql/shims/DeltaShims.scala | 6 +-
.../vectorized/CHNativeExpressionEvaluator.java | 4 +
.../vectorized/ExpressionEvaluatorJniWrapper.java | 5 +-
.../gluten/backendsapi/clickhouse/CHBackend.scala | 7 +-
.../backendsapi/clickhouse/CHListenerApi.scala | 20 +-
.../gluten/backendsapi/clickhouse/CHRuleApi.scala | 3 +
.../extension/JoinAggregateToAggregateUnion.scala | 996 +++++++++++++++++++++
.../org/apache/gluten/utils/CHExpressionUtil.scala | 3 +-
.../execution/GlutenEliminateJoinSuite.scala | 472 ++++++++++
.../GlutenClickHouseTPCHSaltNullParquetSuite.scala | 12 +
.../gluten/component/VeloxDeltaComponent.scala | 1 +
.../gluten/component/VeloxHudiComponent.scala | 2 +-
.../gluten/backendsapi/velox/VeloxBackend.scala | 10 +-
.../backendsapi/velox/VeloxListenerApi.scala | 35 +-
.../gluten/backendsapi/velox/VeloxRuleApi.scala | 2 +-
.../execution/ColumnarPartialProjectExec.scala | 13 +-
.../apache/gluten/utils/ParquetMetadataUtils.scala | 14 +-
.../execution/JsonFunctionsValidateSuite.scala | 70 +-
.../execution/ScalarFunctionsValidateSuite.scala | 49 +-
.../gluten/expression/UDFPartialProjectSuite.scala | 9 +
cpp-ch/clickhouse.version | 4 +-
.../AggregateFunctionDVRoaringBitmap.h | 2 +-
.../ObjectStorages/MetadataStorageFromRocksDB.cpp | 5 +
.../ObjectStorages/MetadataStorageFromRocksDB.h | 1 +
cpp-ch/local-engine/Functions/SparkParseURL.cpp | 398 +++-----
.../Storages/MergeTree/SparkMergeTreeMeta.cpp | 2 +-
.../Storages/MergeTree/SparkStorageMergeTree.h | 7 +-
.../Delta/Bitmap/DeltaDVRoaringBitmapArray.cpp | 16 +-
.../Delta/Bitmap/DeltaDVRoaringBitmapArray.h | 8 +-
.../Storages/SubstraitSource/Delta/DeltaUtil.cpp | 80 ++
.../SubstraitSource/Delta/DeltaUtil.h} | 37 +-
.../Storages/SubstraitSource/Delta/DeltaWriter.cpp | 252 ++++++
.../Storages/SubstraitSource/Delta/DeltaWriter.h | 84 ++
.../Storages/SubstraitSource/ORCFormatFile.cpp | 3 +-
cpp-ch/local-engine/local_engine_jni.cpp | 83 +-
.../tests/gtest_clickhouse_roaring_bitmap.cpp | 2 +-
cpp-ch/local-engine/tests/gtest_iceberge_test.cpp | 7 -
cpp/velox/jni/VeloxJniWrapper.cc | 1 +
docs/developers/NewToGluten.md | 18 +-
docs/velox-backend-aggregate-function-support.md | 71 ++
docs/velox-backend-generator-function-support.md | 16 +
docs/velox-backend-scalar-function-support.md | 80 +-
docs/velox-backend-support-progress.md | 76 +-
docs/velox-backend-window-function-support.md | 18 +
ep/build-velox/src/setup-centos7.sh | 7 -
.../java/org/apache/gluten/jni/JniLibLoader.java | 11 -
.../java/org/apache/gluten/jni/JniWorkspace.java | 26 +-
.../columnar/offload/OffloadSingleNode.scala | 85 +-
.../gluten/backendsapi/BackendSettingsApi.scala | 5 +-
.../execution/BasicScanExecTransformer.scala | 6 +-
.../gluten/execution/WholeStageTransformer.scala | 8 +-
.../gluten/expression/ExpressionConverter.scala | 27 +-
.../gluten/expression/ExpressionMappings.scala | 1 +
.../columnar/offload/OffloadSingleNodeRules.scala | 2 +-
.../GlutenAutoAdjustStageResourceProfile.scala | 13 +-
.../org/apache/spark/sql/GlutenTestUtils.scala | 3 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 1 -
.../utils/clickhouse/ClickHouseTestSettings.scala | 1 -
.../gluten/utils/velox/VeloxTestSettings.scala | 9 +-
.../utils/clickhouse/ClickHouseTestSettings.scala | 1 -
.../utils/clickhouse/ClickHouseTestSettings.scala | 1 -
.../utils/velox/VeloxSQLQueryTestSettings.scala | 17 -
.../apache/gluten/expression/ExpressionNames.scala | 1 +
.../gluten/sql/shims/spark34/Spark34Shims.scala | 4 +-
.../gluten/sql/shims/spark35/Spark35Shims.scala | 4 +-
tools/scripts/gen-function-support-docs.py | 357 +++++---
78 files changed, 4071 insertions(+), 916 deletions(-)
create mode 100644
backends-clickhouse/src-delta-32/main/java/org/apache/gluten/vectorized/DeltaWriterJNIWrapper.java
create mode 100644
backends-clickhouse/src-delta-32/main/scala/org/apache/spark/sql/delta/commands/DMLWithDeletionVectorsHelper.scala
create mode 100644
backends-clickhouse/src-delta-32/main/scala/org/apache/spark/sql/execution/DeletionVectorWriteTransformer.scala
create mode 100644
backends-clickhouse/src/main/scala/org/apache/gluten/extension/JoinAggregateToAggregateUnion.scala
create mode 100644
backends-clickhouse/src/test/scala/org/apache/gluten/execution/GlutenEliminateJoinSuite.scala
create mode 100644
cpp-ch/local-engine/Storages/SubstraitSource/Delta/DeltaUtil.cpp
copy cpp-ch/local-engine/{Shuffle/ShuffleWriter.h =>
Storages/SubstraitSource/Delta/DeltaUtil.h} (59%)
create mode 100644
cpp-ch/local-engine/Storages/SubstraitSource/Delta/DeltaWriter.cpp
create mode 100644
cpp-ch/local-engine/Storages/SubstraitSource/Delta/DeltaWriter.h
create mode 100644 docs/velox-backend-aggregate-function-support.md
create mode 100644 docs/velox-backend-generator-function-support.md
create mode 100644 docs/velox-backend-window-function-support.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
