This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 1f9c9a0eb8 [GLUTEN-6887][VL] Daily Update Velox Version (2026_10_06)
(#13209)
1f9c9a0eb8 is described below
commit 1f9c9a0eb84f57bab88b7ccd936b893765f56bdd
Author: Gluten Performance Bot
<[email protected]>
AuthorDate: Wed Oct 7 13:36:01 2026 +0100
[GLUTEN-6887][VL] Daily Update Velox Version (2026_10_06) (#13209)
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_10_06)
Upstream Velox's New Commits:
b2bff1217 by Xiaoxuan, test(file): Skip on io_uring setup ENOMEM
fa67ff41a by Maria Basmanova, fix(table-write): Handle merge with no input
30b9587a5 by Xiaoxuan, fix(cudf): Update batch concat stream type
7968827db by Xiaoxuan, feat(nimble): Use schema-key presence bitmaps
(#19318)
c1000e6bd by Daniel Bauer, feat(exec)!: Select the exchange client and
operator per node via a registry
081885b87 by Suryadev Sahadevan Rajesh, fix(nimble): Suspend buffered
writes during arbitration (#19303)
a019edc7d by Felipe Pessoto, fix(parquet): Preserve scalar matches in delta
decoding
006717813 by shanhao-203, feat(nimble): Allow a Hybrid FlatMap with only
the Default group (#19308)
303719a4d by Aditi Pandit, perf(exec): Apply abandon logic for
TopNRowNumber with no partition keys case
91ec90cd8 by Karthikeyan, feat(cudf): Add byte target for CudfBatchConcat
6f281b753 by Naveen Mahadevuni, feat(iceberg): Support querying changelog
table
de34f736d by Daniel Bauer, fix(parquet): Include column name in TIME_MICROS
errors
4cf478c86 by Muhammad Haseeb, fix(cudf): Allow case insensitive column
selection by CudfSplitReader
d9a64b889 by Reema, fix(cudf): Reject device resident input in CudfFromVelox
710e672fd by Minni Mittal, build(spark): Add libxml2 dependency for Spark
XPath functions
fff96449e by Julian Rossi, feat(cudf): Expose live-allocated GPU bytes via
statistics_resource_adaptor
f7079643f by Devavret Makkar, fix(cudf): Decouple local exchange hashing
from remote partitioning
c1e0eefc2 by Yabin Ma, fix(s3fs): Bump AWS_SDK_VERSION to 1.11.806 for FIPS
with OpenSSL 3.5
093b0ea67 by LingBin, feat(benchmark): Add --query_configs flag
08a9bc70e by Bradley Dice, feat(cudf): Update to cuDF 26.12 and adapt to
cuda::device_buffer null masks
a4bb8f6ee by dependabot[bot], build(deps): Bump axios from 1.18.0 to 1.20.0
in /website
a7973087f by Pedro Pedreira, perf(common): Drop the function-local static
in gather8Bits (#19302)
1816dac7b by Pedro Pedreira, refactor(vector): Move FlatMapVector key
setters out of the header (#19317)
5b73185d6 by Pedro Pedreira, test(vector): Add FlatMapVector key index
benchmark (#19316)
Signed-off-by: glutenperfbot <[email protected]>
* bump aws sdk version
Signed-off-by: Yuan <[email protected]>
* fix delta build native lib
Signed-off-by: Yuan <[email protected]>
* fix gpu
Signed-off-by: Yuan <[email protected]>
* fix format
Signed-off-by: Yuan <[email protected]>
* fix
Signed-off-by: Yuan <[email protected]>
* fix for vcpkg + dynamic openssl feature
Signed-off-by: Yuan <[email protected]>
* fix format
Signed-off-by: Yuan <[email protected]>
* fix mvn version
Signed-off-by: Yuan <[email protected]>
---------
Signed-off-by: glutenperfbot <[email protected]>
Signed-off-by: Yuan <[email protected]>
Co-authored-by: glutenperfbot <[email protected]>
Co-authored-by: Yuan <[email protected]>
---
.github/workflows/delta_spark_ut.yml | 33 +++++++++++-------
build/mvn | 27 +++++++-------
cpp/velox/CMakeLists.txt | 6 ++++
cpp/velox/utils/GpuBufferBatchResizer.cc | 23 ++++++------
dev/vcpkg/ports/aws-sdk-cpp/portfile.cmake | 2 +-
dev/vcpkg/ports/aws-sdk-cpp/vcpkg.json | 2 +-
ep/build-velox/src/get-velox.sh | 4 +--
package/pom.xml | 56 +++++++++++++-----------------
8 files changed, 83 insertions(+), 70 deletions(-)
diff --git a/.github/workflows/delta_spark_ut.yml
b/.github/workflows/delta_spark_ut.yml
index a1d4d50211..043a65acad 100644
--- a/.github/workflows/delta_spark_ut.yml
+++ b/.github/workflows/delta_spark_ut.yml
@@ -203,29 +203,38 @@ jobs:
# This workflow always builds its own native lib -- there is no caller to
# provide one, and the `paths:` filter already decided whether the run
happens.
runs-on: ubuntu-22.04
+ container: apache/gluten:vcpkg-almalinux-8-gcc13
steps:
+ - name: Install CI tools (temporary - until image is rebuilt with
gh/jq/bison/python3.9)
+ run: |
+ dnf install -y jq bison tzdata
+ dnf module -y install python39 && alternatives --set python3
/usr/bin/python3.9
+ dnf config-manager --add-repo
https://cli.github.com/packages/rpm/gh-cli.repo
+ dnf install -y gh --setopt=gpgcheck=0
- uses: actions/checkout@v7
with:
ref: ${{ env.DELTA_CHECKOUT_REF }}
- name: Get Ccache from Apache Stash
+ id: ccache-stash
uses:
apache/infrastructure-actions/stash/restore@69afc125e535c4c41e7f1b7470f583087e0f344b
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-almalinux8-release-default-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from actions/cache (fallback on Stash miss)
+ if: steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-almalinux8-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-almalinux8-release-default
- name: Build Gluten native libraries
run: |
- docker run -v $GITHUB_WORKSPACE:/work -w /work
apache/gluten:vcpkg-almalinux-8-gcc13 bash -c "
- set -e
- dnf install -y bison tzdata
- df -a
- cd /work
- export CCACHE_DIR=/work/.ccache
- export CCACHE_MAXSIZE=1G
- mkdir -p /work/.ccache
- ccache -sz
- bash dev/ci-velox-buildstatic-centos-8.sh
- ccache -s
- "
+ df -a
+ export CCACHE_MAXSIZE=1G
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-8.sh
+ ccache -s
- name: Upload native build diagnostics
if: failure()
uses: actions/upload-artifact@v7
diff --git a/build/mvn b/build/mvn
index 550f1d1435..b66df5c907 100755
--- a/build/mvn
+++ b/build/mvn
@@ -29,18 +29,6 @@ MVN_BIN=""
# Read Maven version from pom.xml if not set
install_mvn() {
- # Check for system Maven first
- local SYSTEM_MVN
- SYSTEM_MVN="$(command -v mvn)"
-
- if [ -n "$SYSTEM_MVN" ]; then
- local MVN_DETECTED_VERSION
- MVN_DETECTED_VERSION="$(mvn --version 2>/dev/null | grep '^Apache
Maven' | awk '{print $3}')"
- echo "Using system Maven: $SYSTEM_MVN (version $MVN_DETECTED_VERSION)"
>&2
- MVN_BIN="$SYSTEM_MVN"
- return 0
- fi
-
# Get Maven version from pom.xml using grep and sed
local VERSION
VERSION=$(grep "<maven.version>" "${GLUTEN_HOME}/pom.xml" | head -n1 | \
@@ -51,6 +39,21 @@ install_mvn() {
exit 1
fi
+ # Use system Maven only if it matches the required version
+ local SYSTEM_MVN
+ SYSTEM_MVN="$(command -v mvn)"
+
+ if [ -n "$SYSTEM_MVN" ]; then
+ local MVN_DETECTED_VERSION
+ MVN_DETECTED_VERSION="$(mvn --version 2>/dev/null | grep '^Apache
Maven' | awk '{print $3}')"
+ if [ "$MVN_DETECTED_VERSION" = "$VERSION" ]; then
+ echo "Using system Maven: $SYSTEM_MVN (version
$MVN_DETECTED_VERSION)" >&2
+ MVN_BIN="$SYSTEM_MVN"
+ return 0
+ fi
+ echo "System Maven $SYSTEM_MVN (version $MVN_DETECTED_VERSION) does
not match required version $VERSION" >&2
+ fi
+
local MVN_LOCAL_BIN="${MVN_DOWNLOAD_DIR}/apache-maven-${VERSION}/bin/mvn"
if [ ! -f "${MVN_LOCAL_BIN}" ]; then
diff --git a/cpp/velox/CMakeLists.txt b/cpp/velox/CMakeLists.txt
index 550b1c92bf..71bb0f9a60 100644
--- a/cpp/velox/CMakeLists.txt
+++ b/cpp/velox/CMakeLists.txt
@@ -383,6 +383,12 @@ target_link_libraries(velox PUBLIC
${GLUTEN_ROARING_LINK_LIBRARY})
find_re2()
target_link_libraries(velox PUBLIC ${RE2_LIBRARY})
+if(ENABLE_GLUTEN_VCPKG)
+ # FBThrift -> mvfst -> fizz -> OpenSSL; with VCPKG_DYNAMIC_OPENSSL=ON vcpkg
+ # only provides libssl.so/libcrypto.so, so resolve it before forcing ".a".
+ find_package(OpenSSL REQUIRED)
+endif()
+
set(CMAKE_FIND_LIBRARY_SUFFIXES_BCK ${CMAKE_FIND_LIBRARY_SUFFIXES})
set(CMAKE_FIND_LIBRARY_SUFFIXES ".a")
diff --git a/cpp/velox/utils/GpuBufferBatchResizer.cc
b/cpp/velox/utils/GpuBufferBatchResizer.cc
index ea7b947c07..5f56ba80e3 100644
--- a/cpp/velox/utils/GpuBufferBatchResizer.cc
+++ b/cpp/velox/utils/GpuBufferBatchResizer.cc
@@ -25,6 +25,8 @@
#include <arrow/buffer.h>
+#include <algorithm>
+
#include <cuda_runtime.h>
#include <cudf/column/column.hpp>
#include <cudf/column/column_factories.hpp>
@@ -48,14 +50,14 @@ struct DispatchColumn {
const int32_t numRows;
int32_t bufferIdx = 0;
- std::unique_ptr<rmm::device_buffer> getMaskBuffer(const
std::shared_ptr<arrow::Buffer>& buffer) {
+ cuda::device_buffer<std::byte> getMaskBuffer(const
std::shared_ptr<arrow::Buffer>& buffer) {
if (buffer == nullptr || buffer->size() == 0) {
- return std::make_unique<rmm::device_buffer>(0, stream, mr);
+ return cudf::create_null_mask(0, cudf::mask_state::UNALLOCATED, stream,
mr);
}
- auto mask = std::make_unique<rmm::device_buffer>(buffer->size(), stream,
mr);
- CUDF_CUDA_TRY(
- cudaMemcpyAsync(mask->data(), buffer->data(), buffer->size(),
cudaMemcpyHostToDevice, stream.value()));
+ auto mask = cudf::create_null_mask(numRows,
cudf::mask_state::UNINITIALIZED, stream, mr);
+ auto copySize = std::min(static_cast<size_t>(buffer->size()), mask.size());
+ CUDF_CUDA_TRY(cudaMemcpyAsync(mask.data(), buffer->data(), copySize,
cudaMemcpyHostToDevice, stream.value()));
return mask;
}
@@ -78,8 +80,8 @@ struct DispatchColumn {
cudf::data_type cudfType{typeId};
size_t nullCount = nulls == nullptr || nulls->size() == 0
? 0
- : cudf::null_count(static_cast<const
cudf::bitmask_type*>(nullBuf->data()), 0, numRows, stream);
- return std::make_unique<cudf::column>(cudfType, numRows,
std::move(dataBuf), std::move(*nullBuf), nullCount);
+ : cudf::null_count(reinterpret_cast<const
cudf::bitmask_type*>(nullBuf.data()), 0, numRows, stream);
+ return std::make_unique<cudf::column>(cudfType, numRows,
std::move(dataBuf), std::move(nullBuf), nullCount);
}
/// We can optimize it in shuffle writer side, returns the offset buffer
instead of length buffer.
@@ -93,7 +95,7 @@ struct DispatchColumn {
cudaMemcpyAsync(offsetBuf.data(), offsets->data(), offsets->size(),
cudaMemcpyHostToDevice, stream.value()));
// --- 3. Empty null mask (no nulls in offset column) ---
- rmm::device_buffer nullBuf(0, stream, mr);
+ auto nullBuf = cudf::create_null_mask(0, cudf::mask_state::UNALLOCATED,
stream, mr);
// --- 4. Create cudf::column ---
return std::make_unique<cudf::column>(
@@ -118,15 +120,14 @@ struct DispatchColumn {
// === Step 3: create cudf::column ===
size_t nullCount = nulls == nullptr || nulls->size() == 0
? 0
- : cudf::null_count(static_cast<const
cudf::bitmask_type*>(mask->data()), 0, numRows, stream);
+ : cudf::null_count(reinterpret_cast<const
cudf::bitmask_type*>(mask.data()), 0, numRows, stream);
auto offsetColumn = getOffsetsColumn(offsets);
rmm::device_buffer chars(valueBuffer->size(), stream, mr);
CUDF_CUDA_TRY(cudaMemcpyAsync(
chars.data(), valueBuffer->data_as<uint8_t>(), chars.size(),
cudaMemcpyDefault, stream.value()));
- return cudf::make_strings_column(
- numRows, std::move(offsetColumn), std::move(chars), nullCount,
std::move(*mask.release()));
+ return cudf::make_strings_column(numRows, std::move(offsetColumn),
std::move(chars), nullCount, std::move(mask));
}
};
diff --git a/dev/vcpkg/ports/aws-sdk-cpp/portfile.cmake
b/dev/vcpkg/ports/aws-sdk-cpp/portfile.cmake
index 3ec3854b0c..847fca8f77 100644
--- a/dev/vcpkg/ports/aws-sdk-cpp/portfile.cmake
+++ b/dev/vcpkg/ports/aws-sdk-cpp/portfile.cmake
@@ -4,7 +4,7 @@ vcpkg_from_github(
OUT_SOURCE_PATH SOURCE_PATH
REPO aws/aws-sdk-cpp
REF "${VERSION}"
- SHA512
aee9e7e733f4d54d58dc28b59374cb1457acde8634fc3319a22865ccece4e46f34ef96c0778a9ae3b6b1738cad71429a164913b1470de4dc32954a7c7bd04ff0
+ SHA512
321993dee4bc4b34c5e2dda5c646f8ca190a15b850893ee49b88d32c5178c20b4210d821c8a6c7180f6d3a910c9a02de7093e1af48b9fbcd984210446ffb8ede
PATCHES
fix-aws-root.patch
lock-curl-http-and-tls-settings.patch
diff --git a/dev/vcpkg/ports/aws-sdk-cpp/vcpkg.json
b/dev/vcpkg/ports/aws-sdk-cpp/vcpkg.json
index c8bd95b5f3..93f56f82b6 100644
--- a/dev/vcpkg/ports/aws-sdk-cpp/vcpkg.json
+++ b/dev/vcpkg/ports/aws-sdk-cpp/vcpkg.json
@@ -1,7 +1,7 @@
{
"$note": "Automatically generated by generateFeatures.ps1",
"name": "aws-sdk-cpp",
- "version": "1.11.654",
+ "version": "1.11.806",
"port-version": 1,
"description": "AWS SDK for C++",
"homepage": "https://github.com/aws/aws-sdk-cpp";,
diff --git a/ep/build-velox/src/get-velox.sh b/ep/build-velox/src/get-velox.sh
index e1b63b0103..256f341ab1 100755
--- a/ep/build-velox/src/get-velox.sh
+++ b/ep/build-velox/src/get-velox.sh
@@ -18,8 +18,8 @@ set -exu
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
VELOX_REPO=https://github.com/IBM/velox.git
-VELOX_BRANCH=dft-2026_10_05
-VELOX_ENHANCED_BRANCH=ibm-2026_10_05
+VELOX_BRANCH=dft-2026_10_06
+VELOX_ENHANCED_BRANCH=ibm-2026_10_06
VELOX_HOME=""
RUN_SETUP_SCRIPT=ON
ENABLE_ENHANCED_FEATURES=OFF
diff --git a/package/pom.xml b/package/pom.xml
index bb2f7003d0..09c1d5fa13 100644
--- a/package/pom.xml
+++ b/package/pom.xml
@@ -194,6 +194,31 @@
</arguments>
</configuration>
</execution>
+ <execution>
+ <!--
+ Verify that the bundled Arrow C-Data classes have public method
+ signatures referencing the unshaded Apache Arrow API
+ (org.apache.arrow.memory.*, org.apache.arrow.vector.*) and not
+ the gluten-shaded copies. Catches a regression where shading
+ relocates classes that the unshaded org.apache.arrow.c.* API
+ transitively depends on, producing a bundle whose public Arrow
+ C-Data API is incompatible with vanilla Apache Arrow callers
+ (gluten#12225).
+ -->
+ <id>verify-arrow-c-shading</id>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <phase>verify</phase>
+ <configuration>
+
<executable>${project.basedir}/../dev/check-arrow-c-shading.sh</executable>
+ <arguments>
+
<argument>${project.build.directory}/${project.build.finalName}.jar</argument>
+ <argument>${gluten.shade.packageName}</argument>
+ <argument>${arrow.deps.scope}</argument>
+ </arguments>
+ </configuration>
+ </execution>
</executions>
</plugin>
<plugin>
@@ -302,37 +327,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>exec-maven-plugin</artifactId>
- <executions>
- <execution>
- <!--
- Verify that the bundled Arrow C-Data classes have public method
- signatures referencing the unshaded Apache Arrow API
- (org.apache.arrow.memory.*, org.apache.arrow.vector.*) and not
- the gluten-shaded copies. Catches a regression where shading
- relocates classes that the unshaded org.apache.arrow.c.* API
- transitively depends on, producing a bundle whose public Arrow
- C-Data API is incompatible with vanilla Apache Arrow callers
- (gluten#12225).
- -->
- <id>verify-arrow-c-shading</id>
- <goals>
- <goal>exec</goal>
- </goals>
- <phase>verify</phase>
- <configuration>
-
<executable>${project.basedir}/../dev/check-arrow-c-shading.sh</executable>
- <arguments>
-
<argument>${project.build.directory}/${project.build.finalName}.jar</argument>
- <argument>${gluten.shade.packageName}</argument>
- <argument>${arrow.deps.scope}</argument>
- </arguments>
- </configuration>
- </execution>
- </executions>
- </plugin>
</plugins>
</build>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
