This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 0b102a92c7 [GLUTEN-6887][VL] Daily Update Velox Version (2026_11_05)
(#13540)
0b102a92c7 is described below
commit 0b102a92c7c2bd67dfdc86668d9c54c1df702d38
Author: Gluten Performance Bot
<[email protected]>
AuthorDate: Thu Nov 5 18:11:20 2026 +0000
[GLUTEN-6887][VL] Daily Update Velox Version (2026_11_05) (#13540)
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_11_05)
Upstream Velox's New Commits:
0fdc35ff8 by Jimmy Lu, refactor: Add MultiStagePlan and build Axiom's plans
on it (#19993)
72c8d9021 by Matt Gara, fix(cudf): Support the intermediate step of a
global avg
09ecbcfb6 by shanhao-203, perf(nimble): Hybrid FlatMap copy per key -> copy
per group (#20021)
82cb0425a by tanjialiang, feat(nimble): Read multi-chunk streams in
StreamReader (#20079)
d2daad780 by tanjialiang, feat(nimble): Add BlockBitPackingV2 and move V1
to deprecated (#19959)
bcb783990 by Xiaoxuan, perf(index): Read the keys of a key cursor in
batches (#20040)
b28089cb5 by Richard Barnes, refactor: Use std::identity instead of
folly::identity (2/2)
2b1a2278f by Ke, feat(nimble): Default chunk stats min_avg_chunks to 0
(#20050)
587ce8953 by Pedro Pedreira, test(nimble): Re-enable the flat map as struct
E2E filter test (#20068)
b1e72846e by kevinwilfong, fix: Fix MergeJoin filter tracking across output
batch boundaries (#19947)
f4a075922 by Apurva Kumar, test(nimble): Run the uncached FSST memory pool
test instead of skipping it
ea94e1a45 by Paul Meng, fix(nimble): Stop overwriting buffers a slice still
views
67f9e7a2a by Pedro Pedreira, refactor: Detect sanitizers with folly instead
of __has_feature
8046f196b by GALI PREM SAGAR, feat(cudf): Honor FILTER masks for avg,
stddev and approx_distinct
e18158699 by XiaoDu, refactor(common): Share identity projections
63898d03c by Ke, perf(nimble): Skip chunks in sparse string dictionary
reads (#20004)
b48bacdd4 by macduan, fix(paimon): Make task cancellation deterministic
f5f84ad66 by peterenescu, fix(dwio): Select the in-map bits of filtered
rows in flat map readers
d88336369 by Morgan McLain-Smith, fix(exec): Free sorted aggregation input
of finished groups (#19750)
28e6372b4 by Wechar Yu, feat(parquet): Support size statistics
88d662804 by Richard Barnes, fix(iceberg): Guard error-message constant
[unused-const-variable]
d5d85c016 by Rong Ma, fix(arrow): Export Utf8View without modifying the
source vector
17170a5a1 by Maria Basmanova, docs: Describe how to update the version of a
dependency
2435b824d by Naveen Mahadevuni, refactor(iceberg): Tests to use Iceberg
table and column handle
841502a2f by rdtr, fix(parquet): Read a column with the UNKNOWN annotation
as any requested type
7ec01c0e6 by PRASHANT GOLASH, test(nimble): Fuzz filtered FSST and
dictionary reads (#19901)
b3833a48d by Daniel Bauer, fix(hive): Preserve zero-column scan output type
f4938c81c by Apurva Kumar, refactor(nimble): Remove the stripe-stats
section and stripe pruning
949fd583b by Maria Basmanova, build: Enable FAISS in the Linux Clang CI
build
4e773faae by Maria Basmanova, build: Build the Clang adapters CI job with
Clang
a40c93fe0 by hu hengrui, perf(filters): Optimize SIMD evaluation of
IsNotNull filters
5929f86ce by dependabot[bot], build(deps): Bump body-parser from 1.20.5 to
1.20.9 in /website
65d264d94 by Sutou Kouhei, build: Update simdjson to 4.3.0 (from 4.1.0)
09b7d580a by dependabot[bot], build(deps): Bump follow-redirects from
1.16.1 to 1.16.2 in /website
002494945 by Rui Mo, feat(parquet): Support dictionary-encoded FLBA strings
in Parquet reader
3d36a984d by dependabot[bot], build(deps): Bump postcss-selector-parser
from 6.1.4 to 6.1.5 in /website
ee28304db by dependabot[bot], build(deps): Bump joi from 17.13.8 to 17.13.9
in /website
9f0d5919a by dependabot[bot], build(deps): Bump fast-uri from 3.1.8 to
3.1.9 in /website
f6433b574 by Xiaoxuan, perf(nimble): Gather Dictionary alphabet values in
bulk (#20022)
f494258f2 by PRASHANT GOLASH, feat(nimble): Cache decoded strings for
repeated FSST payloads (#19819)
896faf1c1 by PRASHANT GOLASH, refactor: Stop including folly/Format.h
ab8ba883f by macduan, refactor(paimon): Decouple shared file infrastructure
from Hive
eb2241df7 by Xiaoxuan, Add a hierarchical cluster index over RocksDB
internal keys (#20023)
3a9c5a322 by bowen.yang, feat(spark): Add Spark arrays_overlap (#19805)
bfd81c220 by Pedro Pedreira, fix(expression): Stop copying unchanged nodes
when folding
aaaf76a0e by Wannuc-M, feat(rpc): Count overloaded in-transport retries
c2f70d4cf by PRASHANT GOLASH, test(nimble): Fuzz FSST lifecycle and
encoding views (#19900)
02ee2a433 by Ke, feat(nimble): Reject top-level nulls with chunk stats
563585c95 by bowen.yang, refactor(sparksql): Move JavaFloatingPoint to
sparksql (#19921)
7710fdce4 by Simon Eves, test(cudf): Re-enable decimal scan and unicode
LIKE tests
adb42d91b by Xiaoxuan, feat(index): Let a hierarchical cluster index
factory define its key format (#20012)
53d16900a by Wannuc-M, fix(trace): Redact cat_token from the traced plan
470da3804 by Ke, fix(nimble): Free the open chunk on skips to the stream
end (#20002)
839cb7beb by Xiaoxuan, feat(nimble): Let NimbleIndexProjector cache
encoding views (#20013)
ef70f8b4d by Rui Mo, fix(parquet): Fix reading array of row
1911d1762 by Apurva Kumar, perf(iceberg): Merge new deletes into an
existing deletion vector without re-sorting
76175211c by PRASHANT GOLASH, fix(nimble): Check final FSST compression
ratio (#19818)
f30da5918 by shanhao-203, perf(nimble): Avoid re-sorting sorted Hybrid
FlatMap keys
cd11e9e82 by Wannuc-M, feat(rpc): Report time spent pacing in the RPC rate
limiter (#20000)
fc421db3f by Denis Yaroshevskiy, perf(exec): Add StringView prefix fast
path to RowContainer string comparison (#19890)
1e11aefa3 by Apurva Kumar, perf(iceberg): Serialize deletion vectors in one
pass over sorted positions
68911ce83 by Apurva Kumar, docs(blog): Add deletion vector write benchmark
to Iceberg V3 post (#19998)
58c6924eb by PRASHANT GOLASH, feat(nimble): Compare negated FSST ranges
(#19816)
32bb15262 by Richard Barnes, build(serialization): Include <map> in
Serializable.h
e0c8817bf by GALI PREM SAGAR, feat(cudf): Run Presto's array_agg on the GPU
992cf2e58 by Apurva Kumar, perf(iceberg): Read deletion vectors without
expanding rows
541dd3d7e by Richard Barnes, refactor(tzdb): Use std::remove_cvref instead
of folly::remove_cvref
7fdcf6fe6 by dependabot[bot], build(ci): Bump
apache/infrastructure-actions/stash/save from
b1b1ab191498302581efdef55279706f79249c0e to
bd72894b857767d42834c186c90767d5c86624bb
0e1764863 by dependabot[bot], build(ci): Bump pypa/cibuildwheel from 4.2.0
to 4.3.4
e7ea3332b by Maria Basmanova, feat(type): Add RowType::unionWith overloads
for new fields
14cdfa2ad by Morgan McLain-Smith, fix(core): Move serialized children into
their parent (#19751)
77cdb0454 by Minni Mittal, feat(spark): Add encode function
eb67e0dbf by Maria Basmanova, refactor(vector): Template createFromVariants
on vector type
1004340de by Maria Basmanova, build(ci): Use the system FAISS from the CI
images
bcdd6686d by Maria Basmanova, feat: Add VELOX_NYI_CHECK macros
fb4a9f47a by Maria Basmanova, docs: Ask error messages to name the mistake
ad3654098 by Maria Basmanova, feat(parse): Add PlaceholderExpr (#19971)
488de5f69 by Maria Basmanova, feat(vectorindex): Add VECTOR_INDEX type,
cardinality and vector_count (#19969)
40499efb2 by Denis Yaroshevskiy, test: Add StringView and RowContainer
compare benchmarks (#19935)
186ef051c by Denis Yaroshevskiy, test: Add string comparison cases for
StringView and RowContainer (#19937)
fb0d5cf13 by PRASHANT GOLASH, perf(nimble): Reuse skipped encoded arena
chunks (#19962)
4f98e096a by PRASHANT GOLASH, feat(nimble): Filter FSST ranges without
decoding (#19817)
0057888c8 by PRASHANT GOLASH, fix(nimble): Count live strings in
first-chunk gate (#19961)
491fb91b1 by PRASHANT GOLASH, test(nimble): Generate ALP and ALP_RD Java
reader fixtures (#19903)
5978fc8ef by PRASHANT GOLASH, fix(nimble): Allocate exact cached FSST
buffers (#19825)
086c431d2 by GALI PREM SAGAR, feat(cudf): Project constants over a
zero-column input on the GPU
e67ed5114 by Apurva Kumar, docs(blog): Update Iceberg V3 benchmark results
2df03e5a9 by Maria Basmanova, refactor(vectorindex): Rename VectorIndexType
to VectorIndexKind (#19968)
98d666ad3 by zhichenxu-meta, feat(rpc): Add count limits to capacity-retry
hints (#19876)
dbfe6d418 by Maria Basmanova, fix(ci): Fail clang-tidy only on errors from
checks
c9c8919a7 by macduan, refactor(paimon): Consolidate table and column handle
state and serde
ecec5119e by Xiaoxuan, Index integer user keys hierarchically in Nimble
(#19966)
c21d1de4e by Hongze Zhang, refactor: Add SelectivityVector::complement and
simplify call sites
2463cebd0 by oerling, feat(torchwave): Execute every node from a step
program bound before the first run (#19836)
52f87fe00 by Apurva Kumar, docs(blog): Add delete benchmark results to
Iceberg V3 post
63b948ade by Sutou Kouhei, fix(build): Link gmock to the Nimble tests that
use it
ff72368a2 by Felipe Pessoto, fix(parquet): Widen DATE columns to
TIMESTAMP_UTC
d9a0c1c6c by hu hengrui, perf(HashTable): Prefetch accumulators when
probing a group table
34d0bae55 by Akshay Tayal, fix(spark): Throw overflow error in date_trunc
for out-of-range results
34ea27a1d by Akshay Tayal, feat: Add Spark normalize_nan_and_zero function
6d4e7d542 by Zhu, Lipeng, perf(functions): Inline Spark XxHash64 integer
hashing into the header
24614c3f9 by Matt Gara, test(cudf): Re-enable timestamp filter test, drop
return-type test
f95091078 by Maria Basmanova, fix(exec): Fix fixed-point hang when a reader
stops early (#19960)
fa0739103 by RindsSchei225e, refactor(nimble): Delete the legacy encoding
path (#19957)
3083dfc82 by Maria Basmanova, fix(build): Upgrade FAISS to 1.12.0 (from
1.11.0) and find OpenMP for system installs
b74f6d743 by Luis Peñaranda, feat(sparksql): Implement and register
integral ROUND with ANSI handling
8f6c94686 by Maria Basmanova, fix(exec): Wait for sub-tasks in
Task::taskDeletionFuture
9e25a47a5 by GALI PREM SAGAR, fix(build): Link gmock into the vector index
aggregate test
a65cd5b5a by Apurva Kumar, docs(blog): Add Iceberg V3 in Velox post (#19954)
---
.github/workflows/util/delta-spark-ut/known-failures.txt | 7 -------
.github/workflows/velox_backend_arm.yml | 7 +++++++
cpp/velox/CMakeLists.txt | 7 ++++++-
cpp/velox/substrait/SubstraitToVeloxPlan.cc | 11 +++++++++--
cpp/velox/utils/VeloxWriterUtils.cc | 4 ++++
dev/vcpkg/ports/simdjson/portfile.cmake | 2 +-
dev/vcpkg/ports/simdjson/vcpkg.json | 2 +-
ep/build-velox/src/get-velox.sh | 4 ++--
8 files changed, 30 insertions(+), 14 deletions(-)
diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt
b/.github/workflows/util/delta-spark-ut/known-failures.txt
index 92b199816c..333f26e7c0 100644
--- a/.github/workflows/util/delta-spark-ut/known-failures.txt
+++ b/.github/workflows/util/delta-spark-ut/known-failures.txt
@@ -484,10 +484,3 @@
org.apache.spark.sql.delta.storage.dv.DeletionVectorWriteDurabilitySuite#control
org.apache.spark.sql.delta.storage.dv.DeletionVectorWriteDurabilitySuite#the
kill-switch flag off does not apply the propagate-close fix
org.apache.spark.sql.delta.test.DeltaV2CDCStreamDeletionVectorSuite#cdc
streams with noop merge
org.apache.spark.sql.delta.test.DeltaV2CDCStreamSuite#cdc streams with noop
merge
-org.apache.spark.sql.delta.typewidening.TypeWideningAlterTableSuite#type
widening DATE -> TIMESTAMP_NTZ, partitioned=false
-org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicDSv2Suite#INSERT
- always automatic type widening DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicDSv2Suite#INSERT
- automatic type widening DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT
- always automatic type widening DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT
- automatic type widening DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.typewidening.TypeWideningMergeIntoSchemaEvolutionSuite#MERGE
- automatic type widening DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.typewidening.TypeWideningTableFeatureAdvancedSuite#drop
feature after type change DATE -> TIMESTAMP_NTZ
diff --git a/.github/workflows/velox_backend_arm.yml
b/.github/workflows/velox_backend_arm.yml
index 37ff33d445..a35c3a1b29 100644
--- a/.github/workflows/velox_backend_arm.yml
+++ b/.github/workflows/velox_backend_arm.yml
@@ -216,6 +216,13 @@ jobs:
df -a
bash dev/ci-velox-buildshared-centos-9.sh
ccache -s
+ - name: "Save ccache"
+ if: always()
+ uses:
apache/infrastructure-actions/stash/save@0ba14156c9f4c3cfbe4b0c9f36339ab0f8d81e53
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-centos9-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
+ include-hidden-files: true
- name: Run CPP unit test
run: |
cd ./cpp/build && ctest -V
diff --git a/cpp/velox/CMakeLists.txt b/cpp/velox/CMakeLists.txt
index 5f416eadcc..23dee4ac72 100644
--- a/cpp/velox/CMakeLists.txt
+++ b/cpp/velox/CMakeLists.txt
@@ -376,12 +376,17 @@ if(BUILD_TESTS)
facebook::velox::vector_fuzzer
INTERFACE facebook::velox::vector_fuzzer_util
facebook::velox::constrained_vector_generator)
+ import_library(
+ facebook::velox::local_exchange_source
+
${VELOX_BUILD_PATH}/velox/exec/tests/utils/libvelox_local_exchange_source.a)
import_library(
facebook::velox::exec_test_lib
${VELOX_BUILD_PATH}/velox/exec/tests/utils/libvelox_exec_test_lib.a)
target_link_libraries(
facebook::velox::exec_test_lib
- INTERFACE facebook::velox::vector_test_lib facebook::velox::vector_fuzzer
+ INTERFACE facebook::velox::local_exchange_source
+ facebook::velox::vector_test_lib
+ facebook::velox::vector_fuzzer
facebook::velox::dwio_common_test
facebook::velox::file_test_utils)
set(GLUTEN_VELOX_TEST_LIBRARIES facebook::velox::exec_test_lib)
diff --git a/cpp/velox/substrait/SubstraitToVeloxPlan.cc
b/cpp/velox/substrait/SubstraitToVeloxPlan.cc
index bb57cc6696..df076d0f6f 100644
--- a/cpp/velox/substrait/SubstraitToVeloxPlan.cc
+++ b/cpp/velox/substrait/SubstraitToVeloxPlan.cc
@@ -32,6 +32,7 @@
#endif
#include "velox/connectors/hive/HiveDataSink.h"
#include "velox/connectors/hive/iceberg/IcebergColumnHandle.h"
+#include "velox/connectors/hive/iceberg/IcebergTableHandle.h"
#include "velox/exec/TableWriter.h"
#include "velox/type/Type.h"
@@ -1714,8 +1715,14 @@ core::PlanNodePtr
SubstraitToVeloxPlanConverter::toVeloxPlan(const ::substrait::
#endif
}
common::SubfieldFilters subfieldFilters;
- tableHandle = std::make_shared<connector::hive::HiveTableHandle>(
- connectorId, "hive_table", std::move(subfieldFilters), remainingFilter,
dataColumns);
+ if (connectorId == connectorIds_.iceberg) {
+ // The Iceberg connector's data source requires an IcebergTableHandle.
+ tableHandle =
std::make_shared<connector::hive::iceberg::IcebergTableHandle>(
+ connectorId, "hive_table", std::move(subfieldFilters),
remainingFilter, dataColumns);
+ } else {
+ tableHandle = std::make_shared<connector::hive::HiveTableHandle>(
+ connectorId, "hive_table", std::move(subfieldFilters),
remainingFilter, dataColumns);
+ }
// Required subfields of scan columns (map-key pruning), from the
RequiredSubfieldsExtension the
// ScanMapKeyPruning rule packs into the ReadRel advanced extension. Each
path is transported as
diff --git a/cpp/velox/utils/VeloxWriterUtils.cc
b/cpp/velox/utils/VeloxWriterUtils.cc
index a2491836ea..703b97d9d4 100644
--- a/cpp/velox/utils/VeloxWriterUtils.cc
+++ b/cpp/velox/utils/VeloxWriterUtils.cc
@@ -148,6 +148,10 @@
std::shared_ptr<facebook::velox::dwio::common::WriterOptions> makeParquetWriteOp
enableWritePageIndex = boost::iequals(it->second, "true");
}
parquetOptions->enableWritePageIndex = enableWritePageIndex;
+ // Velox's writer emits Parquet size statistics by default, which parquet-mr
+ // used by Spark 3.x does not write. Disable them to keep file output (and
the
+ // size-based planning that depends on it) unchanged.
+ parquetOptions->sizeStatisticsLevel =
parquet::arrow::SizeStatisticsLevel::kNone;
writeOption->formatSpecificOptions = std::move(parquetOptions);
return writeOption;
}
diff --git a/dev/vcpkg/ports/simdjson/portfile.cmake
b/dev/vcpkg/ports/simdjson/portfile.cmake
index 6253089100..c0da4ee7cf 100644
--- a/dev/vcpkg/ports/simdjson/portfile.cmake
+++ b/dev/vcpkg/ports/simdjson/portfile.cmake
@@ -8,7 +8,7 @@ vcpkg_from_github(
HEAD_REF
master
SHA512
-
a7e683625fa7e17cfe39cf3915b13fa34c589a8bcace51a9a8a98948f4cbdb1fd4346a273a44d934aced4aad1da7bc695b3b4f9fea078ae4bdf2b48234e99de3
+
e0aee07decb7d6f82b4b32d3a6f41ec35c1902558c7278a6fad595fbe151a957fab4f16292a13a103a6d1f828003656f7ae855f725fcac96e91585425c868eb4
PATCHES
add-definition.patch)
diff --git a/dev/vcpkg/ports/simdjson/vcpkg.json
b/dev/vcpkg/ports/simdjson/vcpkg.json
index b7da2b0ab4..472d4c3d8a 100644
--- a/dev/vcpkg/ports/simdjson/vcpkg.json
+++ b/dev/vcpkg/ports/simdjson/vcpkg.json
@@ -1,6 +1,6 @@
{
"name": "simdjson",
- "version": "4.1.0",
+ "version": "4.3.0",
"description": "An extremely fast JSON library that can parse gigabytes of
JSON per second",
"homepage": "https://simdjson.org/";,
"license": "Apache-2.0",
diff --git a/ep/build-velox/src/get-velox.sh b/ep/build-velox/src/get-velox.sh
index 284f9b92a3..2245a84906 100755
--- a/ep/build-velox/src/get-velox.sh
+++ b/ep/build-velox/src/get-velox.sh
@@ -18,8 +18,8 @@ set -exu
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
VELOX_REPO=https://github.com/IBM/velox.git
-VELOX_BRANCH=dft-2026_11_01
-VELOX_ENHANCED_BRANCH=ibm-2026_11_01
+VELOX_BRANCH=dft-2026_11_05
+VELOX_ENHANCED_BRANCH=ibm-2026_11_05
VELOX_HOME=""
RUN_SETUP_SCRIPT=ON
ENABLE_ENHANCED_FEATURES=OFF
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
