This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 839eec4d4e [GLUTEN-12564][CORE] Exclude
DefaultAllocationManagerFactory from gluten-package shade bundle (#12565)
839eec4d4e is described below
commit 839eec4d4e91ae69bc0716ac588d86857449971c
Author: Yuming Wang <[email protected]>
AuthorDate: Wed Aug 12 23:44:55 2026 +0800
[GLUTEN-12564][CORE] Exclude DefaultAllocationManagerFactory from
gluten-package shade bundle (#12565)
The shaded gluten-package.jar bundles DefaultAllocationManagerFactory.class
(both the CORE path in Arrow <= 15.x and the unsafe/netty subpaths in
Arrow >= 16.x) because the shade relocation excludes
org.apache.arrow.memory.** from relocation but still keeps the class
files in the bundle. The class is redundant — it always exists in the
upstream arrow-memory-unsafe / arrow-memory-netty jar that is on the
runtime classpath.
Starting with Arrow 16.0.0, CheckAllocator.check() inspects the URL of
DefaultAllocationManagerFactory.class and throws
"IllegalStateException: Unknown allocation manager type to infer"
when the URL does not contain "memory-core", "memory-unsafe", or
"memory-netty". When the class is loaded from inside
gluten-package.jar the URL matches none of these patterns (in
particular for Arrow <= 15.x where the class lives at the CORE path
org/apache/arrow/memory/DefaultAllocationManagerFactory.class).
This is currently silent on upstream CI because:
- Spark 3.3/3.4/3.5 community uses Arrow 12.0.1 whose CheckAllocator
predates the URL check.
- Spark 4.0/4.1 uses Arrow 18.1.0; the spark-4.0 profile overrides
arrow.version to 18.1.0 so gluten-package bundles the unsafe/-
subpath class whose URL contains "/org/apache/arrow/memory/unsafe/"
and matches the URL pattern.
But the bug manifests whenever a Spark distribution ships Arrow >=
16.0.0 while Gluten is built with arrow.version <= 15.x (e.g. a
downstream Spark 3.5 fork that upgrades Arrow from 12.0.1 to 18.3.0
while Gluten's spark-3.5 profile still uses arrow.version=15.0.0).
Fix: exclude the three known DefaultAllocationManagerFactory class
locations from the shaded bundle so the classloader always finds the
class in the upstream arrow-memory-*.jar, whose URL contains a
recognized substring. The class is functionally identical (same FACTORY
field providing an AllocationManager.Factory); the only behavioral
change is that Gluten stops forcing the unsafe allocator and lets
Arrow's own default selection apply (which is netty per
DefaultAllocationManagerOption.java:80).
Closes #12564.
---
package/pom.xml | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/package/pom.xml b/package/pom.xml
index 55318f85cf..513a3352bd 100644
--- a/package/pom.xml
+++ b/package/pom.xml
@@ -163,6 +163,25 @@
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>LICENSE.txt</exclude>
<exclude>NOTICE.txt</exclude>
+ <!--
+ Exclude DefaultAllocationManagerFactory from the shade
bundle.
+ Arrow 16.0.0+'s CheckAllocator inspects the URL of this
class
+ and throws "Unknown allocation manager type to infer" if
the
+ URL does not contain "memory-core", "memory-unsafe", or
+ "memory-netty". When the class is bundled inside
+ gluten-package.jar the URL does not match any of these
+ patterns (in particular for Arrow <= 15.x where the class
+ lives at org/apache/arrow/memory/DefaultAllocationManager
+ Factory.class), breaking allocator initialization
whenever
+ the runtime Arrow version is >= 16.0.0 (e.g. Spark
+ distributions that ship a newer Arrow than Gluten's
+ build-time arrow.version). Defer to the upstream
+ arrow-memory-unsafe / arrow-memory-netty jar which
provides
+ a recognizable URL.
+ -->
+
<exclude>org/apache/arrow/memory/DefaultAllocationManagerFactory.class</exclude>
+
<exclude>org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.class</exclude>
+
<exclude>org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.class</exclude>
</excludes>
</filter>
<filter>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
