wangyum opened a new pull request, #12565: URL: https://github.com/apache/gluten/pull/12565
## What changes are proposed in this pull request? Exclude `DefaultAllocationManagerFactory.class` (in all three known Arrow locations — CORE path, `unsafe/` subpath, `netty/` subpath) from the `gluten-package.jar` shade bundle. The class is bundled today because the shade plugin's relocation excludes `org.apache.arrow.memory.**` from relocation but still includes the class files in the bundle. The class is **redundant** — the same class is always available on the runtime classpath via the upstream `arrow-memory-unsafe` / `arrow-memory-netty` jar declared as a `gluten-arrow` dependency. Starting with Arrow 16.0.0, `CheckAllocator.check()` inspects the URL of `DefaultAllocationManagerFactory.class` and throws `IllegalStateException: Unknown allocation manager type to infer` when the URL does not contain `memory-core`, `memory-unsafe`, or `memory-netty`. When the class is loaded from inside `gluten-package.jar` the URL matches none of these patterns (especially for Arrow <= 15.x where the class lives at the CORE path `org/apache/arrow/memory/DefaultAllocationManagerFactory.class`). This is currently silent on upstream CI because: - Spark 3.3/3.4/3.5 community uses Arrow 12.0.1 whose `CheckAllocator` predates the URL check. - Spark 4.0/4.1 uses Arrow 18.1.0; the `spark-4.0` profile overrides `arrow.version` to 18.1.0 so `gluten-package` bundles the `unsafe/`-subpath class whose URL contains `/org/apache/arrow/memory/unsafe/` and matches. But the bug manifests whenever a Spark distribution ships Arrow >= 16.0.0 while Gluten is built with `arrow.version` <= 15.x — e.g. a downstream Spark 3.5 fork that upgrades Arrow from 12.0.1 to 18.3.0 while Gluten's `spark-3.5` profile still uses `arrow.version=15.0.0`. Removing the duplicate class lets the classloader find the upstream Arrow jar, whose URL contains a recognized substring. The class is functionally identical (same `FACTORY` field providing an `AllocationManager.Factory`). The only behavioral change is that Gluten stops forcing the *unsafe* allocator and lets Arrow's own default selection apply (which is `netty` per `DefaultAllocationManagerOption.java:80`). ## How was this patch tested? - Verified the bundled jar before/after the change confirms the class is no longer present in `gluten-package.jar`. - Verified the upstream `arrow-memory-netty-*.jar` / `arrow-memory-unsafe-*.jar` always provides a `DefaultAllocationManagerFactory.class` whose URL matches `CheckAllocator`'s pattern (for Arrow >= 16.x). For Arrow <= 15.x the `CheckAllocator` does not perform URL inspection, so the fix is a no-op there. - Existing `gluten-ut` suites continue to pass (no source-code change). Closes #12564. --- ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
