sezruby commented on PR #12226: URL: https://github.com/apache/gluten/pull/12226#issuecomment-4616505224
> Could you please also share the arrow usage used in your case - does those libs also shade their arrow API? Sure — the trigger is [Lance Java](https://github.com/lance-format/lance/tree/main/java) (`org.lance.spark.write.LanceDataWriter`), which uses the standard Arrow C-Data interface to ship Spark `RecordBatch`es into a native (Rust) Lance writer. The relevant call is: ```java try (ArrowArrayStream stream = ArrowArrayStream.allocateNew(allocator)) { Data.exportArrayStream(allocator, sparkArrowReader, stream); Fragment.create(datasetUri, stream, params); } ``` `allocator` here is `org.apache.arrow.memory.BufferAllocator` (the public Arrow type). Lance Java does **not** shade Arrow — it depends on `org.apache.arrow.*` directly. Arrow Java 15.0.2 is pulled in transitively via `lance-core`. Three observations from running this on a Spark cluster that ships the gluten-velox bundle on the AppClassLoader: 1. JVM resolves `org.apache.arrow.c.ArrowArrayStream` to gluten's bundled copy (gluten on AppClassLoader, our jar on the child `MutableURLClassLoader`). 2. Gluten's `ArrowArrayStream.allocateNew` signature is `allocateNew(org.apache.gluten.shaded.org.apache.arrow.memory.BufferAllocator)`. 3. Lance's call passes `org.apache.arrow.memory.BufferAllocator` → `NoSuchMethodError`. Other public-API Arrow C-Data callers on the same classpath would hit the same shape — Iceberg's Arrow vector layer, Snowflake JDBC's Arrow result decoder, etc. — but I haven't tested those directly, just calling out that the issue isn't Lance-specific. Workarounds tried at the user-jar layer before this PR: `userClassPathFirst=true` (cascading `LinkageError` on Arrow `Field` because Spark internals on AppClassLoader pre-load Arrow before the user-jar takes over); shade-relocating `org.apache.arrow.c → org.lance.shaded.arrow.c` in the user fat jar (gets past `NoSuchMethodError` but trips `UnsatisfiedLinkError` because `libarrow_cdata_jni.so` has the JNI natives bound to `org/apache/arrow/c/jni/JniWrapper`). Neither is a clean user-side fix, which is why this needs to be addressed at gluten's shading layer (or via the unbundling path you proposed above). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
