shubham19may commented on code in PR #14499:
URL: https://github.com/apache/iceberg/pull/14499#discussion_r2495553609
##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowAllocation.java:
##########
@@ -21,14 +21,29 @@
import org.apache.arrow.memory.RootAllocator;
public class ArrowAllocation {
+ private static final String ALLOCATION_MANAGER_TYPE_PROPERTY =
+ "arrow.memory.allocation.manager.type";
+
static {
- ROOT_ALLOCATOR = new RootAllocator(Long.MAX_VALUE);
+ if (System.getProperty(ALLOCATION_MANAGER_TYPE_PROPERTY) == null) {
+ System.setProperty(ALLOCATION_MANAGER_TYPE_PROPERTY, "Netty");
+ }
Review Comment:
well the issue is, arrow's auto-detection of allocation manager type breaks
when classes are shaded `org.apache.iceberg.shaded.org.apache.arrow.*`, because
CheckAllocator.check() inspects JAR paths via
`ProtectionDomain.getCodeSource().getLocation()` and doesn't recognize shaded
package structures. Setting this property bypasses the broken-based detection,
which is essential since our `BigIntVector` allocation for `TIMESTAMP_MILLIS`
requires a working RootAllocator in shaded Spark runtime environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
