shubham19may commented on code in PR #14499:
URL: https://github.com/apache/iceberg/pull/14499#discussion_r2509758888
##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowAllocation.java:
##########
@@ -19,16 +19,43 @@
package org.apache.iceberg.arrow;
import org.apache.arrow.memory.RootAllocator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
public class ArrowAllocation {
+ private static final Logger LOG =
LoggerFactory.getLogger(ArrowAllocation.class);
+
+ private static final String ALLOCATION_MANAGER_TYPE_PROPERTY =
+ "arrow.memory.allocation.manager.type";
+
static {
- ROOT_ALLOCATOR = new RootAllocator(Long.MAX_VALUE);
+ // Set Arrow allocation manager to Netty if not already configured
+ // This prevents Arrow's auto-detection from failing when classes are
shaded
+ // (e.g.; org.apache.iceberg.shaded.org.apache.arrow.*) since the
path-based
+ // detection in CheckAllocator.check() doesn't recognize shaded package
structures
Review Comment:
Well, yes, `arrow-java` tries to detect which allocation manager to use by
checking the JAR file paths (you can check in arrow-java:
[here](https://github.com/apache/arrow-java/blob/03e0d4da63e1de747e645b0053d737b263d7f26d/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java#L42-L59)).
I added `Netty` as the default allocation manager as a “good practice” as
`iceberg-arrow` module explicitly depends on `arrow-memory-netty`, (check
[here](https://github.com/apache/iceberg/blob/ae4f5c1f8893f199b633f3ccbc6ddd94170db596/build.gradle#L917C1-L921C6)
in iceberg).
But, if we skip it, I don’t see any breaking changes for this bug, or inside
the current code base or any of the test cases I can primarily think of as of
now.
Hence, I have removed it as of now to keep the scope focused on the
`TIMESTAMP_MILLIS` fix, as without it as well, the current solution to the bug,
works perfectly fine.
@pvary, @nandorKollar please do give a final review to the PR whenever you
are free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
