andygrove commented on code in PR #2169:
URL: https://github.com/apache/datafusion-comet/pull/2169#discussion_r2280511357
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -240,29 +287,37 @@ class CometExecIterator(
traceMemoryUsage()
}
- // The allocator thoughts the exported ArrowArray and ArrowSchema
structs are not released,
- // so it will report:
- // Caused by: java.lang.IllegalStateException: Memory was leaked by
query.
- // Memory leaked: (516) Allocator(ROOT) 0/516/808/9223372036854775807
(res/actual/peak/limit)
- // Suspect this seems a false positive leak, because there is no
reported memory leak at JVM
- // when profiling. `allocator` reports a leak because it calculates the
accumulated number
- // of memory allocated for ArrowArray and ArrowSchema. But these
exported ones will be
- // released in native side later.
- // More to clarify it. For ArrowArray and ArrowSchema, Arrow will put a
release field into the
- // memory region which is a callback function pointer (C function) that
could be called to
- // release these structs in native code too. Once we wrap their memory
addresses at native
- // side using FFI ArrowArray and ArrowSchema, and drop them later, the
callback function will
- // be called to release the memory.
- // But at JVM, the allocator doesn't know about this fact so it still
keeps the accumulated
- // number.
- // Tried to manually do `release` and `close` that can make the
allocator happy, but it will
- // cause JVM runtime failure.
-
- // allocator.close()
Review Comment:
This comment refers to an `allocator` that no longer exists so it doesn't
seem very useful to keep around
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
