andygrove commented on code in PR #6048:
URL: https://github.com/apache/datafusion-comet/pull/6048#discussion_r4053935463
##########
spark/src/main/scala/org/apache/comet/Tracing.scala:
##########
@@ -23,6 +23,16 @@ object Tracing {
private val nativeLib = new Native
+ /**
+ * Emits the Arrow memory counters for the JVM side. `jvm_arrow_imported` is
the part of
+ * `jvm_arrow_allocated` that came from native over the C Data Interface, so
the difference is
+ * the Arrow memory the JVM allocated itself. See
[[CometArrowImportAllocator]].
+ */
+ def logArrowMemory(): Unit = {
+ nativeLib.logMemoryUsage("jvm_arrow_allocated",
CometArrowAllocator.getAllocatedMemory)
+ nativeLib.logMemoryUsage("jvm_arrow_imported",
CometArrowImportAllocator.getAllocatedMemory)
Review Comment:
Confirmed and fixed in d5faf5626. Full response in
https://github.com/apache/datafusion-comet/pull/6048#issuecomment-5743763058.
I reproduced both mechanisms against the Arrow 18.3 sources before changing
anything:
`ArrayImporter.java:58` allocates the owning `ArrowArray` from this
allocator, and
`BitVectorHelper.loadValidityBuffer:328` allocates a validity bitmap here
when an imported vector
is all-valid or all-null and carries no validity buffer. With the struct
that matches the 640 bytes
you measured for 4096 rows.
Taking your second option for the general case: `jvm_arrow_imported` is now
documented as what the
import path holds, so `jvm_arrow_allocated - jvm_arrow_imported` is
described as a close lower
bound rather than an exact split, in the tracing guide and in both scaladocs.
The UDF case is fixed rather than documented, since it was the one unbounded
contaminant.
`CometUdfBridge` now transfers the result to the root before export when the
UDF allocated it
elsewhere, which `TransferPair` does as an ownership move rather than a
payload copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
