andygrove commented on issue #2162:
URL:
https://github.com/apache/datafusion-comet/issues/2162#issuecomment-3192285635
So in this case, the native code is fetching batches from JVM in a loop and
retains the batches in memory as part of the shuffle writing process. Each call
to `next` can cause resources to be freed.
I am also concerned that the `next` method releases the reference to the
current batch before passing control back to native code:
```
public int next(long[] arrayAddrs, long[] schemaAddrs) {
if (currentBatch == null) {
return -1;
}
int numRows = nativeUtil.exportBatch(arrayAddrs, schemaAddrs,
currentBatch);
currentBatch = null; // <------- problematic
return numRows;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
