viirya opened a new issue, #381: URL: https://github.com/apache/datafusion-comet/issues/381
### Describe the bug There are a few test failures caused by memory leak reported by Java Arrow. They are found in #250 after enabling columnar shuffle by default on Spark SQL tests. For example, In `AdaptiveQueryExecSuite`: ``` [info] - SPARK-35455: Unify empty relation optimization between normal and AQE optimizer - single join *** FAILED *** (3 seconds, 170 milliseconds) [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 729.0 failed 1 times, most recent failure: Lost task 0.0 in stage 729.0 (TID 1631) (e2b4fe719fb3 executor driver): org.apache.comet.CometNativeException: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (32) [info] Allocator(StreamReader/CometBlockStoreShuffleReader) 0/32/32/9223372036854775807 (res/actual/peak/limit) [info] [info] at org.apache.comet.Native.executePlan(Native Method) [info] at org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:71) [info] at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:123) [info] at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:138) ``` After debugging these failures, seems it is triggered if AQE coalesce partitions enabled. ### Steps to reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
