Midhunpottammal commented on issue #36994:
URL: https://github.com/apache/arrow/issues/36994#issuecomment-1978396551
> @Midhunpottammal can you share more details? which version of Arrow and
spark are you using and the stack trace.
java -version
> java version "21.0.2" 2024-01-16 LTS
> Java(TM) SE Runtime Environment (build 21.0.2+13-LTS-58)
> Java HotSpot(TM) 64-Bit Server VM (build 21.0.2+13-LTS-58, mixed mode,
sharing)
Spark - Version
> pyspark==3.5.0
pyarrow -Version
> pyarrow 15.0.0
Error Log
TaskContextImpl: Error in TaskCompletionListener
java.lang.IllegalStateException: Memory was leaked by query. Memory leaked:
(49152)
Allocator(toArrowBatchIterator) 0/49152/49152/9223372036854775807
(res/actual/peak/limit)
at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:476)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$ArrowBatchIterator.$anonfun$new$2(ArrowConverters.scala:97)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$ArrowBatchIterator.$anonfun$new$2$adapted(ArrowConverters.scala:95)
at
org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:132)
at
org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(TaskContextImpl.scala:144)
at
org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(TaskContextImpl.scala:144)
at
org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:199)
at
org.apache.spark.TaskContextImpl.invokeTaskCompletionListeners(TaskContextImpl.scala:144)
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:137)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:172)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
24/03/05 14:02:59 ERROR Executor: Exception in task 11.0 in stage 0.0 (TID
11)
java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.<init>(long, int) not available
at
org.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:174)
at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229)
Full Stack Trace link
[](https://drive.google.com/file/d/1eyHhJsqgCjCVnAFFR290eOMekvWC42so/view?usp=sharing)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
