Yi Zhu created SPARK-41920:
------------------------------
Summary: Task throw Exception call cleanUpAllAllocatedMemory cause
throw NPE
Key: SPARK-41920
URL: https://issues.apache.org/jira/browse/SPARK-41920
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.2.1
Reporter: Yi Zhu
{code:java}
23/01/03 21:41:18 INFO SortBasedPusher: Pushdata is not empty , do push.
Traceback (most recent call last):
File
"/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/daemon.py",
line 186, in manager
File
"/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/daemon.py",
line 74, in worker
File
"/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/worker.py",
line 643, in main
if read_int(infile) == SpecialLengths.END_OF_STREAM:
File
"/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/serializers.py",
line 564, in read_int
raise EOFError
EOFError
23/01/03 21:41:29 ERROR Executor: Exception in task 605.1 in stage 94.0 (TID
58026)
java.lang.NullPointerException
at
org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:399)
at
org.apache.spark.shuffle.rss.SortBasedPusher.pushData(SortBasedPusher.java:155)
at
org.apache.spark.shuffle.rss.SortBasedPusher.spill(SortBasedPusher.java:317)
at
org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:177)
at
org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:289)
at
org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:116)
at
org.apache.spark.sql.execution.python.HybridRowQueue.createNewQueue(RowQueue.scala:227)
at
org.apache.spark.sql.execution.python.HybridRowQueue.add(RowQueue.scala:250)
at
org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$10(EvalPythonExec.scala:125)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at
scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:1159)
at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:1174)
at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1212)
at
scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:1215)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at
org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:307)
at
org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.writeIteratorToStream(PythonUDFRunner.scala:53)
at
org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:397)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2066)
at
org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:232)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
