[
https://issues.apache.org/jira/browse/SPARK-53759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028361#comment-18028361
]
Bjarne Thorsted commented on SPARK-53759:
-----------------------------------------
I don't know if this is related to Windows 11, somehow? I have the exact same
issue on pyspark versions 3.5.2 and 4.0.0, but I know that it used to work for
3.5.2 on my previous machine running Windows 10.
> PySpark crashes with Python 3.12+ on Windows
> --------------------------------------------
>
> Key: SPARK-53759
> URL: https://issues.apache.org/jira/browse/SPARK-53759
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.5.5, 4.0.0, 4.0.1
> Environment: * PySpark 4.0.1, 4.0.0, and 3.5.5
> * Python 3.12 and 3.13 (the last working version is 3.11.13)
> * Java 17
> * Spark Classic (not Spark Connect) in local mode
> * All Windows machines we've tested
> * Tested with just PySpark installed, and Pandas (2.3.2) & PyArrow (21.0.0)
> installed
> * Tested installation via pip and conda
> Reporter: Max Payson
> Priority: Critical
>
> Python 3.12+ crashes locally on Windows when using the `createDataFrame` API.
> All dataframe creation methods in the [Quickstart:
> Dataframe|https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html#DataFrame-Creation]
> seem to crash but other operations work as expected.
> Reproduction:
> {code:java}
> import os
> import sys
> from pyspark.sql import SparkSession
> os.environ["PYSPARK_PYTHON"] = sys.executable
> spark = SparkSession.builder.getOrCreate()
> df = spark.createDataFrame([(1,), (2,)], ["myint"])
> df.show() {code}
>
> Stack trace. This is with "spark.python.worker.faulthandler.enabled" enabled,
> but the stack trace is the same with it disabled:
> {code:java}
> Traceback (most recent call last):
> File "<your_script>.py", line 10, in <module>
> df.show()
> File "<pyspark>/sql/classic/dataframe.py", line 285, in show
> print(self._show_string(n, truncate, vertical))
> File "<pyspark>/sql/classic/dataframe.py", line 303, in _show_string
> return self._jdf.showString(n, 20, vertical)
> File "<py4j>/java_gateway.py", line 1362, in __call__
> return_value = get_return_value(
> answer, self.gateway_client, self.target_id, self.name)
> File "<pyspark>/errors/exceptions/captured.py", line 282, in deco
> return f(*a, **kw)
> File "<py4j>/protocol.py", line 327, in get_return_value
> raise Py4JJavaError(
> "An error occurred while calling {0}{1}{2}.\n".
> format(target_id, ".", name), value)
> py4j.protocol.Py4JJavaError: An error occurred while calling o48.showString.:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0
> (TID 0) (executor driver): org.apache.spark.SparkException: Python worker
> exited unexpectedly (crashed)
> at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:624)
> at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:599)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
> at
> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:945)
> at
> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:925)
> at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:532)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:601)
> at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
> at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:402)
> at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:901)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:901)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171)
> at org.apache.spark.scheduler.Task.run(Task.scala:147)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:647)
> at
> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:80)
> at
> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:77)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:650)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)Caused by:
> java.io.EOFException
> at java.base/java.io.DataInputStream.readInt(DataInputStream.java:386)
> at
> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:933)
> ... 26 more{code}
> Note, I originally posted in the Python 3.13 issue but it seemed better to
> create a new issue since that one was closed. Please let us know if our team
> can help debug this further, it seems relatively low level. Thank you!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
