yantzu commented on issue #406:
URL: https://github.com/apache/incubator-livy/issues/406#issuecomment-1556043500
We haven't tried Spark 3.3 yet, but Spark 3.2 should work. The log show
error happens in Spark's code /pyspark.zip/pyspark/find_spark_home.py which
seems not related to Livy, 2 suggestion:
1, set SPARK_HOME in livy, to avoid find spark home
2, check your python version
cocdkl ***@***.***> 于2023年5月18日周四 17:46写道：
> 这是通过spark 3.2.4 提交任务报的错：
> 23/05/18 17:43:00 INFO SparkEntries: Created Spark session.
> 23/05/18 17:43:00 INFO BlockManagerMasterEndpoint: Registering block
> manager localhost:33381 with 912.3 MiB RAM, BlockManagerId(2, localhost,
> 33381, None)
> 23/05/18 17:43:25 ERROR PythonInterpreter: Process has died with 1
> 23/05/18 17:43:25 ERROR PythonInterpreter: Traceback (most recent call
> last):
> File "/tmp/8001765088071541442", line 722, in
> sys.exit(main())
> File "/tmp/8001765088071541442", line 570, in main
> exec('from pyspark.sql import HiveContext', global_dict)
> File "", line 1, in
> File
>
"/home/cocdkl/soft/spark-3.2.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/
> *init*.py", line 53, in
> File
>
"/home/cocdkl/soft/spark-3.2.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py",
> line 34, in
> File
>
"/home/cocdkl/soft/spark-3.2.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/java_gateway.py",
> line 31, in
> File
>
"/home/cocdkl/soft/spark-3.2.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/find_spark_home.py",
> line 68
> print("Could not find valid SPARK_HOME while searching {0}".format(paths),
> file=sys.stderr)
> ^
> SyntaxError: invalid syntax
>
> 奇怪的是在yarn ，spark的集群是启动的，但是一提交具体代码作业，就会报错。
>
> 希望得到解决，谢谢
>
> —
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/incubator-livy/issues/406#issuecomment-1552807723>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AB35X7DEAC6DHKPA7JR76HDXGXVX7ANCNFSM6AAAAAAYGA2E2I>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
