Philipp Dallig created ZEPPELIN-4802:
----------------------------------------
Summary: pyspark warnings with spark-3.0.0-preview - Sync of
properties between JVM and PVM
Key: ZEPPELIN-4802
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4802
Project: Zeppelin
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Philipp Dallig
Attachments: pyspark_warning.png
Hi
I'm using Zeppelin (Version: master branch) with spark-3.0.0-preview. I get
the following warning if I start a simple pyspark paragraph. Can someone help
to solve this issue?
pyspark code
{code}
%pyspark
spark
{code}
Warning
{code}
/opt/spark/python/lib/pyspark.zip/pyspark/util.py:141: UserWarning: Currently,
'setJobGroup' (set to local properties) with multiple threads does not properly
work.
Internally threads on PVM and JVM are not synced, and JVM thread can be reused
for multiple threads on PVM, which fails to isolate local properties for each
thread on PVM.
To work around this, you can set PYSPARK_PIN_THREAD to true (see SPARK-22340).
However, note that it cannot inherit the local properties from the parent
thread although it isolates each thread on PVM and JVM with its own local
properties.
To work around this, you should manually copy and set the local properties from
the parent thread to the child thread when you create another thread.
/opt/spark/python/lib/pyspark.zip/pyspark/util.py:141: UserWarning: Currently,
'setLocalProperty' (set to local properties) with multiple threads does not
properly work.
Internally threads on PVM and JVM are not synced, and JVM thread can be reused
for multiple threads on PVM, which fails to isolate local properties for each
thread on PVM.
To work around this, you can set PYSPARK_PIN_THREAD to true (see SPARK-22340).
However, note that it cannot inherit the local properties from the parent
thread although it isolates each thread on PVM and JVM with its own local
properties.
To work around this, you should manually copy and set the local properties from
the parent thread to the child thread when you create another thread.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
