[
https://issues.apache.org/jira/browse/SPARK-30002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaushal Prajapati updated SPARK-30002:
--------------------------------------
Description:
In PySpark, we create SparkContext via user spark configurations or the default
ones, and it gets launched through py4j gateway internally.
Let's say if I have launched py4j gateway from another application then to
communicate with the same py4j gateway I have to set below configuration:-
{code:java}
export PYSPARK_GATEWAY_PORT=12345
export PYSPARK_GATEWAY_SECRET=***********************
{code}
So when PySpark tries to create its own SparkContext after the communication
has been set up, it doesn't check whether there is any available SparkContext
in the same JVM.
Current code snippet:-
{code:java}
def _initialize_context(self, jconf):
"""
Initialize SparkContext in function to allow subclass specific
initialization
"""
return self._jvm.JavaSparkContext(jconf){code}
After changing it to the following, it works fine for me.
{code:java}
def _initialize_context(self, jconf):
"""
Initialize SparkContext in function to allow subclass specific
initialization
"""
return
self._jvm.JavaSparkContext(self._jvm.org.apache.spark.SparkContext.getOrCreate(jconf)){code}
It looks like a good use case for improvement.
was:
In PySpark, we create SparkContext via user spark configurations or the default
ones, and it gets launched through py4j gateway internally.
Let's say if I have launched py4j gateway from another application then to
communicate with the same py4j gateway I have to set below configuration:-
{code:java}
export PYSPARK_GATEWAY_PORT=12345
export PYSPARK_GATEWAY_SECRET=***********************
{code}
So when PySpark tries to create its own SparkContext after the communication
has been set up, it doesn't check whether there is any available SparkContext
in the same JVM.
Current code snippet:-
{code:java}
def _initialize_context(self, jconf):
"""
Initialize SparkContext in function to allow subclass specific
initialization
"""
return self._jvm.JavaSparkContext(jconf){code}
If we change it to the following, it works fine for me.
{code:java}
def _initialize_context(self, jconf):
"""
Initialize SparkContext in function to allow subclass specific
initialization
"""
return
self._jvm.JavaSparkContext(self._jvm.org.apache.spark.SparkContext.getOrCreate(jconf)){code}
It looks like a good use case for improvement.
> Reuse SparkSession in pyspark via Gateway
> -----------------------------------------
>
> Key: SPARK-30002
> URL: https://issues.apache.org/jira/browse/SPARK-30002
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.4.0, 2.4.4
> Reporter: Kaushal Prajapati
> Priority: Minor
>
> In PySpark, we create SparkContext via user spark configurations or the
> default ones, and it gets launched through py4j gateway internally.
> Let's say if I have launched py4j gateway from another application then to
> communicate with the same py4j gateway I have to set below configuration:-
>
> {code:java}
> export PYSPARK_GATEWAY_PORT=12345
> export PYSPARK_GATEWAY_SECRET=***********************
> {code}
>
> So when PySpark tries to create its own SparkContext after the communication
> has been set up, it doesn't check whether there is any available SparkContext
> in the same JVM.
> Current code snippet:-
>
> {code:java}
> def _initialize_context(self, jconf):
> """
> Initialize SparkContext in function to allow subclass specific
> initialization
> """
> return self._jvm.JavaSparkContext(jconf){code}
>
>
> After changing it to the following, it works fine for me.
> {code:java}
> def _initialize_context(self, jconf):
> """
> Initialize SparkContext in function to allow subclass specific
> initialization
> """
> return
> self._jvm.JavaSparkContext(self._jvm.org.apache.spark.SparkContext.getOrCreate(jconf)){code}
>
> It looks like a good use case for improvement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
