Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

[416161...@qq.com]

+1




RuifengZheng
ruife...@foxmail.com








--Original--
From:   
 "Chao Sun" 
   
https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o
 
 
  cc: Ruifeng (co-author) and Hyukjin 
(shepherd)
 
 
  Regards,
  Yikun

 
-
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

[Chao Sun]

+1 (non-binding)

On Mon, Sep 19, 2022 at 10:17 PM Wenchen Fan  wrote:
>
> +1
>
> On Mon, Sep 19, 2022 at 2:59 PM Yang,Jie(INF)  wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> Yang Jie
>>
>> 
>> 发件人: Yikun Jiang 
>> 发送时间: 2022年9月19日 14:23:14
>> 收件人: Denny Lee
>> 抄送: bo zhaobo; Yuming Wang; Kent Yao; Gengliang Wang; Hyukjin Kwon; dev; zrf
>> 主题: Re: [DISCUSS] SPIP: Support Docker Official Image for Spark
>>
>> Thanks for your support!  @all
>>
>> > Count me in to help as well, eh?! :)
>>
>> @Denny Sure, It would be great to have your help! I'm going to create a JIRA 
>> and TASKS if the SPIP vote passes.
>>
>>
>> On Mon, Sep 19, 2022 at 10:34 AM Denny Lee  wrote:
>>>
>>> +1 (non-binding).
>>>
>>> This is a great idea and we should definitely do this.  Count me in to help 
>>> as well, eh?! :)
>>>
>>> On Sun, Sep 18, 2022 at 7:24 PM bo zhaobo  
>>> wrote:

 +1 (non-binding)

 This will bring the good experience to customers. So excited about this. 
 ;-)

 Yuming Wang  于2022年9月19日周一 10:18写道：
>
> +1.
>
> On Mon, Sep 19, 2022 at 9:44 AM Kent Yao  wrote:
>>
>> +1
>>
>> Gengliang Wang  于2022年9月19日周一 09:23写道：
>> >
>> > +1, thanks for the work!
>> >
>> > On Sun, Sep 18, 2022 at 6:20 PM Hyukjin Kwon  
>> > wrote:
>> >>
>> >> +1
>> >>
>> >> On Mon, 19 Sept 2022 at 09:15, Yikun Jiang  
>> >> wrote:
>> >>>
>> >>> Hi, all
>> >>>
>> >>>
>> >>> I would like to start the discussion for supporting Docker Official 
>> >>> Image for Spark.
>> >>>
>> >>>
>> >>> This SPIP is proposed to add Docker Official Image(DOI) to ensure 
>> >>> the Spark Docker images meet the quality standards for Docker 
>> >>> images, to provide these Docker images for users who want to use 
>> >>> Apache Spark via Docker image.
>> >>>
>> >>>
>> >>> There are also several Apache projects that release the Docker 
>> >>> Official Images, such as: flink, storm, solr, zookeeper, httpd (with 
>> >>> 50M+ to 1B+ download for each). From the huge download statistics, 
>> >>> we can see the real demands of users, and from the support of other 
>> >>> apache projects, we should also be able to do it.
>> >>>
>> >>>
>> >>> After support:
>> >>>
>> >>> The Dockerfile will still be maintained by the Apache Spark 
>> >>> community and reviewed by Docker.
>> >>>
>> >>> The images will be maintained by the Docker community to ensure the 
>> >>> quality standards for Docker images of the Docker community.
>> >>>
>> >>>
>> >>> It will also reduce the extra docker images maintenance effort (such 
>> >>> as frequently rebuilding, image security update) of the Apache Spark 
>> >>> community.
>> >>>
>> >>>
>> >>> See more in SPIP DOC: 
>> >>> https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o
>> >>>
>> >>>
>> >>> cc: Ruifeng (co-author) and Hyukjin (shepherd)
>> >>>
>> >>>
>> >>> Regards,
>> >>> Yikun
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

回复：

[javacaoyu]

Try:


import os
os.environ['PYSPARK_PYTHON'] = “python path”
os.environ[’SPARK_HOME’] = “SPARK path”






在 2022年9月20日 17:51，yogita bhardwaj 写道：


 
I have installed pyspark using pip.
I m getting the error while running the following code.
from pyspark import SparkContext
sc=SparkContext()
a=sc.parallelize([1,2,3,4])
print(f"a_take:{a.take(2)}")
 
py4j.protocol.Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in 
stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 
0) (DESKTOP-DR2QC97.mshome.net executor driver): 
org.apache.spark.SparkException: Python worker failed to connect back.
at 
org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)
at 
org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
at 
org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
at 
org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)
at 
org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at 
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
 
Can anyone please help me to resolve this issue.

[no subject]

[yogita bhardwaj]

I have installed pyspark using pip.
I m getting the error while running the following code.
from pyspark import SparkContext
sc=SparkContext()
a=sc.parallelize([1,2,3,4])
print(f"a_take:{a.take(2)}")

py4j.protocol.Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in 
stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 
0) (DESKTOP-DR2QC97.mshome.net executor driver): 
org.apache.spark.SparkException: Python worker failed to connect back.
    at 
org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)
    at 
org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
    at 
org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
    at 
org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)
    at 
org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
    at 
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)

Can anyone please help me to resolve this issue.

Re: Issue with SparkContext

[javacaoyu]

Is you using the pyspark?


If pyspark, you can try to set env about PYSPARK_PYTHON  SPARK_HOME
Example:


import os
os.environ['PYSPARK_PYTHON'] = “python path”
os.environ[’SPARK_HOME’] = “SPARK path”


you can try this code…may it can resolved this.


在 2022年9月20日 17:34，Bjørn Jørgensen 写道：


Hi, we have a user group at u...@spark.apache.org 


You must install a java JRE 


If you are on ubuntu you can type
apt-get install openjdk-17-jre-headless



tir. 20. sep. 2022 kl. 06:15 skrev yogita bhardwaj :

 
I am getting the py4j.protocol.Py4JJavaError while running SparkContext. Can 
you please help me to resolve this issue.
 
Sent from Mail for Windows
 




-- 

Bjørn Jørgensen 
Vestre Aspehaug 4, 6010 Ålesund 
Norge

+47 480 94 297

Re: Issue with SparkContext

[Bjørn Jørgensen]

Hi, we have a user group at u...@spark.apache.org

You must install a java JRE

If you are on ubuntu you can type
apt-get install openjdk-17-jre-headless

tir. 20. sep. 2022 kl. 06:15 skrev yogita bhardwaj <
yogita.bhard...@iktara.ai>:

>
>
> I am getting the py4j.protocol.Py4JJavaError while running SparkContext.
> Can you please help me to resolve this issue.
>
>
>
> Sent from Mail  for
> Windows
>
>
>


-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297

Pyspark SparkContext issue

[yogita bhardwaj]

I am getting the py4j.protocol.Py4JJavaError while running SparkContext. Can 
you please help me to resolve this issue.
from pyspark import SparkContext
sc=SparkContext()
a=sc.parallelize([1,2,3,4])
print(f"a_take:{a.take(2)}")


py4j.protocol.Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.


Sent from Mail for Windows