[
https://issues.apache.org/jira/browse/SPARK-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135153#comment-15135153
]
leo wu commented on SPARK-13218:
--------------------------------
One more point :
If spark-default.conf and spark-env.sh are configured over a remote Spark
standalone cluster manually, and then launch iPython notebook or pyspark,
everything works fine.
So I strongly suspect it's a problem with sparkContext/sparkconf initialization
after stop and start again.
> Executor failed after SparkContext start and start again
> ----------------------------------------------------------
>
> Key: SPARK-13218
> URL: https://issues.apache.org/jira/browse/SPARK-13218
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.0
> Environment: Run IPython/Jupyter along with Spark on ubuntu 14.04
> Reporter: leo wu
>
> In a python notebook, I am trying to stop SparkContext which is initialized
> with local master and then start again with conf over a remote Spark
> standalone cluster, like :
> import sys
> from random import random
> import atexit
> import os
> import platform
> import py4j
> import pyspark
> from pyspark import SparkContext, SparkConf
> from pyspark.sql import SQLContext, HiveContext
> from pyspark.storagelevel import StorageLevel
> os.environ["SPARK_HOME"] = "/home/notebook/spark-1.6.0-bin-hadoop2.6"
> os.environ["PYSPARK_SUBMIT_ARGS"] = "--master spark://10.115.89.219:7077"
> os.environ["SPARK_LOCAL_HOSTNAME"] = "wzymaster2011"
> SparkContext.setSystemProperty("spark.master", "spark://10.115.89.219:7077")
> SparkContext.setSystemProperty("spark.cores.max", "4")
> SparkContext.setSystemProperty("spark.driver.host", "wzymaster2011")
> SparkContext.setSystemProperty("spark.driver.port", "9000")
> SparkContext.setSystemProperty("spark.blockManager.port", "9001")
> SparkContext.setSystemProperty("spark.fileserver.port", "9002")
> conf = SparkConf().setAppName("Python-Test")
> sc = SparkContext(conf=conf)
> However, I always get error in Executor like :
> 16/02/05 14:37:32 DEBUG BlockManager: Getting remote block broadcast_0_piece0
> from BlockManagerId(driver, localhost, 9002)
> 16/02/05 14:37:32 DEBUG TransportClientFactory: Creating new connection to
> localhost/127.0.0.1:9002
> 16/02/05 14:37:32 ERROR RetryingBlockFetcher: Exception while beginning fetch
> of 1 outstanding blocks
> java.io.IOException: Failed to connect to localhost/127.0.0.1:9002
> at
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
> at
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
> at
> org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
> at
> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
> I suspect that new SparkConf isn't properly passed to executor through Spark
> Master for some reason.
> Please advise it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
