leo wu created SPARK-13218:
------------------------------
Summary: Executor failed after SparkContext start and start again
Key: SPARK-13218
URL: https://issues.apache.org/jira/browse/SPARK-13218
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.6.0
Environment: Run IPython/Jupyter along with Spark on ubuntu 14.04
Reporter: leo wu
In a python notebook, I am trying to stop SparkContext which is initialized
with local master and then start again with conf over a remote Spark
standalone cluster, like :
import sys
from random import random
import atexit
import os
import platform
import py4j
import pyspark
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, HiveContext
from pyspark.storagelevel import StorageLevel
os.environ["SPARK_HOME"] = "/home/notebook/spark-1.6.0-bin-hadoop2.6"
os.environ["PYSPARK_SUBMIT_ARGS"] = "--master spark://10.115.89.219:7077"
os.environ["SPARK_LOCAL_HOSTNAME"] = "wzymaster2011"
SparkContext.setSystemProperty("spark.master", "spark://10.115.89.219:7077")
SparkContext.setSystemProperty("spark.cores.max", "4")
SparkContext.setSystemProperty("spark.driver.host", "wzymaster2011")
SparkContext.setSystemProperty("spark.driver.port", "9000")
SparkContext.setSystemProperty("spark.blockManager.port", "9001")
SparkContext.setSystemProperty("spark.fileserver.port", "9002")
conf = SparkConf().setAppName("Python-Test")
sc = SparkContext(conf=conf)
However, I always get error in Executor like :
16/02/05 14:37:32 DEBUG BlockManager: Getting remote block broadcast_0_piece0
from BlockManagerId(driver, localhost, 9002)
16/02/05 14:37:32 DEBUG TransportClientFactory: Creating new connection to
localhost/127.0.0.1:9002
16/02/05 14:37:32 ERROR RetryingBlockFetcher: Exception while beginning fetch
of 1 outstanding blocks
java.io.IOException: Failed to connect to localhost/127.0.0.1:9002
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at
org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
I suspect that new SparkConf isn't properly passed to executor through Spark
Master for some reason.
Please advise it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
