[
https://issues.apache.org/jira/browse/SPARK-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114283#comment-15114283
]
Sean Owen commented on SPARK-4878:
----------------------------------
[~viirya] maybe you know better; I still see this code in
{{CoarseGrainedExecutorBackend}}, but I am not clear whether it's still live
and used?
{code}
// Bootstrap to fetch the driver's Spark properties.
val executorConf = new SparkConf
val port = executorConf.getInt("spark.executor.port", 0)
val fetcher = RpcEnv.create(
"driverPropsFetcher",
hostname,
port,
executorConf,
new SecurityManager(executorConf),
clientMode = true)
val driver = fetcher.setupEndpointRefByURI(driverUrl)
val props = driver.askWithRetry[Seq[(String,
String)]](RetrieveSparkProps) ++
Seq[(String, String)](("spark.app.id", appId))
fetcher.shutdown()
{code}
> driverPropsFetcher causes spurious Akka disassociate errors
> -----------------------------------------------------------
>
> Key: SPARK-4878
> URL: https://issues.apache.org/jira/browse/SPARK-4878
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Stephen Haberman
> Priority: Minor
>
> The dedicated Akka system to fetching driver properties seems fine, but it
> leads to very misleading "AssociationHandle$Disassociated", dead letter, etc.
> sort of messages that can lead the user to believe something is wrong with
> the cluster.
> (E.g. personally I thought it was a Spark -rc1/-rc2 bug and spent awhile
> poking around until I saw in the code that driverPropsFetcher is
> purposefully/immediately shutdown.)
> Is there any way to cleanly shutdown that initial akka system so that the
> driver doesn't log these errors?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
