Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4066#discussion_r23256922
--- Diff: core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala ---
@@ -105,10 +107,24 @@ class SparkHadoopWriter(@transient jobConf: JobConf)
def commit() {
val taCtxt = getTaskContext()
val cmtr = getOutputCommitter()
+ val conf = SparkEnv.get.conf
+ val dagSchedulerActor =
+ AkkaUtils.makeDriverRef("DAGSchedulerSupervisor/DAGScheduler", conf,
SparkEnv.get.actorSystem)
--- End diff --
The idea here was modeled after how Executor -> Driver communication works
for the HeartbeatReceiver actor.
On the driver, HeartbeatReceiver is started with an explicit name:
https://github.com/apache/spark/blob/e200ac8e53a533d64a79c18561b557ea445f1cc9/core/src/main/scala/org/apache/spark/SparkContext.scala#L327
The executors use this name to construct a reference to the actor:
https://github.com/apache/spark/blob/3453d578ad9933be6881488c8ca3611e5b686af9/core/src/main/scala/org/apache/spark/executor/Executor.scala#L367
Looking at the [Akka
docs](http://doc.akka.io/docs/akka/snapshot/scala/actors.html#Creating_Actors_with_Props):
> The name parameter is optional, but you should preferably name your
actors, since that is used in log messages and for identifying actors. The name
must not be empty or start with $, but it may contain URL encoded characters
(eg. %20 for a blank space). If the given name is already in use by another
child to the same parent an InvalidActorNameException is thrown.
If you don't pass an explicit name, the actor is assigned an
automatically-generated name
([reference](http://doc.akka.io/api/akka/2.0/akka/actor/ActorSystem.html)).
So, my hunch is that we're somehow creating multiple actors with the same
name. I wonder if this is due to sharing between tests or improper cleanup, or
whether there's something wrong with my use of explicit actor names here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
