[
https://issues.apache.org/jira/browse/SPARK-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363303#comment-15363303
]
Sean Owen commented on SPARK-16379:
-----------------------------------
It's true, I can't say for sure the problem exists without that line. It's
suspicious. In any event it seems worth doing away with it while fixing this
up, which may still entail reverting the main change for safety but also trying
to prove there's no similar problem still lurking in the previous Logging
approach that needs any working-around anywhere.
> Spark on mesos is broken due to race condition in Logging
> ---------------------------------------------------------
>
> Key: SPARK-16379
> URL: https://issues.apache.org/jira/browse/SPARK-16379
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.0
> Reporter: Stavros Kontopoulos
> Priority: Blocker
> Attachments: out.txt
>
>
> This commit introduced a transient lazy log val:
> https://github.com/apache/spark/commit/044971eca0ff3c2ce62afa665dbd3072d52cbbec
> This has caused problems in the past:
> https://github.com/apache/spark/pull/1004
> One commit before that everything works fine.
> I spotted that when my CI started to fail:
> https://ci.typesafe.com/job/mit-docker-test-ref/191/
> You can easily verify it by installing mesos on your machine and try to
> connect with spark shell from bin dir:
> ./spark-shell --master mesos://zk://localhost:2181/mesos --conf
> spark.executor.url=$(pwd)/../spark-2.0.0-SNAPSHOT-bin-test.tgz
> It gets stuck at the point where it tries to create the SparkContext.
> Logging gets stuck here:
> I0705 12:10:10.076617 9303 group.cpp:700] Trying to get
> '/mesos/json.info_0000000152' in ZooKeeper
> I0705 12:10:10.076920 9304 detector.cpp:479] A new leading master
> ([email protected]:5050) is detected
> I0705 12:10:10.076956 9303 sched.cpp:326] New master detected at
> [email protected]:5050
> I0705 12:10:10.077057 9303 sched.cpp:336] No credentials provided.
> Attempting to register without authentication
> I0705 12:10:10.090709 9301 sched.cpp:703] Framework registered with
> 13553f8b-f42c-4f20-88cd-16f1cc153ede-0001
> I verified it also by changing @transient lazy val log to def and it works as
> expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
