[
https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244592#comment-14244592
]
Erik Erlandson commented on SPARK-4514:
---------------------------------------
The root problem appears to be the behavior of {{InheritableThreadLocal}} and
{{childValue}} -- in particular, {{childValue}} only gets called once. When a
thread is reused, its value is not updated. So multiple calls to methods such
as {{setJobGroup}} are effectively ignored in the case of threads being reused.
Changing implementations of asynchronous actions to use
{{ComplexFutureAction}} causes more threads to be invoked, and reused, and this
causes the problem to show up in the unit testing.
In general, {{InheritableThreadLocal}} seems like a difficult structure to use
properly. It only works intuitively if inherited values will not change after
inheritance.
http://docs.oracle.com/javase/8/docs/api/java/lang/InheritableThreadLocal.html
Discussion of this topic in forums suggests that passing data down via
constructor parameters is the more reliable and intuitive approach in the use
cases where inheritance with updates is required.
IMO, this adds weight to solutions involving parameters to
{{ComplexFutureAction}}.
> ComplexFutureAction does not preserve job group IDs
> ---------------------------------------------------
>
> Key: SPARK-4514
> URL: https://issues.apache.org/jira/browse/SPARK-4514
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0, 1.1.1, 1.2.0
> Reporter: Erik Erlandson
> Assignee: Josh Rosen
> Priority: Critical
>
> The current job group id of a Spark context is stored in the
> {{localProperties}} member value. This data structure is designed to be
> thread local, and its settings are not preserved when {{ComplexFutureAction}}
> instantiates a new {{Future}}.
> One consequence of this is that {{takeAsync()}} does not behave in the same
> way as other async actions, e.g. {{countAsync()}}. For example, this test
> (if copied into StatusTrackerSuite.scala), will fail, because
> {{"my-job-group2"}} is not propagated to the Future which actually
> instantiates the job:
> {code:java}
> test("getJobIdsForGroup() with takeAsync()") {
> sc = new SparkContext("local", "test", new SparkConf(false))
> sc.setJobGroup("my-job-group2", "description")
> sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
> val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
> val firstJobId = eventually(timeout(10 seconds)) {
> firstJobFuture.jobIds.head
> }
> eventually(timeout(10 seconds)) {
> sc.statusTracker.getJobIdsForGroup("my-job-group2") should be
> (Seq(firstJobId))
> }
> }
> {code}
> It also impacts current PR for SPARK-1021, which involves additional uses of
> {{ComplexFutureAction}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
