[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980901#comment-15980901
]
Sergey Zhemzhitsky commented on SPARK-20404:
[~srowen]
Here it is: https://github.com/apache/spark/pull/17740
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980900#comment-15980900
]
Apache Spark commented on SPARK-20404:
--
User 'szhem' has created a pull request for this issue:
https://github.com/apache/spark/pull/17740
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980344#comment-15980344
]
Sean Owen commented on SPARK-20404:
---
Ping [~szhemzhitsky] are you going to make a pull request?
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976732#comment-15976732
]
Sean Owen commented on SPARK-20404:
---
We use pull requests, not patches. See http://spark.apache.org/contributing.html
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976726#comment-15976726
]
Sergey Zhemzhitsky commented on SPARK-20404:
Thanks for drawing attention to this. Just updated the patch.
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976599#comment-15976599
]
Sean Owen commented on SPARK-20404:
---
Hm, no I reverse myself. Yes it should fail fast if it's going to fail. It's
probably not great to allow a null name in the method that takes a name
argument. However, clearly the underlying implementation can accept no name. In
that case your fix looks good to me. There's a third place, in register(), that
probably needs the same change.
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976506#comment-15976506
]
Sergey Zhemzhitsky commented on SPARK-20404:
I would agree, if the error occurs just at the time of creating the
accumulator, but in this case the error may occur at any time (possibly after
hours of running the job) when you will try to update the accumulator. So, for
me the current behaviour seems to be misleading and confusing, rather than an
expected one.
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976405#comment-15976405
]
Sean Owen commented on SPARK-20404:
---
The argument is not an Option, which suggests that it's required, so passing
null and getting NullPointerException doesn't seem surprising. Right?
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976400#comment-15976400
]
Sergey Zhemzhitsky commented on SPARK-20404:
Is it a normal behavior for spark 2+? It seems that using Option to wrap a
value, that may or may not be null is more appropriate way, than using Some.
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20404) Regression with accumulator names when migrating from 1.6 to 2.x
[
https://issues.apache.org/jira/browse/SPARK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976349#comment-15976349
]
Sean Owen commented on SPARK-20404:
---
That doesn't sound like something that should be supported. Behavior changes
are normal from 1.x to 2.x
> Regression with accumulator names when migrating from 1.6 to 2.x
>
>
> Key: SPARK-20404
> URL: https://issues.apache.org/jira/browse/SPARK-20404
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Environment: Spark: 2.1
> Scala: 2.11
> Spark master: local
>Reporter: Sergey Zhemzhitsky
> Attachments: spark-context-accum-option.patch
>
>
> Creating accumulator with explicitly specified name equal to _null_, like the
> following
> {code:java}
> sparkContext.accumulator(0, null)
> {code}
> throws exception at runtime
> {code:none}
> ERROR | DAGScheduler | dag-scheduler-event-loop | Failed to update
> accumulators for task 0
> java.lang.NullPointerException
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at
> org.apache.spark.util.AccumulatorV2$$anonfun$1.apply(AccumulatorV2.scala:106)
> at scala.Option.exists(Option.scala:240)
> at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:106)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1091)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
> The issue is in wrapping name into _Some_ instead of _Option_ when creating
> accumulators.
> Patch is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
