[
https://issues.apache.org/jira/browse/SPARK-43012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709224#comment-17709224
]
Mayur Bhosale commented on SPARK-43012:
---------------------------------------
[~gurwls223] Can you please take a look?
> Name based access of accumulators from tasks
> --------------------------------------------
>
> Key: SPARK-43012
> URL: https://issues.apache.org/jira/browse/SPARK-43012
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 3.5.0
> Reporter: Mayur Bhosale
> Priority: Minor
>
> At present, accumulators must be registered on the driver and subsequently
> accessed (i.e., added/reset) by the code running on the executor using the
> same object. As a result, the accumulator object must be passed throughout
> the code, which leads to verbosity, particularly in multi-module Spark
> applications where user code is distributed across multiple modules.
> Instead why not have name based access to accumulator so that objects don't
> need to be passed explicitly? (Sort of global named accumulators)
> Currently,
> {code:java}
> val accName = "custom_acc_experimental"
> val la = new LongAccumulator()
> sc.register(la, accName)
> sc.parallelize(0 to 10).mapPartitions(partition => {
> la.add(100L)
> partition
> }).count{code}
> Can be,
> {code:java}
> val accName = "custom_acc_experimental"
> val la = new LongAccumulator()
> sc.register(la, accName)
> sc.parallelize(0 to 10).mapPartitions(partition => {
> // Assuming new end-point
> AccumulatorV2.add(accName, 100L)
> partition
> }).count {code}
> I was able to do a [working POC of
> this|https://github.com/apache/spark/compare/master...mayurdb:spark:experiment]
> where the user defined accumulators are internally passed to task executable.
> Probably a new endpoint needs to be added for users to create such
> accumulators as handling all current accumulators in this manner would cause
> pressure on the task flow RPC. I can write a small doc on approach but wanted
> to get a ack if this seems workable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
