Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54295028
@srowen Thanks for comment. In fact I want some persistent mutable states
per data partition. I just achieve that goal with mutable broadcasted
variables. I know that a broadcasted variable would be shared across many
tasks. So I do use a Map variable to separate different variables for different
data partition since I can differentiate them with partition index given by
mapPartitionWithIndex.
It might look at odds with Spark's design then. However, if mutable
Broadcast variable is a bad idea, can we have a mechanism to have persistent
mutable states per data partition? Or it is a bad idea too for Spark's design?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
