Github user viirya closed the pull request at:
https://github.com/apache/spark/pull/2217
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54396114
Thanks for your comments. I agree with Reynold too that it is a rough idea
and I need much consideration with a clear design doc for that if I think we
really need that pr
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54381921
This discussion probably belongs in the dev/user list, but a simple
workaround is to create an RDD of your 'mutable' state in the first iteration,
cache it while computi
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54369065
I agree with Reynold; mutability seems like it would add a lot of potential
for confusion in exchange for not much benefit. Even if we did decide that
mutability was a
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54344150
Thanks for the PR. Do you mind closing it?
In any case, this changes a fundamental assumption in broadcast variables,
and can brings in potentially unsafe consequenc
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54342643
Why don't you just use a global static variable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proj
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54295028
@srowen Thanks for comment. In fact I want some persistent mutable states
per data partition. I just achieve that goal with mutable broadcasted
variables. I know that a br
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54281202
@viirya I also don't quite get why mutability helps the use case you
describe. You seem to want some persistent mutable state per executor. But that
isn't the purpose of a
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54279202
@rxin, I can get the idea that immutability makes the whole thing safer for
broadcasted variables. So I am just wondering if it is worth providing such
mutability for the
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54278826
@rxin. I need a way to modify broadcasted variables locally and keep those
variables for later use. The locally modified variables are used to store some
values calculated
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-54261213
Why do you need to change the mutability of broadcast variable in order to
enable that? In general, immutability makes the whole thing safer.
You should still be abl
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-53983398
Thanks for your comments. For the possibility to cause an exception on an
executor, it happens when `synchronized` is not there. As `setValue` is wrapped
in `TorrentBroadc
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-53977801
Hi there,
The immutability of broadcast variables might be assumed in other places in
the code base. Since this approach requires re-broadcasting the entire cont
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2217#issuecomment-53967605
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/2217
[SPARK-3327] Make broadcasted value mutable for caching useful information
This PR makes broadcasted value mutable for caching useful information when
implementing some algorithms that iteratively ru
15 matches
Mail list logo
