mridulm commented on pull request #30139: URL: https://github.com/apache/spark/pull/30139#issuecomment-718528130
If it will be eventually consistent, that is fine: what we can’t have is permanent divergence (positive or to negative). The current situation does exhibit eventual consistency (if updates while computing, then it diverges from reality - but that is transient). If the value will never converge, that is a problem. The current pr, last I checked, has potential for this and that needs to be fixed. On Thu, Oct 29, 2020 at 2:14 AM Jungtaek Lim <[email protected]> wrote: > I somehow revisited the related configuration > spark.shuffle.maxChunksBeingTransferred and realized the default value is > set to Long.MAX_VALUE. (effectively off by default) > > That leads me to wonder, does the value (numChunksBeingTransferred) need > to be accurate at the specific time? Do we really need to make both updates > be in same critical section? Can we tolerate the skew and allow eventual > consistent of the value? > > For the concern of being diverged, I guess updates to both would happen > anyway unless there's no bug (I agree this is a tricky one to guarantee), > as we anyway do the update on both sequentially. Please correct me if I'm > missing something. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/30139#issuecomment-718410305>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMETFEYOUBJFR2AAVQL4BLSNEI43ANCNFSM4S4G3CXQ> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
