Fwd: Will Spark ever run the same task at the same time

First, the word you are looking for is "straggler", not "strangler" -- very different words. Second, "idempotent" doesn't mean "only happens once", but rather "if it does happen more than once, the effect is no different than if it only happened once". It is possible to insert a nearly limitless v

Re: Will Spark ever run the same task at the same time

> On 16 Feb 2017, at 18:34, Ji Yan wrote: > > Dear spark users, > > Is there any mechanism in Spark that does not guarantee the idempotent > nature? For example, for stranglers, the framework might start another task > assuming the strangler is slow while the strangler is still running. This

Will Spark ever run the same task at the same time

Dear spark users, Is there any mechanism in Spark that does not guarantee the idempotent nature? For example, for stranglers, the framework might start another task assuming the strangler is slow while the strangler is still running. This would be annoying sometime when say the task is writing to