Dear spark users, Is there any mechanism in Spark that does not guarantee the idempotent nature? For example, for stranglers, the framework might start another task assuming the strangler is slow while the strangler is still running. This would be annoying sometime when say the task is writing to a file, but have the same tasks running at the same time may corrupt the file. From the documentation page, I know that Spark's speculative execution mode is turned off by default. Does anyone know any other mechanism in Spark that may cause problem in scenario like this?
Thanks Ji -- The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.
