mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-751979314
Before answering specific queries below, I want to set the context.
a) Enabling proactive replication could result in reduced recomputation cost
when executors fail.
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-751934621
Specifically for this
[usecase](https://github.com/apache/spark/pull/30876#issuecomment-750471287),
we dont need to make it a spark default right ?
If I understood right,
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-751926362
(Sigh, github prematurely posted my previous comment - fleshing it out here).
As I mentioned above, the flag helps applications which are fine with paying
the overhead
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-751922978
> Could you elaborate about your concern more specifically?
> What is the negative side-effect of very chatty and noisy?
> How severe it was?
As should be obvious,
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-751186157
@dongjoon-hyun proactive replication only applies to persisted RDD blocks,
not shuffle blocks - not sure if I am missing something here.
Even for persisted RDD blocks, it
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-750689540
Thanks for the details, that definitely sounds like a good rationale to
enable it by default for k8s: for other resource managers, this does not
necessarily apply.
One
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-750453290
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
mridulm commented on pull request #30876:
URL: https://github.com/apache/spark/pull/30876#issuecomment-750448981
Would be nice if we hold off merging when there is ongoing discussion,
unless there is an immediate need to push changes (like hotfix)
