Github user gkc2104 commented on the issue:
https://github.com/apache/spark/pull/17750
> In contrast, in the coarse grained mode the executor would send the 100MB
data to the driver directly without going through mesos agents. The only thing
that agents write to disk are small task status messages like
TASK_RUNNING/TASK_KILLED which are typically several KBbytes.
Do you then think it would be a viable option to enable it by default on
Coarse grained and have it not used in Fine-grained.
> The code in your link is the mesos cluster scheduler, which is a mesos
framework that launches spark drivers for you, not the mesos scheduler inside
the spark driver that launches executors.
This makes sense now, I definitely did not consider this , but this
explains it.
> In contrast you can't the call to reconcileTasks in e.g.
MesosCoarseGrainedSchedulerBackend.
Could you expand on this a bit more, I assume we could maintain the state
of the tasks similar to how driver state is maintained in
`MesosClusterScheduler`, and accordingly update state at crucial points, like
`start`.
I'll start implementing that, if you think we could enable it to
reconcileTasks with state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
