jerrypeng commented on PR #56055:
URL: https://github.com/apache/spark/pull/56055#issuecomment-4606324907
@mridulm we can do that, but that is probably not the most eloquent or
simplest solution.
A multi-stage query isn't a list of independent map stages — it's a
connected stage DAG the planner already produces: shuffle-map stages at each
exchange, a result stage at the sink, real shuffle-dependency edges between
them, and branching where it exists (e.g. a join stage reads two shuffle
inputs). To emulate that as "N map stages wired together" we'd have to re-cut
the plan into separate jobs, pre-mint and inject shuffleIds to recreate the
dependency edges, and re-implement cross-job failure/cancellation/completion so
the micro-batch still behaves as one unit — i.e. re-derive the DAGScheduler's
own stage decomposition and job coordination in streaming code, to fake a DAG
we already have natively.
So instead of faking it, we keep the real plan and its real stages, and
change the only thing that's actually different in real-time mode: when the
stages run. Normally a stage waits for its parent to finish; here the stages
run at the same time, connected by the streaming shuffle. That's a scheduling
decision about an existing DAG, which is why it belongs in the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
