Abacn commented on PR #40103:
URL: https://github.com/apache/beam/pull/40103#issuecomment-5646735254
> So acquire waits, bounded ... to be stopped and then creates a fresh one
... Parallel pipelines in one JVM stay a Spark limitation and will be
documented.
Disallow more than one pipeline submitting at the same time in JVM would be
another regression
I feel the agent (if used) tend to add locks is a tendency, as it is a
generic (lazy) fix for concurrent issues. We should define what would be the
expectation. Considering current behavior
1. what part, currently working, need to be preserved:
a. non-blocking job cancellation
b. non-blocking job submission
2. what part currently working, good to be preserved:
- when jobs submitted and run in sequence, every job configuration is
honored (because previous one closed session so that this one starts a new)
3. what part causing issues, and, given the limitation of Spark API,
- what part need to be fixed
a. pipelineResult.cancel() doesn't cancel the job. For batch it's less
severe as long as job stops itself eventually. As we are adding streaming
capability, this needs to be fixed.
b. Session get cancelled crashing another job sharing session and is
translating
This caused flaky tests and fix it is found to be challenging under
the constraint of **need to be preserved**
- what part is tolerable
c. latter started job not honoring Conf due to reusing session (race
exists in current master as well as the fix).
After isolating each issue and requirements, here is my proposal:
1. Make cancellation actually cancel the job (fix 3a). This part's fix is
already well formed in this PR
2. parallel job submission does not crash (fixing 3b) under constraint of 1
and ideally 2.
To do this we may still need to track active jobs for sessions in some
way, but instead of introducing blocking waits, acknowledging possible conflict
in the case of parallel job submission (it's current behavior so no
regression), but make best effort to keep "when jobs submitted and run in
sequence, every job configuration is honored" (also current behavior)
Based on the observation in #40101, it's **translating** on a stopped
session (or stopped midway) causing crash. If there are two jobs both running,
and the first one ends and stopped its session, the second one can still run
and ends. It's only job management get affected (cancel request won't reach
job?). If this is true, we can make use of this fact to separate the effort of
fixing crash and a proper long term fix, thus make life easier
To fix crash, here is some idea
- We only need to track if there is a pipeline started translating and not
yet submitted to Spark. If so the session is not safe to stop
- We lazily stop previous session on subsequent job run when acquiring for a
session. If previous session is safe to stop, stop it before call getOrCreate;
otherwise just return the same session. This wouldn't eliminate race bug, but
it's still in align with current (incorrect but non-crashing) behavior that
overlapping job submission get same session with individual Conf not honored.
By doing this we still need only one synchronized acquire, and mostly keep
and simplified the current PR's structure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
