aglinxinyuan commented on code in PR #4206:
URL: https://github.com/apache/texera/pull/4206#discussion_r3308566712
##########
common/workflow-core/src/main/scala/org/apache/texera/amber/core/workflow/PhysicalOp.scala:
##########
@@ -198,6 +198,7 @@ case class PhysicalOp(
// schema propagation function
propagateSchema: SchemaPropagationFunc = SchemaPropagationFunc(schemas =>
schemas),
isOneToManyOp: Boolean = false,
+ isLoopEnd: Boolean = false,
Review Comment:
Good point — fixed in 540b7ba274.
The branch had since moved to detecting loop ends by string-matching the
operator id (`startsWith("LoopEnd-operator-")`), which keys the behavior to the
operator type even more tightly than `isLoopEnd` did. Replaced both with a
behavior-named flag:
* `PhysicalOp.outputSurvivesReExecution: Boolean = false` — documented as
"this operator's output storage must be preserved across re-executions of its
region instead of being recreated fresh", explicitly noting any operator can
set it, not just Loop End.
* `LoopEndOpDesc` sets it via `.withOutputSurvivesReExecution(true)`.
* `RegionExecutionCoordinator` now checks
`region.getOperators.exists(_.outputSurvivesReExecution)` instead of the id
prefix.
So the scheduler reasons about the property rather than the operator, and a
future operator needing the same treatment just sets the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
