parshimers commented on code in PR #6897:
URL: https://github.com/apache/texera/pull/6897#discussion_r4049957426
##########
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionManager.scala:
##########
@@ -352,6 +357,23 @@ class RegionExecutionManager(
)
}
+ /**
+ * Mount every repository this region's operators name before any of them
receives its code,
+ * which already refers to the mount paths. It happens here rather than in
a worker because
+ * workers are created before either phase launches. Operators carry no
mount code: they only
+ * name what they need, through `PhysicalOp.mountLocators`.
+ */
+ private def mountRegionRepositories(): Future[Unit] = {
+ val mount = Promise[Unit]()
+ if (mountFutureRef.compareAndSet(null, mount)) {
+ mount.become(Future {
+ val locators = region.getOperators.flatMap(_.mountLocators)
+ if (locators.nonEmpty) ensureMounted(locators)
Review Comment:
ok. i do get why it is blocking but we might have to revisit this in the
future if it becomes a problem. i agree with you that it seems fine on the
happy path, but it doesn't seem particularly robust in the failure and edge
cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
