tanishqgandhi1908 opened a new pull request, #8485: URL: https://github.com/apache/texera/pull/8485
### What changes were proposed in this PR? The first unit on a node waits for the whole image — about 80 seconds for a 3 GB one — while every later unit there starts at once. The same action takes seconds or minutes depending only on which node it landed on. Each ready image now gets a DaemonSet: an init container that is the image, with a command that does nothing, plus a pause container so the node does not reclaim what was just pulled. The same mechanism the chart already uses for the deployment's own image, built in code because a curated image is registered while the cluster is running, so a template cannot describe it. One per image, keyed on the image id, so a refresh that resolves a moved tag repoints the existing pre-pull rather than adding a second one holding bytes nothing runs. Removed when the image is. Three moments, because the service has no background threads: an image reaching READY, an image being deleted, and a read of the list — which also gives a pre-pull to any ready image that has none, since nothing else revisits a row once it is ready. Best-effort throughout. A pre-pull that cannot be created is logged and ignored: the image still works, and the first unit on each node just pays for the pull. Costs node disk, since every node holds every ready image, so `curatedImages.prepull.enabled` turns it off. ### Any related issues, documentation, discussions? Closes #8469 Part of #8466 Stacked on #8475, which this calls into. The diff against that branch is the 8 files above. ### How was this PR tested? Seven new tests for the DaemonSet the code builds, plus the object applied to a real cluster. | Case | What it pins | | --- | --- | | init container | the image is what gets pulled, and its command is a no-op | | pull policy | `IfNotPresent` — a digest cannot resolve to different bytes later | | containers | only pause keeps running, so no unit runs that nobody asked for | | tolerations | `Exists`, so tainted nodes are not the ones left waiting | | name | depends on the image id alone, so a refresh replaces rather than adds | | selector | `app` only — a DaemonSet's selector is immutable, so it must hold nothing that changes | | labels | the owner and image labels a listing reads | ``` sbt "ComputingUnitManagingService/test" "Config/test" scalafmtCheckAll ComputingUnitManagingService 182 passed, 0 failed Config 72 passed, 0 failed scalafmtCheckAll clean ``` `helm template` renders the manager Role gaining `daemonsets` and the two new environment variables. **On minikube,** applying the object the code actually builds: ``` daemonset rolls out, pod Running alpine appears in the node's image cache refresh to a different digest -> "configured", still one daemonset, new image also cached listing by label -> cu-image-prepull-1 -> texera-cu-image=1 delete removes it ``` The refresh case is the one worth having: a DaemonSet's selector cannot be changed after creation, so a selector carrying anything mutable would make every refresh fail permanently. Also confirmed against the cluster rather than assumed: deleting a pre-pull that does not exist returns an empty result instead of throwing, so an image that never reached READY is a silent no-op rather than a warning on every delete. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
