GitHub user tanishqgandhi1908 edited a comment on the discussion: Proposal: Runtime images — bring your own image for a computing unit
This implements the reduced scope Ali described above with the demo video at the bottom walks through it. **User experience - the admin** - Go to Admin→ CU Images, fill in two boxes, and a link to the image. - The link can be a Docker Hub page address copied from the browser, or a plain owner/name: 1.0. - The row shows Pending → Mirroring → Ready (or Failed), and a Log button shows the copy happening line by line. - Expect about 5 minutes for a real image - we measured ~4.3 GB on one. - If something's wrong - tag doesn't exist, image is private, not a Texera image - the log says so plainly and tells you what to do instead. **User experience - user using models** - Creating a computing unit now has an image dropdown. Pick one, or ignore it and get the standard one. - The choice is made when the unit is created and stays for its life. You can't swap an image under a running unit. **Architecture** - The cluster runs its own small registry. It has a fixed address, and its storage survives restarts - otherwise every image would need re-copying after a reboot. - Copying is a background job. A tool called skopeo copies registry-to-registry. No download-and-repack, so it stays fast and unprivileged. That job's output is the log you see. - The image is checked before any of it is copied. We read just the image's config, a few kilobytes, so a wrong link fails in seconds instead of after gigabytes. - Each copy gets its own tag. Refreshing an image publishes a new one and leaves the old alone, so units already running keep the exact bytes they started with. - Images are stored in pieces, and a piece the registry already holds is neither fetched again nor stored again. So a second image built on the same base costs only its own layers - in disk and in time. **What happens when a unit is created** - The image is checked first, before anything is saved. It must be Ready, and the registry must really still have it. - The image is simply substituted into the pod — one container, one image. Nothing else about the unit changes: same limits, same permissions. - The container runs as a non-root user, and that's pinned by the pod, not left to the image to decide. - The unit remembers the image's name, not just its ID - so if the image is later deleted, "what is this running?" still has an answer. - If anything goes wrong mid-creation, the half-made pod is cleaned up so nothing is left stranded. **For database: one new table: cu_image** Please find the demo video below to understand user experience Please let me know if anything here needs changing, or if there's anything you'd want covered before I open a PR. Demo Video: https://github.com/user-attachments/assets/69925954-afe1-40c0-acfa-77a062ec430c GitHub link: https://github.com/apache/texera/discussions/7812#discussioncomment-18298466 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
