GitHub user yrenat added a comment to the discussion: Proposal: Automatically terminate idle Kubernetes Computing Units
> Hi @yrenat We need this feature and thank you for working on this task! I did > some research before and concluded that Kubernetes does not natively support > this feature. I agree we need to have our own way mechanism to terminate idle > CUs. > > What's missing in both this discussion post and your PR is how you defined > what "idle" CU means and the architectural design of how computing unit > manager monitors all the CUs. Can you be more specific on those parts? Please > include some visual diagrams if you can. We need to agree on the design > before reviewing the PR. Hi @kunwp1 For this proposal, I define a CU as idle when: there is no workflow currently running on the CU, and the CU has had no workflow activity for longer than the configured idle timeout. The idea is that a CU should not be considered idle simply because its CPU or memory usage is low. A workflow may still be using the CU even when resource usage is temporarily low. Instead, the cleanup decision is based on Texera's workflow state. We first make sure that no workflow is actively using the CU, and then check how long it has been since the CU was last used. For a newly created CU that has never run a workflow, its creation time is treated as the beginning of its idle period. As for the design, my proposed idea is to let the Computing Unit Managing Service periodically check all active Kubernetes CUs. During each check, the service gets the Kubernetes CUs that are still active and examines them one by one. For each CU, it first checks whether there is a workflow currently running on it. If there is, the CU is kept and no further cleanup action is taken. If there is no running workflow, the service checks when the CU was last used by a workflow. If the CU has remained unused for longer than the configured timeout, it is terminated. Otherwise, it is kept for the next check. So the design uses one centralized periodic cleanup process in the Computing Unit Managing Service. It does not create a separate timer or monitoring process for each CU. GitHub link: https://github.com/apache/texera/discussions/6264#discussioncomment-17609674 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
