GitHub user yrenat added a comment to the discussion: Proposal: Automatically terminate idle Kubernetes Computing Units
Hi All, a brief update about the topic: 1. To provide a better picture of my [PR](https://github.com/apache/texera/pull/6046) that is related to this discussion, I think I should explain one of the key variables that I was using. One of the main functional variable to decide which CU has been idle is `last_active`, which is first introduced in this [PR](https://github.com/apache/texera/pull/2123). This PR aims to clean garbage workflow execution results after a configured TTL. It starts a recurring cleanup task that periodically finds expired executions and removes it. I use `last_active` in my code because a CU is considered occupied only if there is at least one workflow running in this CU. So, if all workflow inside the given CU is inactive, it is safe to say the whole CU is inactive. We can then clean it. 2. As for on-the-fly update of the CU parameters (`CU_idle_time_before_removal` in my PR, to be specific), I have tried the methods mentioned above, but it turns out that they don't fit my use case well. What we really need is a pathway at CU level, after the CU is created. Sadly I didn't find a corresponding pathway, and I guess it would be a moderate level task to build such a thing. So, for now, I think it's better to keep the PR as it is (i.e., provide a global `CU_idle_time_before_removal` that fits into all CUs) GitHub link: https://github.com/apache/texera/discussions/6264#discussioncomment-18039711 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
