GitHub user yrenat added a comment to the discussion: Proposal: Automatically terminate idle Kubernetes Computing Units
@chenlica Based on my findings, this resource-waste problem is common. There are several generic solutions on "system" level, where they have a global control of what/how many resources are needed, but they can seldom address our specific problem where we need to control a specific pod/CU. The following are some of my findings, regarding garbage resource collection. 1. **HPA / custom metrics** This is the native idea from Kubernetes. It automatically changes the replica count of a scalable Kubernetes workload, such as a Deployment, ReplicaSet, ReplicationController, or StatefulSet, based on metrics like CPU, memory, or custom/external metrics. In simpler words, it can automatically scale up/down the number of pods/resources invovled. However, a Texera CU is a pod-specific per-user application resource with a DB record, access control, resource settings, and a corresponding runtime Pod (not just a pod). Also, “idle” is another Texera-specific, idea, which cannot be natively well-described by Kubernetes. Besides, HPA cannot fully shutdown the system. It has to have at least one pod running to keep track of all the computation resources. 2. **KEDA** KEDA (Kubernetes Event-driven Autoscaling) is designed for event-driven autoscaling and can scale workloads down to zero when event sources become inactive. This is a good solution for queue/event-driven services. However, using KEDA directly would require Texera to expose CU activity as an external/custom metric or event source, which will create multi layers of interference with Texera. It also would not automatically update Texera’s CU database state unless Texera adds extra integration. 3. **Knative Serving** Knative supports scale-to-zero for HTTP request/traffic-driven services. This is useful for serverless HTTP workloads, but a lot of Texera resources/commands are not handled in HTTP format. Adopting Knative would be a larger architectural change and still may not capture Texera’s definition of CU idleness. Besides, Knative also does not focus on pod/CU level logistics. To sum up, for the current issue, I think the most suitable first solution is to implement our tailored idle detection inside Texera’s Computing Unit Managing Service or a Texera-side scheduled cleanup component. The cleanup should use Texera’s own CU termination logic rather than directly deleting Pods, so that both the Kubernetes Pod and the Texera DB state remain consistent. GitHub link: https://github.com/apache/texera/discussions/6264#discussioncomment-17570089 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
