GitHub user yrenat added a comment to the discussion: Proposal: Automatically terminate idle Kubernetes Computing Units
@chenlica **Ray**: According to the Ray Kubernetes documentation, a Ray node is implemented as a Kubernetes pod: the head node runs in one pod, and each worker node also runs in a pod. The Ray autoscaler scales the cluster by adding or removing worker pods. **Dask**: Dask is also similar at the implementation level. A Dask cluster contains a scheduler pod and a set of worker pods. **Spark/Flink**: Spark and Flink also map their runtime components to Kubernetes pods. Spark uses driver and executor pods, while Flink uses JobManager and TaskManager pods. >From my understanding, I think the main difference from Texera is that Texera >exposes the CU (which sits on top of a k8s pod) as a user-controllable and >reusable resource with its own ownership and access control, resource >configuration, lifecycle state, runtime endpoint, and workflow-execution >association.However, Ray and Dask generally expose the cluster as the >higher-level abstraction, while individual pods represent nodes or workers >inside the cluster. Normal users do not directly manage or select these pods. Personally, based on my recent research, I also feel that our current CU abstraction is sort of "bulky". A CU currently combines several responsibilities, including access control, resource configuration, runtime location, lifecycle management, and workflow association. My concern is that mixing all of these responsibilities directly to the CU may make CU creation, removal, and future changes to its runtime implementation more complicated. I am also thinking about whether we could separate the logical CU abstraction from some of these lower-level details. For example, a CU could reference or be linked to separate resource, runtime, and access-control information, rather than directly owning all of these attributes. GitHub link: https://github.com/apache/texera/discussions/6264#discussioncomment-17642746 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
