GitHub user jsjasonseba added a comment to the discussion: Deploying Apache Celebron for Large-Scale Spark SQL Workloads
Thanks for sharing, this makes sense. I was mostly thinking about case 1, autoscaling from cluster utilization metrics. But your response gave me good insight into case 2, where some jobs can have much bigger shuffle needs than the others, so metric-based scaling may not be enough. I also recently discovered this PR discussion related to HPA-based scaling, which might be relevant for a Kubernetes-based autoscaling of case 1 https://github.com/apache/celeborn/pull/2776#issuecomment-2398989795 I’m planning to try this on EKS and share what I find. GitHub link: https://github.com/apache/celeborn/discussions/3191#discussioncomment-16835712 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
