GitHub user jakubfijolek added a comment to the discussion: Deploying Apache Celebron for Large-Scale Spark SQL Workloads
For scaleup i have 2 drivers. 1. As majority of my workloads are run over JDBC on fleet of long running thriftservers 1st signal is simply total celeborn cluster utilization (polled from celeborn master API). This is to avoid saturating particular celeborn cluster by one job and blocking everything. This assumes any particular shuffle will easily fit the cluster base capacity and covers the case where no-of-jobs * their shuffle requirements go above cluster size. Each sparkSQL query either way have their own LifecycleManager so as long as there is enough capacity when the query starts it should finish successfully 2. For jobs that I know will need a lot of shuffle (ex. insert into table A as select from table B with repartition) i either do it manually for one offs or implement as part of Airflow dag to send scaleup signal. In most of the cases shuffle space needed can be roughly estimated by dataset size + replication multiplier (if used) + overhead multiplier (hardcoded) For scaledown signal is either utilization <threshold in 2nd case heavy job sends scaledown signal on exit. Decomissioning For now I'm simply using following sequence Exclude workers to be scaled down, wait, trigger scaledown on ASG, decomission workers (ASG lifecycle hook), terminate instances. I dont have this extensively tested and it's likely I'm running into cases where the node decomissions before data is fully expired but beauty of spark is that at worst case scenario it will do partial re-run of previous stage. https://celeborn.apache.org/docs/latest/decommissioning/ GitHub link: https://github.com/apache/celeborn/discussions/3191#discussioncomment-16815550 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
