Hey Jayabindu, We use thriftserver for on K8S. May I ask why you are not going for Trino instead? I know it didn't support autoscaling when we tested it in the past but not sure if it does now. Autoscaling also means that users might have to wait for the cluster to autoscale but that usually happens not so slow and once its done then other queries have the new nodes available. Also the workload on our thriftserver is not so large so it solves the purpose for now. You can also take a look at Apache Kyuubi.
I can put in some details below and attach the config we use for spark thriftserver, you can pick whatever is relevant for you: * We run thriftserver on default(stable) nodes and its executors on preemptible(spot) nodes * We use driver and executor templates to make above possible by using node selectors * We use fair scheduling to manage workload Mvg/Regards Saurabh ________________________________ From: Jayabindu Singh <jayabi...@gmail.com> Sent: 09 March 2023 06:31 To: u...@spark.incubator.apache.org <u...@spark.incubator.apache.org> Subject: [EXTERNAL] Spark Thrift Server - Autoscaling on K8 Caution! This email originated outside of FedEx. Please do not open attachments or click links from an unknown or suspicious origin. Hi All, We are in the process of moving our workloads to K8 and looking for some guidance to run Spark Thrift Server on K8. We need the executor pods to autoscale based on the workload vs running it with a static number of executors. If any one has done it and can share the details, it will be really appreciated. Regards Jayabindu Singh
spark-defaults.conf
Description: spark-defaults.conf
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
