Hi Shivang, This sounds like classic Spark-on-Kubernetes behavior: -
Executors do not necessarily shut down immediately after finishing their tasks, unless: - Dynamic resource allocation is enabled. - And Spark knows it can safely scale down - The Driver pod manages the whole lifecycle. If you’re not using dynamic allocation or spark.kubernetes.executor.deleteOnTermination, executors live until the driver completes the job and exits. Try, spark.dynamicAllocation.enabled=true spark.dynamicAllocation.shuffleTracking.enabled=true This should address your first issue. Best Regards Soumasish Goswami in: www.linkedin.com/in/soumasish # (415) 530-0405 - On Fri, Apr 4, 2025 at 4:43 AM Shivang Modi <sm...@provenir.com.invalid> wrote: > Hi Team, > > > > We are using spark java 3.5.3 and we have a requirement to run a batch of > millions of transactions. i.e. I am running batch of 1M with 640 executors > and each executor is having 8 cores and 16 GB memory running under > kubernetes cluster. > > > > But our conversation is that one executor is done with completed task, it > is not going away. All executors are released at the end of job with driver > pod. > > > > And also, any performance suggestion which can help in reducing the > timing. Currently we have 10M running with 640 executors and taking 60 plus > mins to be finished. > > > > Let me know if need more information. > > > > Thanks, > > Shivang. > > *This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended > solely for the use of the addressee(s). If you are not the intended > recipient, please notify the sender by e-mail and delete the original > message. Further, you are not to copy, disclose, or distribute this e-mail > or its contents to any other person and any such actions maybe unlawful*. > This e-mail may contain viruses. Provenir has taken every reasonable > precaution to minimize this risk, but is not liable for any damage you may > sustain as a result of any virus in this e-mail. You should carry out your > own virus checks before opening the e-mail or attachment. Provenir reserves > the right to monitor and review the content of all messages sent to or from > this e-mail address. Messages sent to or from this e-mail address may be > stored on the Provenir e-mail system. > > > > Think before printing - please consider the environment >
