Re: Dynamic allocation on K8

[Shrikant Prasad]

Hi Nikhil,

Spark on Kubernetes supports dynamic allocation using shuffle tracking
feature instead of the external shuffle service.
In order to enable dynamic allocation, you should set these two configs as
true:
spark.dynamicAllocation.enabled and
spark.dynamicAllocation.shuffleTracking.enabled

Regards,
Shrikant


On Tue, 25 Oct 2022 at 10:44 PM, Nikhil Goyal  wrote:

> Hi folks,
> When running spark on Kubernetes is it possible to use dynamic allocation?
> Some blog posts
> 
> mentioned that dynamic allocation is available, however I am not sure how
> it works. Spark official docs
> 
> say that shuffle service is not yet available.
>
> Thanks
>
> Nikhil
>
-- 
Regards,
Shrikant Prasad

Dynamic allocation on K8

[Nikhil Goyal]

Hi folks,
When running spark on Kubernetes is it possible to use dynamic allocation?
Some blog posts

mentioned that dynamic allocation is available, however I am not sure how
it works. Spark official docs

say that shuffle service is not yet available.

Thanks
Nikhil