unsubscribe
unsubscribe
From: Mich Talebzadeh
Sent: Tuesday, August 8, 2023 4:43 PM
To: user @spark
Subject: [EXTERNAL] Use of ML in certain aspects of Spark to improve the
performance
I am currently pondering and sharing my thoughts openly. Given our reliance on
gathered
I am currently pondering and sharing my thoughts openly. Given our reliance
on gathered statistics, it prompts the question of whether we could
integrate specific machine learning components into Spark Structured
Streaming. Consider a scenario where we aim to adjust configuration values
on the fly
So if you disable shuffle tracking but enable shuffle block decommissioning
it should work from memory
On Tue, Aug 8, 2023 at 4:13 AM Mich Talebzadeh
wrote:
> Hm. I don't think it will work
>
> --conf spark.dynamicAllocation.shuffleTracking.enabled=false
>
> In Spark 3.4.1 running spark in k8s
>
Hm. I don't think it will work
--conf spark.dynamicAllocation.shuffleTracking.enabled=false
In Spark 3.4.1 running spark in k8s
you get
: org.apache.spark.SparkException: Dynamic allocation of executors requires
the external shuffle service. You may enable this through
spark.shuffle.service.ena