Hello all, we are running into some issues while attempting graceful decommissioning of executors. We are running spark-thriftserver (3.2.0) on Kubernetes (GKE 1.20.15-gke.2500). We enabled:
- spark.decommission.enabled - spark.storage.decommission.rddBlocks.enabled - spark.storage.decommission.shuffleBlocks.enabled - spark.storage.decommission.enabled and set spark.storage.decommission.fallbackStorage.path to a path in our bucket. The logs from the driver seems to suggest the decommissioning process started but then unexpectedly exited and failed while the executor logs seem to suggest that decommissioning was successful. Attached are the error logs: https://gist.github.com/yeachan153/9bfb2f0ab9ac7f292fb626186b014bbf Thanks in advance.