Hi Dongjoon, Thank you for sharing about Old Protocol and clearing my doubt. I was able to understand the difference between Spark 2 & 3. For now `KubernetesLocalDiskShuffleDataIO` works fine for me.
Thanks, Arun Ravi M V B.Tech (Batch: 2010-2014) Computer Science and Engineering Govt. Model Engineering College Cochin University Of Science And Technology Kochi arunrav...@gmail.com +91 9995354581 Skype : arunravimv On Fri, 11 Aug 2023 at 23:52, Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Hi, Arun. > > SPARK-35593 (Support shuffle data recovery on the reused PVCs) was Apache > Spark 3.2.0 feature whose plugin follows only the legacy Spark shuffle > directory structure to be safe. > > You can see the AS-IS test coverage in the corresponding > `KubernetesLocalDiskShuffleDataIOSuite`. > > > https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala > > To be clear, Apache Spark keeps the supported directory structure without > any changes for historic reasons. > > You can use different structures by simply implementing your own plugin > like KubernetesLocalDiskShuffleDataIO. It's extensible. > > Dongjoon. > > > On Fri, Aug 11, 2023 at 4:52 AM Arun Ravi <arunrav...@gmail.com> wrote: > >> Hi Team, >> >> I am using the recently released shuffle recovery feature using >> `KubernetesLocalDiskShuffleDataIO` plugin class on Spark 3.4.1. >> >> Can someone explain why the mount path has spark-x/executor-x/ pattern >> dependency? I got this path detail from this PR >> <https://github.com/apache/spark/pull/42417>. Is it to avoid other >> folders in the volume ? Also, does this mean the path should use executor >> ID and spark app id or just hardcoded spark-x/executor-x/? Sorry, I >> couldn't fully understand the reasoning for this. Any help will be super >> useful. >> >> >> Arun Ravi M V >> B.Tech (Batch: 2010-2014) >> >> Computer Science and Engineering >> >> Govt. Model Engineering College >> Cochin University Of Science And Technology >> Kochi >> arunrav...@gmail.com >> +91 9995354581 >> Skype : arunravimv >> >
