thank you all. to clarify i want to deploy celeborn in k8s. while our spark is running in amazon EMR. but the networking is straightforward if everything is in the same k8s context. i am just wondering if anyone has gone the route where bare metal spark on yarn is using celeborn which resides inside k8s.
thank you again for all your help. i very much appreciate it. rahul On Monday, June 17, 2024, Cheng Pan <pan3...@gmail.com> wrote: > There is a document[1] to guide how to deploy Celeborn on K8s. One notable > network configuration is > > celeborn.network.bind.preferIpAddress=false > > [1] https://celeborn.apache.org/docs/latest/deploy_on_k8s/ > > Thanks, > Cheng Pan > > > On Jun 18, 2024, at 04:53, rahul gidwani <chu...@apache.org> wrote: > > Hello, > > I was wondering if anyone here has tried running celeborn in k8s for spark > on EMR? Our spark clusters run in EMR currently and want to utilize > celeborn as our shuffle, but was wondering if there was any tricky > networking setup to get this working? Since I assume that any node in EMR > would eventually talk directly to a worker node. > > Thank you > > >
