avantgardnerio commented on issue #2581: URL: https://github.com/apache/arrow-rs/issues/2581#issuecomment-1227605371
> This likely will not work for more than a single executor, and won't work across nodes. > multi-mount block storage Ah, yes, I was surprised to see it not be of type `NFS`, and this was my hesitation to go this route - shared storage seemed like a nightmare. > a kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) for the executors From the document you linked: > Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them. This is what I still don't understand: if there is data locality to the executor that did the processing, and kubernetes is loadbalancing / DNS round robin / whatever to a random pod, then what happens if it routes the client TCP connection to executor pod A when the Flight results reside on local disk on executor pod B? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
