tustvold commented on issue #2581: URL: https://github.com/apache/arrow-rs/issues/2581#issuecomment-1227558948
> Also, I was not aware of [this document](https://github.com/spaceandtimelabs/arrow-ballista/blob/master/docs/source/user-guide/distributed/deployment/kubernetes.md), or more specifically: > > I believe I now understand how others are deploying Ballista in k8s - using shared storage to allow any executor to access any data. That's not what that is doing, that's creating a single PVC which can be attached to a single executor pod, and then associating a local path with a PV for that PVC. This likely will not work for more than a single executor, and won't work across nodes. It's not something I would advise doing... If you need local scratch space I'd recommend using EmptyDir or similar, w.r.t using shared storage for sharing data between executors - that is likely a dead-end. Kubernetes is really not designed for that deployment model, not to mention most cloud providers don't support multi-mount block storage anyway. > I agree that having the option for the JDBC driver to just interact with the scheduler and not require the ability to reach the executors is highly desirable and will make it much easier for users to try Ballista out Agree with the sentiment, disagree with the mechanism. Why not just provide k8s manifests that deploy Ballista idiomatically, in this case adding a kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) for the executors? The user would then just run `kubectl apply -f manifest.yaml` and be done. Kubernetes is specifically designed to make this easy and not need application support. That all being said if Ballista wants to support proxying via the scheduler I'm not going to block it, it just seems a little odd to buy into all the overheads of kubernetes and not actually use it :sweat_smile: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
