Hi Yifeng! I can tell that the most spectacular step toward tez on Kubernetes was the unmanaged sessions umbrella: https://issues.apache.org/jira/browse/TEZ-3991 I can see some of those tasks are still not committed, however, unmanaged sessions have been working for years in a Cloudera product, so we would be happy to review these if someone picked them up and created a PR.
Once unmanaged sessions are done, we can explore other ways. Tez is still tightly coupled with Yarn I believe, so on the road to k8s, we have to explore the following: 1. *Tez AM* to be started by another resource manager than Yarn (which is slightly different from the unmanaged sessions above, as unmanaged sessions start independently of the client (e.g. Hive's hiveserver2)) <- this is not a must-have as long as the unmanaged sessions approach is fine for the K8s use-cases. 2. *Tez containers: t*his is a huge gap: the product I know works by Hive LLAP as executors, so similarly to the sessions, the Hive LLAP containers start independently of Tez, so if we want to make this work in Tez out of the box, we need to implement new things like start a TezChild container or rather a group of TezChild containers as a k8s pod for instance, just thinking aloud. Thanks for coming up with this topic. Let us get back to this later if any plans, in the meantime, finishing the upstream contribution of TEZ-3991 is more than welcome. Regards, Laszlo Bodor YifengLu <noisymons...@163.com> ezt írta (időpont: 2024. jan. 29., H, 3:52): > Thank you very much for your response. We will continue to explore Tez on > Kubernetes, and if there are any further developments, we will reach out to > the community. Once again, thank you. > Best regards, > Yifeng Lu
