Hi, Thanks for initiating the discussion. I’d be glad if I were allowed to present my opinion as one of Hive on Tez users.
I can see we have one immediate problem and one fundamental problem. First, only a few PMC members are voting for 0.10.4 today. Second, it is tough to nominate a committer or PMC continuously. I don’t have a short-term idea of how to resolve the first problem. As for the second problem, I think Tez should ideally be a part of Hive for some reasons. Why? I sometimes feel two kinds of difficulties in making a consistent effort on Tez. The first one is related to the scope of Tez. The duties of Tez are as narrow as MapReduce or a part of the core of Spark/Flink. Though I sometimes want to improve Tez[1], I more often find areas of improvement about Tez on the Hive side[2]. The second one is Tez's fantastic stability. I usually don’t see many bugs in Tez. Because of these two reasons, it is really hard for a single person to contribute continuously to Tez. I think Hive’s PMC is knowledgeable enough to accept Tez, I guess ASF allows Tez PMC to transfer Tez to Hive PMC[3], and I believe Hive users can benefit from the transfer. I am a bit concerned that it could make it hard to maintain Tez as an independent library for other projects, such as Apache Pig, though. This is just one opinion. Anyway, I am confident that Tez is very competitive, and we’re actually trying to evolve it more, e.g., by improving Fair Routing or adding some innovative features. Thanks for the community’s effort! - [1] I am working on TEZ-3209 to introduce Fair Routing - [2] I have found some places to improve integration with Tez, e.g., HIVE-23831, HIVE-24485 - [3] At least, PMC is likely to be able to salvage an attic project: https://attic.apache.org/ Regards, Okumin On Fri, Sep 6, 2024 at 9:49 PM László Bodor <bodorlaszlo0...@gmail.com> wrote: > > Hi Stamatis! > > Thanks for starting this discussion. > This situation is getting worse, and we still haven't had the chance to > promote new folks to become PMC. Several aspects here: > > 1. Tez is in a local optima in a sense that it's a stable execution engine > under Hive, and most of the improvements are implemented in Hive nowadays. > Even though there are open Tez tickets, we don't have large projects - > however, there is always some stuff - to work on, hence the number of > potential commits decreased in the last years, making it harder to > contribute. > 2. Most of the used-to-be-active PMC members are not active anymore, more > or less because they left companies that contributed to Hive + Tez the > most, and they left sooner than we had been able to involve new people. > > Some thoughts about the future: > 1. As long Tez is the default execution engine in Hive, it doesn't make > sense to move it to the Attic. > 2. Consider using special project-level by-laws to make a release with > committer votes, not only PMC member votes <-- I'm afraid it's not possible > due to ASF by-laws. > 3. New PMC members: let people become committers, then PMC members with > fewer commits, by considering their activities with more weight (still: > commits + code reviews are needed), or considering their activity in e.g. > Hive Runtime development (which is closely related to Tez, but doesn't > always involve changes in Tez code) > > Regards, > Laszlo Bodor > > > Stamatis Zampetakis <zabe...@gmail.com> ezt írta (időpont: 2024. szept. 5., > Cs, 10:55): > > > Hey all, > > > > The most recent vote for Apache Tez 0.10.4 has been open for almost a > > month [1] now. It seems that the project struggles to gather binding > > votes indicating that the majority of the PMC is inactive (or lost > > interest). If this situation continues the project will have to move > > to the Attic [2] and this will be a huge setback for dependent > > projects especially for Hive. > > > > I would like to start some discussions to gather ideas on how we can > > avoid this situation since Tez is the main execution engine that is > > used by Hive and there are no immediate plans to replace it. > > > > Best, > > Stamatis > > > > [1] https://lists.apache.org/thread/yx5lgnxf7k8k7xvf31k37v9hz01j934f > > [2] https://attic.apache.org/ > >
