Hello all, Calcite is great! However, I think that there are several areas where the project could do better: - testing - documentation - legacy code cleanup
(1) Testing: Most of my work in Calcite so far has been about testing. I have contributed the SqlLogicTest suite, which uncovers many bugs. However, I have not filed issues for any of these bugs, and I have not fixed any of these bugs either, since I have been very busy with other bugs I found using other testing methods. I also have written a new test fixture used for SqlOperatorTest tests, which finds lots of bugs, but I haven't sent a PR for this one since there are too many failures. I was hoping other people would be looking at the SLT bugs. (One other test fixture I proposed was merged, though, and I hope it will prevent quite a few bugs too: https://github.com/apache/calcite/pull/3433). I think testing could be improved a lot by getting Calcite to run more tests from other open-source projects (e.g., Postgres, or MySql). For example, it would be great to have a script to convert Postgres regress tests https://github.com/postgres/postgres/tree/master/src/test/regress/expectedto to Quidem tests. These could be slow tests. (2) Documentation; Calcite has a steep learning curve. I think that better documentation could help attract more developers and users. I would like to propose adding a blog to the Calcite project web page. If every committer writes two blog posts per year about some Calcite technical aspect, then the project can quickly accumulate a lot of helpful information. The dev mailing list does not seem to be well indexed by search engines; we could even take some answers for questions on the mailing list and turn them into short articles. I have written a blog post about several Calcite IRs, which I plan to post in the following days on our company website (I will actually send a request for review on this list when I do), but it would be great to cross-post such documents on the Calcite project blog. (3) While poking around I have found examples of data structure that are probably confusing. Some of them have been known to be confusing for a long time: e,g: RexLiteral.typeName. https://github.com/apache/calcite/blob/c83ac69111fd9e75af5e3615af29a72284667a4a/core/src/main/java/org/apache/calcite/rex/RexLiteral.java#L214 A comment from 2006 indicates that this field should not exist. Another example is https://github.com/apache/calcite/pull/3411, where I tried to fix up some problems with the type families. Such PRs are difficult to accept, in the short term they may increase the code entropy (like this one does, by creating deprecated functions and keeping the old ones too), and it is very hard to make sure that they don't break anything for users, and their value is questionable if people do not actually remove all uses that should be deprecated. This requires a sustained effort to cleanup; contributors could periodically self-assign a small cleanup task. Thank you, Mihai -----Original Message----- From: Stamatis Zampetakis Sent: Sunday, October 22, 2023 9:00 AM To: [email protected] Subject: [DISCUSS] State of the project 2023 Hello community and happy birthday Calcite, Eight years ago (22 October 2015) Calcite graduated as a top-level Apache project [1]. At that point, the community decided to have an annual “state of the project” discussion, and we arrived at that time of the year. We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023 [2], with probably one more coming before the end of the year. With roughly 300 commits coming in, it is evident that the community did a great job in fixing numerous bugs, landing new features, and bringing notable improvements and optimizations to the project. It is worth highlighting the collective effort that was done for improving the BigQuery dialect where more than 60 new SQL functions were added. Regarding the sub-project Avatica, we had a release at the beginning of the year (1.23.0) [3] featuring some new configurations for fetchSize and SSL key stores along with various other improvements and bug fixes. Since the last release, there have been roughly 10 new commits in the main branch mostly comprising dependency upgrades and build fixes. As in previous years the activity is rather low but there are still volunteers regularly checking and contributing to the repo. In terms of community, I think this has been a great year. We see more and more people participating in email discussions, Jira tickets and Github PRs. Our list of committers has grown with Istvan Toth, Alex Plehanov, Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ Banghart, Dan Zou; and so has our PMC with Benchao Li joining the team. With nine new committers and one new PMC member till now, it's probably the best year so far (and it's not yet ended). Calcite grows and evolves because of (and thanks to) its community, so I would like to thank everyone for being part of this family and working together in a respectful and motivating environment. It was nice to see our community members giving talks to conferences such as ApacheCon East Asia, VLDB, and Community over Code presenting Calcite and/or its application. As in previous years, we had one online Calcite meetup, which was a great opportunity for the community to virtually meet and share some interesting presentations, and hopefully we could add some in-person events in the years to come. Over the past few years we always had problems with reviewing pull requests with a lot of weight falling upon a few individuals. In the last board report for (Q3 2023), the numbers were a bit more encouraging showing more people involved in reviews. The problem is not yet solved but we are moving in a promising direction. To conclude, I will repeat the questions from previous years: 1) What else are we doing well in the project? 2) What areas do we need to do better? Please take some time to share your thoughts! Note that this discussion is for everyone, not only for committers / PMC members; even if you have never sent an email to the dev list before, now is a good time to do so :) Finally, it has been a privilege to serve as Calcite's PMC Chair this year. I have learnt a lot and I am very grateful for the opportunity that I was given. Following our yearly rotation tradition, I will step down as Chair by the end of the year, and a new one will have to be chosen. As we discussed some time ago [4], if you have any suggestion and you would like to put someone forward as a potential next Chair, please send an email to [email protected]. The nomination period for the new chair will remain open for the next two weeks (till 2023-11-05). The PMC will study all proposals, discuss, and start a vote soon after 2023-11-05. The change will be effective some time in December once the resolution is approved by the board. Best, Stamatis [1] https://calcite.apache.org/news/2015/10/22/calcite-graduates/ [2] https://calcite.apache.org/news/ [3] https://calcite.apache.org/avatica/news/ [4] https://lists.apache.org/thread/gplfqs4snr1b6h62cngyvb65sz41z3fk
