Hi devs, There is no doubt that Calcite boasts powerful capabilities, but the curve of acquiring and learning calcite is quite steep. After studying Calcite for a year, I have only gained a limited understanding of some of its modules. Mihai is about to contribute several documents related to calcite, which is truly exciting. Calcite is a treasure, and I hope there will be more developer documentation next year to enable developers to quickly grasp how to use calcite and how it works.
Secondly, Calcite supports a wide array of SQL syntax, but there are still a few syntaxes that it does not cover. I hope that Calcite can be equipped with more comprehensive SQL syntax parsing capabilities, which will be the selling point that convinces more developers to opt for Calcite. I am anticipating that Calcite will support more syntax next year, both standard and non-standard, and I will continue to contribute in this direction. Best, Hongyu Guo On Tue, Oct 24, 2023 at 3:16 AM <[email protected]> wrote: > Hello all, > > Calcite is great! However, I think that there are several areas where the > project could do better: > - testing > - documentation > - legacy code cleanup > > (1) Testing: Most of my work in Calcite so far has been about testing. I > have contributed the SqlLogicTest suite, which uncovers many bugs. However, > I have not filed issues for any of these bugs, and I have not fixed any of > these bugs either, since I have been very busy with other bugs I found > using other testing methods. I also have written a new test fixture used > for SqlOperatorTest tests, which finds lots of bugs, but I haven't sent a > PR for this one since there are too many failures. I was hoping other > people would be looking at the SLT bugs. (One other test fixture I proposed > was merged, though, and I hope it will prevent quite a few bugs too: > https://github.com/apache/calcite/pull/3433). > > I think testing could be improved a lot by getting Calcite to run more > tests from other open-source projects (e.g., Postgres, or MySql). For > example, it would be great to have a script to convert Postgres regress > tests > https://github.com/postgres/postgres/tree/master/src/test/regress/expectedto > to Quidem tests. These could be slow tests. > > (2) Documentation; Calcite has a steep learning curve. I think that better > documentation could help attract more developers and users. I would like to > propose adding a blog to the Calcite project web page. If every committer > writes two blog posts per year about some Calcite technical aspect, then > the project can quickly accumulate a lot of helpful information. The dev > mailing list does not seem to be well indexed by search engines; we could > even take some answers for questions on the mailing list and turn them into > short articles. I have written a blog post about several Calcite IRs, which > I plan to post in the following days on our company website (I will > actually send a request for review on this list when I do), but it would be > great to cross-post such documents on the Calcite project blog. > > (3) While poking around I have found examples of data structure that are > probably confusing. Some of them have been known to be confusing for a long > time: e,g: RexLiteral.typeName. > https://github.com/apache/calcite/blob/c83ac69111fd9e75af5e3615af29a72284667a4a/core/src/main/java/org/apache/calcite/rex/RexLiteral.java#L214 > A comment from 2006 indicates that this field should not exist. > > Another example is https://github.com/apache/calcite/pull/3411, where I > tried to fix up some problems with the type families. Such PRs are > difficult to accept, in the short term they may increase the code entropy > (like this one does, by creating deprecated functions and keeping the old > ones too), and it is very hard to make sure that they don't break anything > for users, and their value is questionable if people do not actually remove > all uses that should be deprecated. This requires a sustained effort to > cleanup; contributors could periodically self-assign a small cleanup task. > > Thank you, > Mihai > > -----Original Message----- > From: Stamatis Zampetakis > Sent: Sunday, October 22, 2023 9:00 AM > To: [email protected] > Subject: [DISCUSS] State of the project 2023 > > Hello community and happy birthday Calcite, > > Eight years ago (22 October 2015) Calcite graduated as a top-level Apache > project [1]. At that point, the community decided to have an annual “state > of the project” discussion, and we arrived at that time of the year. > > We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023 [2], > with probably one more coming before the end of the year. With roughly 300 > commits coming in, it is evident that the community did a great job in > fixing numerous bugs, landing new features, and bringing notable > improvements and optimizations to the project. It is worth highlighting the > collective effort that was done for improving the BigQuery dialect where > more than 60 new SQL functions were added. > > Regarding the sub-project Avatica, we had a release at the beginning of > the year (1.23.0) [3] featuring some new configurations for fetchSize and > SSL key stores along with various other improvements and bug fixes. Since > the last release, there have been roughly 10 new commits in the main branch > mostly comprising dependency upgrades and build fixes. As in previous years > the activity is rather low but there are still volunteers regularly > checking and contributing to the repo. > > In terms of community, I think this has been a great year. We see more and > more people participating in email discussions, Jira tickets and Github > PRs. Our list of committers has grown with Istvan Toth, Alex Plehanov, > Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ Banghart, Dan > Zou; and so has our PMC with Benchao Li joining the team. With nine new > committers and one new PMC member till now, it's probably the best year so > far (and it's not yet ended). Calcite grows and evolves because of (and > thanks to) its community, so I would like to thank everyone for being part > of this family and working together in a respectful and motivating > environment. > > It was nice to see our community members giving talks to conferences such > as ApacheCon East Asia, VLDB, and Community over Code presenting Calcite > and/or its application. As in previous years, we had one online Calcite > meetup, which was a great opportunity for the community to virtually meet > and share some interesting presentations, and hopefully we could add some > in-person events in the years to come. > > Over the past few years we always had problems with reviewing pull > requests with a lot of weight falling upon a few individuals. In the last > board report for (Q3 2023), the numbers were a bit more encouraging showing > more people involved in reviews. The problem is not yet solved but we are > moving in a promising direction. > > To conclude, I will repeat the questions from previous years: > 1) What else are we doing well in the project? > 2) What areas do we need to do better? > > Please take some time to share your thoughts! > > Note that this discussion is for everyone, not only for committers / PMC > members; even if you have never sent an email to the dev list before, now > is a good time to do so :) > > Finally, it has been a privilege to serve as Calcite's PMC Chair this > year. I have learnt a lot and I am very grateful for the opportunity that I > was given. Following our yearly rotation tradition, I will step down as > Chair by the end of the year, and a new one will have to be chosen. As we > discussed some time ago [4], if you have any suggestion and you would like > to put someone forward as a potential next Chair, please send an email to > [email protected]. The nomination period for the new chair will > remain open for the next two weeks (till 2023-11-05). The PMC will study > all proposals, discuss, and start a vote soon after 2023-11-05. The change > will be effective some time in December once the resolution is approved by > the board. > > Best, > Stamatis > > [1] https://calcite.apache.org/news/2015/10/22/calcite-graduates/ > [2] https://calcite.apache.org/news/ > [3] https://calcite.apache.org/avatica/news/ > [4] https://lists.apache.org/thread/gplfqs4snr1b6h62cngyvb65sz41z3fk > >
