Hi devs,
There is no doubt that Calcite boasts powerful capabilities, but the curve
of acquiring and learning calcite is quite steep. After studying Calcite
for a year, I have only gained a limited understanding of some of its
modules. Mihai is about to contribute several documents related to calcite,
which is truly exciting. Calcite is a treasure, and I hope there will be
more developer documentation next year to enable developers to quickly
grasp how to use calcite and how it works.
Secondly, Calcite supports a wide array of SQL syntax, but there are still
a few syntaxes that it does not cover. I hope that Calcite can be equipped
with more comprehensive SQL syntax parsing capabilities, which will be the
selling point that convinces more developers to opt for Calcite. I am
anticipating that Calcite will support more syntax next year, both standard
and non-standard, and I will continue to contribute in this direction.
Best,
Hongyu Guo
On Tue, Oct 24, 2023 at 3:16 AM <[email protected]> wrote:
Hello all,
Calcite is great! However, I think that there are several areas where the
project could do better:
- testing
- documentation
- legacy code cleanup
(1) Testing: Most of my work in Calcite so far has been about testing. I
have contributed the SqlLogicTest suite, which uncovers many bugs.
However,
I have not filed issues for any of these bugs, and I have not fixed any
of
these bugs either, since I have been very busy with other bugs I found
using other testing methods. I also have written a new test fixture used
for SqlOperatorTest tests, which finds lots of bugs, but I haven't sent a
PR for this one since there are too many failures. I was hoping other
people would be looking at the SLT bugs. (One other test fixture I
proposed
was merged, though, and I hope it will prevent quite a few bugs too:
https://github.com/apache/calcite/pull/3433).
I think testing could be improved a lot by getting Calcite to run more
tests from other open-source projects (e.g., Postgres, or MySql). For
example, it would be great to have a script to convert Postgres regress
tests
https://github.com/postgres/postgres/tree/master/src/test/regress/expectedto
to Quidem tests. These could be slow tests.
(2) Documentation; Calcite has a steep learning curve. I think that
better
documentation could help attract more developers and users. I would like
to
propose adding a blog to the Calcite project web page. If every committer
writes two blog posts per year about some Calcite technical aspect, then
the project can quickly accumulate a lot of helpful information. The dev
mailing list does not seem to be well indexed by search engines; we could
even take some answers for questions on the mailing list and turn them
into
short articles. I have written a blog post about several Calcite IRs,
which
I plan to post in the following days on our company website (I will
actually send a request for review on this list when I do), but it would
be
great to cross-post such documents on the Calcite project blog.
(3) While poking around I have found examples of data structure that are
probably confusing. Some of them have been known to be confusing for a
long
time: e,g: RexLiteral.typeName.
https://github.com/apache/calcite/blob/c83ac69111fd9e75af5e3615af29a72284667a4a/core/src/main/java/org/apache/calcite/rex/RexLiteral.java#L214
A comment from 2006 indicates that this field should not exist.
Another example is https://github.com/apache/calcite/pull/3411, where I
tried to fix up some problems with the type families. Such PRs are
difficult to accept, in the short term they may increase the code entropy
(like this one does, by creating deprecated functions and keeping the old
ones too), and it is very hard to make sure that they don't break
anything
for users, and their value is questionable if people do not actually
remove
all uses that should be deprecated. This requires a sustained effort to
cleanup; contributors could periodically self-assign a small cleanup
task.
Thank you,
Mihai
-----Original Message-----
From: Stamatis Zampetakis
Sent: Sunday, October 22, 2023 9:00 AM
To: [email protected]
Subject: [DISCUSS] State of the project 2023
Hello community and happy birthday Calcite,
Eight years ago (22 October 2015) Calcite graduated as a top-level Apache
project [1]. At that point, the community decided to have an annual
“state
of the project” discussion, and we arrived at that time of the year.
We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023 [2],
with probably one more coming before the end of the year. With roughly
300
commits coming in, it is evident that the community did a great job in
fixing numerous bugs, landing new features, and bringing notable
improvements and optimizations to the project. It is worth highlighting
the
collective effort that was done for improving the BigQuery dialect where
more than 60 new SQL functions were added.
Regarding the sub-project Avatica, we had a release at the beginning of
the year (1.23.0) [3] featuring some new configurations for fetchSize and
SSL key stores along with various other improvements and bug fixes. Since
the last release, there have been roughly 10 new commits in the main
branch
mostly comprising dependency upgrades and build fixes. As in previous
years
the activity is rather low but there are still volunteers regularly
checking and contributing to the repo.
In terms of community, I think this has been a great year. We see more
and
more people participating in email discussions, Jira tickets and Github
PRs. Our list of committers has grown with Istvan Toth, Alex Plehanov,
Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ Banghart, Dan
Zou; and so has our PMC with Benchao Li joining the team. With nine new
committers and one new PMC member till now, it's probably the best year
so
far (and it's not yet ended). Calcite grows and evolves because of (and
thanks to) its community, so I would like to thank everyone for being
part
of this family and working together in a respectful and motivating
environment.
It was nice to see our community members giving talks to conferences such
as ApacheCon East Asia, VLDB, and Community over Code presenting Calcite
and/or its application. As in previous years, we had one online Calcite
meetup, which was a great opportunity for the community to virtually meet
and share some interesting presentations, and hopefully we could add some
in-person events in the years to come.
Over the past few years we always had problems with reviewing pull
requests with a lot of weight falling upon a few individuals. In the last
board report for (Q3 2023), the numbers were a bit more encouraging
showing
more people involved in reviews. The problem is not yet solved but we are
moving in a promising direction.
To conclude, I will repeat the questions from previous years:
1) What else are we doing well in the project?
2) What areas do we need to do better?
Please take some time to share your thoughts!
Note that this discussion is for everyone, not only for committers / PMC
members; even if you have never sent an email to the dev list before, now
is a good time to do so :)
Finally, it has been a privilege to serve as Calcite's PMC Chair this
year. I have learnt a lot and I am very grateful for the opportunity
that I
was given. Following our yearly rotation tradition, I will step down as
Chair by the end of the year, and a new one will have to be chosen. As we
discussed some time ago [4], if you have any suggestion and you would
like
to put someone forward as a potential next Chair, please send an email to
[email protected]. The nomination period for the new chair will
remain open for the next two weeks (till 2023-11-05). The PMC will study
all proposals, discuss, and start a vote soon after 2023-11-05. The
change
will be effective some time in December once the resolution is approved
by
the board.
Best,
Stamatis
[1] https://calcite.apache.org/news/2015/10/22/calcite-graduates/
[2] https://calcite.apache.org/news/
[3] https://calcite.apache.org/avatica/news/
[4] https://lists.apache.org/thread/gplfqs4snr1b6h62cngyvb65sz41z3fk
