Re: [VOTE] Release Apache Calcite 1.37.0 (release candidate 1)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 7.6.1

 * Checked signatures and checksums OK
 * Checked diff between repo and artifacts OK
 * Went over release note OK (Left comments in the PR)
 * Checked README, NOTICE, LICENSE OK
 * All source files have ASF headers OK (raised [1] for two files
where header is needed)
 * No unexpected binary files OK (find . -type f -exec file {} \; |
grep -v text)
 * Checked LICENSE, NOTICE, signature, and checksum for
calcite-core-1.37.0.jar in nexus [2] OK
 * Built from git tag and run tests (./gradlew build) OK
 * Built from source artifacts and run tests (gradle build) KO (Logged
CALCITE-6385 [3])

-1 (binding)

The negative vote is mainly due to CALCITE-6385. Although the test
failure is rather innocent, it makes the build fail so users building
from source may have a hard time figuring out what happens.

Best,
Stamatis

[1] https://issues.apache.org/jira/browse/CALCITE-6384
[2] 
https://repository.apache.org/content/repositories/orgapachecalcite-1225/org/apache/calcite/calcite-core/1.37.0/
[3] https://issues.apache.org/jira/browse/CALCITE-6385

On Wed, Apr 24, 2024 at 11:25 AM Benchao Li  wrote:
>
> +1 (binding)
>
> - Checked signature and checksum (OK)
> - Checked LICENSE and Copyright year (OK)
> - Checked there are no unexpected binary files (OK)
> - Build and test from source (testContributorsFileIsSorted and
> testMailmapFile in LintTest failed due to not in git repo)
> - Reviewed release note, left some comments which can be resolved
> after the release
>
> Ruben Q L  于2024年4月23日周二 16:57写道：
> >
> > Thanks Sergey for preparing this release.
> > Just a minor clarification: the email subject says "release candidate 1",
> > but we are actually voting rc0.
> >
> > - Release notes: ok (with some minor comments that I have left in the PR)
> > - Checksum: ok
> > - Signature: ok
> > - Diff source release and git repository: ok
> > - Build + tests: ok
> > - Calcite-based application test suite: ok
> >
> > +1 (binding)
> >
> > Best,
> > Ruben
> >
> >
> >
> > On Tue, Apr 23, 2024 at 12:55 AM Francis Chuang 
> > wrote:
> >
> > > Thanks for being RM for this release, Sergey!
> > >
> > > My vote is: +1 (binding)
> > >
> > > - Verified GPG signature - OK
> > > - Verified SHA512 - OK
> > > - Diffed source release and git repository - OK
> > > - Checked release notes on tag
> > > (
> > > https://github.com/apache/calcite/blob/calcite-1.37.0-rc0/site/_docs/history.md)
> > >
> > > - OK
> > > - Checked year and versions in NOTICE, README and HOWTO - OK
> > > - Ran tests (gradle check) - OK
> > > - Spot checked Nexus artifacts - OK
> > >
> > > Minor issue with the release notes that should be fixed after the
> > > release: Some of the contributors are listed as their GitHub usernames,
> > > I think it would be better to put in their real names if they are known.
> > > These can usually be retrieved from their GitHub profiles.
> > >
> > > Environment:
> > > Eclipse-temurin:8 docker container in WSL2 (Ubuntu 22.04.4) on Windows
> > > 11 23h2
> > >
> > > $ docker version
> > > Client:
> > >   Cloud integration: v1.0.35+desktop.13
> > >   Version:   26.0.0
> > >   API version:   1.45
> > >   Go version:go1.21.8
> > >   Git commit:2ae903e
> > >   Built: Wed Mar 20 15:16:45 2024
> > >   OS/Arch:   linux/amd64
> > >   Context:   default
> > >
> > > Server: Docker Desktop
> > >   Engine:
> > >Version:  26.0.0
> > >API version:  1.45 (minimum version 1.24)
> > >Go version:   go1.21.8
> > >Git commit:   8b79278
> > >Built:Wed Mar 20 15:18:01 2024
> > >OS/Arch:  linux/amd64
> > >Experimental: false
> > >   containerd:
> > >Version:  1.6.28
> > >GitCommit:ae07eda36dd25f8a1b98dfbf587313b99c0190bb
> > >   runc:
> > >Version:  1.1.12
> > >GitCommit:v1.1.12-0-g51d5e94
> > >   docker-init:
> > >Version:  0.19.0
> > >GitCommit:de40ad0
> > >
> > > $ gradle -v
> > >
> > > 
> > > Gradle 7.6.1
> > > 
> > >
> > > Build time:   2023-02-24 13:54:42 UTC
> > > Revision: 3905fe8ac072bbd925c70ddbf4463341f4b4
> > >
> > > Kotlin:   1.7.10
> > > Groovy:   3.0.13
> > > Ant:  Apache Ant(TM) version 1.10.11 compiled on July 10 2021
> > > JVM:  1.8.0_402 (Temurin 25.402-b06)
> > > OS:   Linux 5.15.146.1-microsoft-standard-WSL2 amd64
> > >
> > > $ java -version
> > > openjdk version "1.8.0_402"
> > > OpenJDK Runtime Environment (Temurin)(build 1.8.0_402-b06)
> > > OpenJDK 64-Bit Server VM (Temurin)(build 25.402-b06, mixed mode)
> > >
> > > Francis
> > >
> > > On 23/04/2024 9:18 am, Sergey Nuyanzin wrote:
> > > > Hi all,
> > > >
> > > > I have created a build for Apache Calcite 1.37.0, release
> > > > candidate 0.
> > > >
> > > > Thanks to everyone who 

[jira] [Created] (CALCITE-6385) LintTest fails when run in source distribution

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6385:


 Summary: LintTest fails when run in source distribution
 Key: CALCITE-6385
 URL: https://issues.apache.org/jira/browse/CALCITE-6385
 Project: Calcite
  Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


Steps to reproduce:
# Download 
[https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.37.0-rc0/apache-calcite-1.37.0-src.tar.gz]
# tar -xvf apache-calcite-1.37.0-src.tar.gz
# cd apache-calcite-1.37.0-src
#  /opt/gradle/gradle-7.6.1/bin/gradle cleanTest :core:test --tests LintTest

{noformat}
FAILURE   0.1sec, org.apache.calcite.test.LintTest > 
testContributorsFileIsSorted()
java.lang.RuntimeException: command [git, ls-files, *.bat, *.cmd, *.csv, 
*.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, *.md, 
*.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: failed with exception
at org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:185)
at org.apache.calcite.util.TestUnsafe.getTextFiles(TestUnsafe.java:158)
at 
org.apache.calcite.test.LintTest.testContributorsFileIsSorted(LintTest.java:400)
Caused by: java.lang.RuntimeException: command [git, ls-files, *.bat, 
*.cmd, *.csv, *.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, 
*.md, *.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: exited with 
status 128
at 
org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:180)
... 2 more

FAILURE   0.0sec, org.apache.calcite.test.LintTest > testMailmapFile()
java.lang.RuntimeException: command [git, ls-files, *.bat, *.cmd, *.csv, 
*.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, *.md, 
*.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: failed with exception
at org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:185)
at org.apache.calcite.util.TestUnsafe.getTextFiles(TestUnsafe.java:158)
at org.apache.calcite.test.LintTest.testMailmapFile(LintTest.java:419)
Caused by: java.lang.RuntimeException: command [git, ls-files, *.bat, 
*.cmd, *.csv, *.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, 
*.md, *.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: exited with 
status 128
at 
org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:180)
... 2 more

FAILURE   0.1sec,6 completed,   2 failed,   2 skipped, 
org.apache.calcite.test.LintTest
{noformat}
 
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-6384) Missing ASF header from buildcache.yml, gradle-wrapper-validation.yml

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6384:


 Summary: Missing ASF header from buildcache.yml, 
gradle-wrapper-validation.yml
 Key: CALCITE-6384
 URL: https://issues.apache.org/jira/browse/CALCITE-6384
 Project: Calcite
  Issue Type: Task
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


The header is required as per [ASF 
policy|https://www.apache.org/legal/src-headers.html].
 * 
[https://github.com/apache/calcite/blob/153494207c24318d4c1d2c908df4a15c4aa4/.github/workflows/buildcache.yml]
 * 
[https://github.com/apache/calcite/blob/153494207c24318d4c1d2c908df4a15c4aa4/.github/workflows/gradle-wrapper-validation.yml]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Supporting ASOF JOIN

[Stamatis Zampetakis]

Regarding the additional joins, I think Calcite's Correlate [1] is the
equivalent of the dependent/binding join that appears in the database
literature. Currently, Correlate does not extend Join but we could
possibly reconsider this decision if necessary.

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/0551b8903391c1706422a2c1b8b648a6941f39a2/core/src/main/java/org/apache/calcite/rel/core/Correlate.java

On Tue, Apr 16, 2024 at 1:22 AM Julian Hyde  wrote:
>
> I would regard this as two separate but related things: a new SQL syntax for 
> joins, and a new relational operator. It is definitely worth keeping them 
> separate; the operator will not map 1-1 to the syntax, may require its input 
> to input to be sorted, and of course we would want queries to be able to use 
> the operator even if they didn’t use the syntax.
>
> The relational operator can have physical implementations in various calling 
> conventions. Or even flags extending existing algorithms (e.g. add a 
> ‘keepAtMostOneOnLeft’ flag to EnumerableMergeJoin).
>
> Regarding whether to represent the operator as a subclass of Join or just a 
> subclass of BiRel. I recommend making it a subclass of join, but we have to 
> take care that rewrite rules and metadata rules designed to apply to regular 
> joins do not accidentally apply to these joins. We’ve already done that with 
> semi-join, so it shouldn’t be too hard to follow those breadcrumbs.
>
> I recently read “The Complete Story of Joins (in HyPer)”, which contains some 
> other interesting and useful join variants: dependent join and mark join. We 
> should consider adding these as relational operators, in the same way that we 
> add asof-join.
>
> Julian
>
> [1] 
> http://btw2017.informatik.uni-stuttgart.de/slidesandpapers/F1-10-37/paper_web.pdf
>
> > On Apr 15, 2024, at 2:19 PM, Mihai Budiu  wrote:
> >
> > Hello,
> >
> > Seems that this new kind of JOIN named AS OF is very useful for processing 
> > time-series data. Here is some example documentation from Snowflake: 
> > https://docs.snowflake.com/en/sql-reference/constructs/asof-join
> >
> > The semantics is similar to a traditional join, but the result always 
> > contains at most one record from the left side, with the last matching 
> > record on the right side (where "time" is any value that can be compared 
> > for inequality). This can be expressed in SQL, but it looks very 
> > cumbersome, using a JOIN, a GROUP BY, and then an aggregation to keep the 
> > last value.
> >
> > I haven't seen anything like that in Calcite, although Calcite does seem to 
> > have support for all sorts of temporal and stream notions.
> >
> > If one were to implement it, what would be the right way to do it? A 
> > subclass of Join? A new type of BiRel RelNode?
> >
> > Thank you,
> > Mihai
>

Re: [DISCUSS] Towards Calcite 1.37.0

[Stamatis Zampetakis]

> > >>>> IMO,
> > >>>>>> and we
> > >>>>>>>>> are trying to get it done shortly with Hongyu.
> > >>>>>>>>>
> > >>>>>>>>> Best regards,
> > >>>>>>>>> Alessandro
> > >>>>>>>>>
> > >>>>>>>>> On Wed, 6 Mar 2024 at 16:49, Sergey Nuyanzin <
> > snuyan...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Updates about 1.37.0 releasing
> > >>>>>>>>>>
> > >>>>>>>>>> According to our Jira dashboard [1] and Github [2], there
> > >>>>>>>>>> are 17 pending issues that could / should be part of the
> > release.
> > >>>>>>>>>> I'd propose to make a collective effort to try to clean up our
> > >>>>> 1.37.0
> > >>>>>>>>>> backlog and merge the PRs which are in a good state. I'd propose
> > >>>> to
> > >>>>>>>>>> aim for about 2 weeks to release 1.37.0, this will give us about
> > >>>>>>>>>> some time to clean up pending PRs for the next version. What do
> > >>>> you
> > >>>>>>>>> think?
> > >>>>>>>>>>
> > >>>>>>>>>> And contributors, if you have any work that is in good shape and
> > >>>>> want
> > >>>>>>>>>> to be included in 1.37.0, please mark the fixVersion to 1.37.0,
> > >>>> this
> > >>>>>>>>>> will inform the release manager, and we'll try our best to get
> > it
> > >>>>> in.
> > >>>>>>>>>>
> > >>>>>>>>>> A quick update on the current status for issues targeted 1.37.0
> > >>>>>>>>>> We need more reviewers for #2
> > >>>>>>>>>>
> > >>>>>>>>>> #1. Issues already have reviewers, and seems promising to be
> > >>>> merged
> > >>>>>>>>> shortly
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6138
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6015
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6283
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-5607
> > >>>>>>>>>>
> > >>>>>>>>>> #2. Issues has been reviewed, and needs another reviewer
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-3522
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6071
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6210
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6259
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-2067
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6226
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6193
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> It could happen that I miss something, please let me know
> > >>>>>>>>>> if there is something with fixVersion 1.37.0 ready for review
> > and
> > >>>>> not
> > >>>>>>>>>> mentioned here
> > >>>>>>>>>>
> > >>>>>>>>>> Also I created a JIRA issue for releasing 1.37.0
> > >>>>>>>>>> https://issues.apache.org/jira/browse/CALCITE-6302
> > >>>>>>>>>>
> > >>>>>>>>>> [1]
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12333950
> > >>>>>>>>>> [2] https://github.com/apache/calcite/pulls
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Feb 23, 2024 at 9:24 PM Julian Hyde <
> > >>>> jhyde.apa...@gmail.com
> > >>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> +1 for a release.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Our users rely on a regular cadence of releases. If someone
> > >>>>> submits a
> > >>>>>>>>>>> patch and doesn’t see it in a release until 6 months later (or
> > it
> > >>>>>> sits
> > >>>>>>>>>>> un-reviewed) they will be less inclined to submit a patch.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yeah, I know Open Source Doesn’t Require Providing Builds [1]
> > but
> > >>>>> it
> > >>>>>>>>>> makes
> > >>>>>>>>>>> everyone’s life easier.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Julian
> > >>>>>>>>>>>
> > >>>>>>>>>>> [1] https://news.ycombinator.com/item?id=39094387
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Feb 21, 2024, at 1:18 AM, Stamatis Zampetakis <
> > >>>>> zabe...@gmail.com
> > >>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> With so many commits it's definitely a good time to cut a new
> > >>>>>>>>> release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I can be the RM for 1.39.0 or 1.40.0 depending on the timing.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Stamatis
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Feb 21, 2024 at 12:10 AM Sergey Nuyanzin <
> > >>>>>>>>> snuyan...@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for starting the discussion
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> @Sergey, are you still available for being the release
> > manager
> > >>>>> for
> > >>>>>>>>>>> 1.37.0?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I think so, in theory I could start with some steps in
> > one/two
> > >>>>>> weeks
> > >>>>>>>>>> if
> > >>>>>>>>>>>>> there is no objections
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Mon, Feb 19, 2024 at 12:29 PM Benchao Li <
> > >>>>> libenc...@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> It's been a bit more than 3 months since our last release
> > >>>>>>>>>> (2023-11-10)
> > >>>>>>>>>>>>>> [1] and there are currently 100+ new commits in main branch.
> > >>>> Per
> > >>>>>>>>> our
> > >>>>>>>>>>>>>> tradition of producing one release every 2-3 months, I think
> > >>>>> it's
> > >>>>>>>>>> time
> > >>>>>>>>>>>>>> to consider for next release now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> According to [2], the following release managers would be:
> > >>>>>>>>>>>>>> - 1.37.0 Sergey Nuyanzin
> > >>>>>>>>>>>>>> - 1.38.0 Julian Hyde
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> @Sergey, are you still available for being the release
> > manager
> > >>>>> for
> > >>>>>>>>>>> 1.37.0?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Besides, we need more volunteers for the next releases
> > >>>> (version
> > >>>>>> =
> > >>>>>>>>>>>>>> 1.39.0), anyone who is interested in doing it can reply in
> > >>>> this
> > >>>>>>>>>>>>>> thread.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [1] https://calcite.apache.org/docs/history.html#v1-36-0
> > >>>>>>>>>>>>>> [4]
> > >>>>>>>>> https://lists.apache.org/thread/tm3t42qvpq3db24xtd2g468ofv83l6hk
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Benchao Li
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>> Sergey
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> Best regards,
> > >>>>>>>>>> Sergey
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Sergey
> > >>>>
> > >>>
> > >>>
> > >
> > >
> > >
> >
>
>
> --
> Best regards,
> Sergey

Re: Calcite Setup Issues

[Stamatis Zampetakis]

Hey Anurag,

The bucket error seems somewhat related to the Gradle caches [1, 2].
Try to disable the cache [3] to see if the error goes away.

The fact that the build gets stuck while running tests may require
further investigation and I guess it is strongly related to your
environment since others are not hitting this. Note that JDK8 has also
some known concurrency bugs that we have been hitting in the past [4].
If the problem is reproducible please create a JIRA ticket with all
necessary information that could help in reproducing the problem
including details about your environment, JDK, exact commands that you
are using to replicate the problem, and ideally stack traces from the
stucked JVM.

Best,
Stamatis

[1] https://issues.apache.org/jira/browse/CALCITE-4140
[2] https://issues.apache.org/jira/browse/CALCITE-4168
[3] https://lists.apache.org/thread/ym59yjbtlmht6p7cxc08j2mz7hdzdgp6
[4] https://issues.apache.org/jira/browse/CALCITE-5433

On Sun, Apr 7, 2024 at 9:03 AM Anurag Naik  wrote:
>
> Nope. I ran the build on jdk 8 but no luck. It still throws the could not
> find bucket error. And when it skips that part of the build, at the end of
> it the build gets stuck on org.apache.calcite.test.SqlLineTest. 8904 tests
> completed, 101 skipped.
>
> I don't see the issue here. I have everything in my environment set up.
> Ensured no version mismatch in jdk.
> I'm following the ./gradlew build method from the docs(If that's any help)
>
>
> On Sun, Apr 7, 2024 at 3:23 AM Tanner Clary 
> wrote:
>
> > I’ve had this issue before when I was using the wrong Java version. Setting
> > to 1.8 normally fixes for me. Could be worth trying.
> >
> > Tanner
> >
> > On Sat, Apr 6, 2024 at 2:53 AM Anurag Naik 
> > wrote:
> >
> > > Thank you, Mr. Francis. I have now subscribed to the mailing list.
> > >
> > > So the issue I have been encountering with calcite starts with the
> > build. I
> > > tried the stable releases, the git repository, and even using IntelliJ on
> > > Windows but none of these approaches gave me a successful build. My
> > purpose
> > > of using Calcite is to connect it to a PostgreSQL database running in the
> > > same system and use Calcite's query planner.
> > >
> > > On IntelliJ it gives the autostyle test fail errors. When I skip those
> > > tests the build fails with the reason :
> > > " Could not load entry 8c38021b652ec64217f946170b7adfb2 from remote build
> > > cache: Bucket 'calcite-gradle-cache' not found ".
> > >
> > > I tried looking for solutions online but none of the approaches helped. I
> > > could use some help with instructions to build calcite and also to
> > connect
> > > it to postgreSQL with the jdbc drivers. Thank You
> > >
> > > On Thu, Apr 4, 2024, 11:29 AM Anurag Naik 
> > > wrote:
> > >
> > > > I write this to follow up on a previously sent email regarding the
> > issues
> > > > I am facing while trying to build calcite. I need it for a research
> > > project
> > > > that I am going to be working on. I would like it if you could guide me
> > > to
> > > > the appropriate communication channels to allow me to interact with the
> > > > committers and devs and get the build issues fixed. Also since I am
> > going
> > > > to be using calcite for the duration of the research project, I might
> > > need
> > > > to propose changes to the code as well. Being in contact with the devs
> > > > would help that purpose.
> > > >
> > >
> >

Re: Draft: board report for 2024 Q1

[Stamatis Zampetakis]

+1, nothing to add!

On Wed, Apr 3, 2024 at 11:11 PM Francis Chuang  wrote:
>
> +1, Great work, Benchao!
>
> On 4/04/2024 5:39 am, Ruben Q L wrote:
> > +1
> >
> > Thanks Benchao for preparing the report!
> >
> >
> > On Wed, Apr 3, 2024 at 7:34 PM Michael Mior  wrote:
> >
> >> +1 Thanks Benchao!
> >>
> >> --
> >> Michael Mior
> >> mm...@apache.org
> >>
> >>
> >> On Wed, Apr 3, 2024 at 6:11 AM Benchao Li  wrote:
> >>
> >>> Hello,
> >>>
> >>> Below you can find a draft of this quarter's board report. I plan to
> >>> submit the final version next Tuesday (Apr 9, 2024).
> >>>
> >>> Please let me know if you have any additions or corrections.
> >>>
> >>> --
> >>>
> >>> Best,
> >>> Benchao Li
> >>>
> >>>
> >>> ## Description:
> >>> Apache Calcite is a highly customizable framework for parsing and
> >> planning
> >>> queries on data in a wide variety of formats. It allows database-like
> >>> access,
> >>> and in particular a SQL interface and advanced query optimization, for
> >> data
> >>> not residing in a traditional database.
> >>>
> >>> Avatica is a sub-project within Calcite and provides a framework for
> >>> building
> >>> local and remote JDBC and ODBC database drivers. Avatica has an
> >> independent
> >>> release schedule and its own repository.
> >>>
> >>> ## Issues:
> >>> There are no issues requiring board attention.
> >>>
> >>> ## Membership Data:
> >>> Apache Calcite was founded 2015-10-22 (8 years ago)
> >>> There are currently 74 committers and 28 PMC members in this project.
> >>> The Committer-to-PMC ratio is roughly 5:2.
> >>>
> >>> Community changes, past quarter:
> >>> - Sergey Nuyanzin was added to the PMC on 2024-03-05
> >>> - No new committers. Last addition was Hongyu Guo on 2023-11-03.
> >>>
> >>> It's worth mentioning that a few people in Calcite PMC have been
> >>> invited as ASF members this year:
> >>> - Benchao Li
> >>> - Francis Chuang
> >>> - Ruben Quesada Lopez
> >>> - Sergey Nuyanzin
> >>>
> >>> ## Project Activity:
> >>> There is no release rolled out in Q1 yet, but we have planned to
> >>> release avatica 1.25.0 and calcite 1.37.0 lately.  For now, avatica
> >>> 1.25.0 RC0 is in voting stage, and calcite 1.37.0 will be kicked off
> >>> just after avatica 1.25.0 is released since we have a few co-related
> >>> issues for these two versions.
> >>>
> >>> ## Community Health:
> >>> The community maintains a super healthy status, due to the new
> >>> invitation of PMC member, previously it's healthy.
> >>>
> >>> Most of the statistics are steady compared to last quarter, except a
> >>> slight decrease for the last three weeks. I assume the reason would be
> >>> that we are in the process of calcite 1.37.0 release.It's a little bit
> >>> longer than usual, the reason is that we have a few co-related issues
> >>> for calcite and avatica, and we need to release avatica before
> >>> calcite.
> >>>
> >>> The number of non-committer (contributor) commits per month:
> >>> +--+---+-+
> >>> | year | month | contributor_commits |
> >>> +--+---+-+
> >>> | 2024 | 1 |  20 |
> >>> | 2024 | 2 |  20 |
> >>> | 2024 | 3 |  14 |
> >>> +--+---+-+
> >>>
> >>> The number of active reviewers per month:
> >>> +--+---+--+
> >>> | year | month | active_reviewers |
> >>> +--+---+--+
> >>> | 2024 | 1 |6 |
> >>> | 2024 | 2 |7 |
> >>> | 2024 | 3 |6 |
> >>> +--+---+--+
> >>>
> >>> Top reviewers in the last 3 months:
> >>> +---+-+
> >>> | committer | reviews |
> >>> +---+-+
> >>> | hongyu guo|  15 |
> >>> | Mihai Budiu   |  13 |
> >>> | Tanner Clary  |  10 |
> >>> +---+-+
> >>>
> >>
> >

Re: [VOTE] Release Apache Calcite Avatica 1.25.0 (release candidate 0)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 8.1.1

 * Checked signatures and checksums OK
 * Checked diff between repo and artifacts OK
 * Went over release note OK (It mentions two Gradle versions 8.4 and 8.5)
 * Checked README, NOTICE, LICENSE OK (minor fixes in LICENSE from
Calcite should be ported to Avatica)
 * All source files have ASF headers OK (grep -RiL "Licensed to the
Apache Software Foundation")
 * No unexpected binary files OK (find . -type f -exec file {} \; |
grep -v text)
 * Checked LICENSE, NOTICE, signature, and checksum for
avatica-core-1.25.0.jar in nexus OK
 * Built from git tag and run tests (./gradlew build) OK
 * Built from source artifacts and run tests (gradle build) OK

+1 (binding)

Best,
Stamatis


On Wed, Apr 3, 2024 at 11:36 PM Francis Chuang  wrote:
>
> Opps, good catch! Please ignore the links to the
> calcite-avatica-site-preview repo. Those are generated by the
> asf-release plugin, but we don't use any preview repos for Calcite.
>
> On 3/04/2024 9:41 pm, Benchao Li wrote:
> > +1 (binding)
> >
> > - checked signature and checksum [OK]
> > - checked copyright year in notice [OK]
> > - downloaded src, make sure no unexpected files [OK]
> > - compiled and tested from source [OK]
> > - reviewed release note [OK]
> > - checked a few files in Nexus for signature and checksum [OK]
> > - downloaded javadoc jar, make sure it's in English [OK]
> >
> > One minor comment is that those gitbox urls seem not valid.
> >
> > Francis Chuang  于2024年4月2日周二 11:40写道：
> >>
> >> Hi all,
> >>
> >> I have created a build for Apache Calcite Avatica 1.25.0, release
> >> candidate 0.
> >>
> >> Thanks to everyone who has contributed to this release.
> >>
> >> You can read the release notes here:
> >> https://github.com/apache/calcite-avatica/blob/avatica-1.25.0-rc0/site/_docs/history.md
> >>
> >> The commit to be voted upon:
> >> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=62b0fdd3f4e0173d8492eb0e203056f0938dd6f5
> >>
> >> Its hash is 62b0fdd3f4e0173d8492eb0e203056f0938dd6f5
> >>
> >> Tag:
> >> https://github.com/apache/calcite-avatica/tree/avatica-1.25.0-rc0
> >>
> >> The artifacts to be voted on are located here:
> >> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.25.0-rc0
> >>
> >> RAT report:
> >> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/rat/rat-report.txt
> >>
> >> Site preview is here:
> >> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/
> >>
> >> JavaDoc API preview is here:
> >> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/api
> >>
> >> The hashes of the artifacts are as follows:
> >> 0daa1da33ca0712cb3a4918260081e707c336e06eaacee87962a8ff914f377e8ad21d5fd24490612dbca7d56b83e9d062c39b5cd51701564cf8cba7840ed4806
> >> *apache-calcite-avatica-1.25.0-src.tar.gz
> >>
> >> A staged Maven repository is available for review at:
> >> https://repository.apache.org/content/repositories/orgapachecalcite-1223/org/apache/calcite/
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/francischuang.asc
> >> https://www.apache.org/dist/calcite/KEYS
> >>
> >> To create the jars and test Apache Calcite Avatica: "gradle build
> >> -Prelease -PskipSign".
> >>
> >> If you do not have a Java/Gradle environment available, you can run the
> >> tests using docker. To do so, install docker and docker-compose, then
> >> run "docker compose run test" from the root of the directory.
> >>
> >> Please vote on releasing this package as Apache Calcite Avatica 1.25.0.
> >>
> >> The vote is open for the next 72 hours and passes if a majority of at
> >> least three +1 PMC votes are cast.
> >>
> >> [ ] +1 Release this package as Apache Calcite Avatica 1.25.0
> >> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> >> [ ] -1 Do not release this package because...
> >>
> >>
> >> Here is my vote:
> >>
> >> +1 (binding)
> >>
> >> Francis
> >
> >
> >

Re: [ANNOUNCE] Sergey Nuyanzin joins Calcite PMC

[Stamatis Zampetakis]

Congratulations Sergey, very well deserved!

On Wed, Mar 6, 2024 at 3:33 AM Hongyu Guo  wrote:
>
> Congrats Sergey!
>
> Hongyu Guo
>
> On Wed, Mar 6, 2024 at 10:01 AM Jiajun Xie 
> wrote:
>
> > Congratulations, Sergey!
> >
> > On Wed, 6 Mar 2024 at 09:43, Cancai Cai  wrote:
> >
> > > Congratulations, Sergey!
> > >
> >

Re: [DISCUSS] Ensuring that Calcite is consistent with other SQL systems

[Stamatis Zampetakis]

The Quidem approach definitely makes sense.

Alternatively, since we are focusing on the behavior of functions with
certain inputs it may be simpler to go with a Java parameterized test
backed by a CSV/JSON file ([1] outlines the general idea).
As others mentioned, using the org.testcontainers framework can
significantly alleviate the burden of setting up and managing
different DBMS.

Best,
Stamatis

[1] https://github.com/apache/calcite/pull/3704


On Mon, Feb 26, 2024 at 4:01 AM Yiwen Wu  wrote:
>
> I think it is difficult and impossible to make calcite function results
> completely consistent with other SQL systems. In practice, when
> calculations are pushed down to different data source adapters for
> execution, there is no guarantee that the final results will be completely
> consistent in calcite.
>
> I think, for the built-in functions defined in `SqlStdOperatorTable`, we
> can choose to follow one of the definitive SQL systems, such as Oracle. For
> the engine-related extension functions in `SqlLibraryOperators`, we can
> follow the behavior related to the specific engine. If an already defined
> function behaves inconsistently with the new engine, we can add a new
> Function to the definition, such as the `CONCAT` function.
>
> At the same time, I agree it is a great suggestion to add execution tests
> of different engines in Quidem, which is very effective in verifying the
> engine execution behavior.
>
> Cancai Cai  于2024年2月26日周一 09:59写道：
>
> > Thank you very much to the calcite community for raising these questions.
> > This is what I have been doubting. I am very sorry that this doubt has been
> > discussed for so long.
> >
> > Maybe we also need to consider another issue, that is, the database version
> > issue. Versions like mysql and postgres are very stable, but components
> > like spark still seem to have function bugs. exists, then how should we
> > consider version issues?
> >
> > I don't know what I can do. Recently I am sorting out some documents about
> > the use of some functions of mysql and postgres in calcite. I don't know if
> > this is helpful.
> >
> > Best wishes,
> > Cancai Cai
> >

Re: [DISCUSS] Towards Calcite 1.37.0

[Stamatis Zampetakis]

With so many commits it's definitely a good time to cut a new release.

I can be the RM for 1.39.0 or 1.40.0 depending on the timing.

Best,
Stamatis

On Wed, Feb 21, 2024 at 12:10 AM Sergey Nuyanzin  wrote:
>
> Thanks for starting the discussion
>
> > @Sergey, are you still available for being the release manager for 1.37.0?
>
> I think so, in theory I could start with some steps in one/two weeks if
> there is no objections
>
>
>
> On Mon, Feb 19, 2024 at 12:29 PM Benchao Li  wrote:
>
> > It's been a bit more than 3 months since our last release (2023-11-10)
> > [1] and there are currently 100+ new commits in main branch. Per our
> > tradition of producing one release every 2-3 months, I think it's time
> > to consider for next release now.
> >
> > According to [2], the following release managers would be:
> > - 1.37.0 Sergey Nuyanzin
> > - 1.38.0 Julian Hyde
> >
> > @Sergey, are you still available for being the release manager for 1.37.0?
> >
> > Besides, we need more volunteers for the next releases (version >=
> > 1.39.0), anyone who is interested in doing it can reply in this
> > thread.
> >
> > [1] https://calcite.apache.org/docs/history.html#v1-36-0
> > [4] https://lists.apache.org/thread/tm3t42qvpq3db24xtd2g468ofv83l6hk
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>
>
> --
> Best regards,
> Sergey

Re: "FunctionJoin" advice

[Stamatis Zampetakis]

The TableFunctionScan indeed seems to fit exactly the "FunctionJoin"
use-case. If the TableFunctionScan alone cannot fulfil your use-case
then I am pretty sure that a combination of Correlate +
TableFunctionScan should do the trick. The examples in the Calcite
codebase are a bit scarce but you can get some inspiration by looking
at the test cases of SqlToRelConverter [1].

Creating a RexNode to call a java method is a separate topic. If you
want to call arbitrary java methods the probably you have to create a
custom SqlOperator/SqlFunction (maybe extending SqlUserDefinedFunction
[2] and/or implementing the ImplementableFunction interface [3]) where
one of the arguments is a character literal specifying the canonical
name of the method. Instead of allowing calls to arbitrary java
methods via reflection it may be wiser to implement dedicated
SqlOperators for each separate case you need. Giving the user the
option to call arbitrary methods via reflection opens up security
holes for the application.

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/351ddeb47b8dfb5c196c563920290a79575e9864/core/src/test/resources/org/apache/calcite/test/SqlToRelConverterTest.xml
[2] 
https://github.com/apache/calcite/blob/351ddeb47b8dfb5c196c563920290a79575e9864/core/src/main/java/org/apache/calcite/sql/validate/SqlUserDefinedFunction.java
[3] 
https://github.com/apache/calcite/blob/351ddeb47b8dfb5c196c563920290a79575e9864/core/src/main/java/org/apache/calcite/schema/ImplementableFunction.java

On Wed, Jan 31, 2024 at 9:09 AM Julian Hyde  wrote:
>
> ‘Function join’ sounds similar to ‘CROSS APPLY’ - for each row on the left, 
> call a function to generate a list of rows on the right. Is that how you see 
> it?
>
> If so, there are some beautiful parallels between CROSS APPLY and lateral 
> join; for example, they can both be de-correlated. See [1] and later work in 
> Froid. When I have a spare year, we’ll put these optimizations into Calcite… 
> :)
>
> Julian
>
> [1] 
> https://www.cse.iitb.ac.in/infolab/Data/Courses/CS632/2014/2009/Papers/subquery-proc-elhemali-sigmod07.pdf
>
> > On Jan 30, 2024, at 11:45 PM, Thomas Rynne 
> >  wrote:
> >
> > I have created a RelNode "FunctionJoin" that takes in input relation and 
> > the name of a java method to call to create new rows from an existing row 
> > (Array[Any] => Array[Array[Any]])
> >
> > It's like a join but the rows of right relation are created on demand using 
> > the rows of the left relation.
> >
> > I have something working using  a converter rule that creates an 
> > EnumerableFunctionJoin that calls the java method for each of the input 
> > rows.
> > I also have rules to move filters & projects past the FunctionJoin to the 
> > FunctionJoin input (I think operating on LogicalCalc might simplify things 
> > as I can't move filters and projects at the same time).
> >
> > It has been a good learning experience but before I go too far I thought I 
> > would ask whether this already exists or whether a different approach would 
> > make sense.
> >
> > For example, is TableFunctionScan the same thing? It seems similar but I 
> > can't find any example uses and I don't know how I would create a RexNode 
> > that calls a java method.
> >
> > thanks for any pointers
> > Thomas
> >
>

Re: EnumerableProject question

[Stamatis Zampetakis]

Hey Eric,

EnumerabeProject and EnumerableFilter are not meant to appear in the
plan because they cannot be implemented; as you observed the implement
method throws an exception. For this reason it does not make sense to
use ENUMERABLE_PROJECT_RULE and ENUMERABLE_FILTER_RULE and this is why
the example does not contain them. I assume that you are getting the
UnsupportedOperationException because you added those rules to the
example.

If you instruct the planner to come up with an ENUMERABLE plan and you
include the aforementioned rules then the resulting plan may include
EnumerableProject and EnumerableFilter and that is a valid plan from a
planning perspective because it respects the requested convention. The
planner does not use the EnumerableRel#implement method and the method
becomes relevant only during execution.

Both projection and filter in the enumerable convention are
implemented via the Calc operator and that's why the PROJECT_TO_CALC
rule exists.

Best,
Stamatis

On Mon, Jan 29, 2024 at 4:51 PM Eric Berryman  wrote:
>
> correction:
> When compared to the jdbc implementation, which works for me using the
> EnumerableConvention.
> I see the difference is that the EnumerableProject gets mapped to a
> EnumerableCalc.
> Mapped to LogicalProject.NONE
> https://github.com/apache/calcite/blob/75511b82ab13ff95bc65f6180d40cf1e234b834e/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1061
> Then to EnumerableCalc
> https://github.com/apache/calcite/blob/75511b82ab13ff95bc65f6180d40cf1e234b834e/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1065
>
>
> On Mon, Jan 29, 2024 at 10:10 AM Eric Berryman 
> wrote:
>
> > When compared to the jdbc implementation, which works for me using the
> > EnumerableConvention.
> > I see the difference is that the EnumerableProject gets mapped to a
> > CalcProject.
> > Mapped to Project.NONE
> >
> > https://github.com/apache/calcite/blob/75511b82ab13ff95bc65f6180d40cf1e234b834e/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1061
> > Then to CalcProject
> >
> > https://github.com/apache/calcite/blob/75511b82ab13ff95bc65f6180d40cf1e234b834e/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1065
> >
> > That sounds like this rule: CoreRules.PROJECT_TO_CALC
> > But, I don't see this getting applied as such.
> >
> > Thank you again,
> > Eric
> >
> > On Sun, Jan 28, 2024 at 12:35 PM Eric Berryman 
> > wrote:
> >
> >> I was trying some examples, the first being the EndToEndExampleBindable:
> >>
> >> https://github.com/zabetak/calcite/blob/demo-january-2021/core/src/test/java/org/apache/calcite/examples/foodmart/java/EndToEndExampleBindable.java
> >>
> >> This worked fine for me, but when I attempted to use the pattern in
> >> EndToEndExampleEnumerable:
> >>
> >> https://github.com/zabetak/calcite/blob/demo-january-2021/core/src/test/java/org/apache/calcite/examples/foodmart/java/EndToEndExampleEnumerable.java
> >>
> >> I am getting the following error message:
> >> Suppressed: java.lang.UnsupportedOperationException: null
> >> at
> >> org.apache.calcite.adapter.enumerable.EnumerableProject.implement(EnumerableProject.java:91)
> >> at
> >> org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:114)
> >> ... 53 common frames omitted
> >>
> >> from the line:
> >> Bindable executablePlan = EnumerableInterpretable.toBindable(
> >> new HashMap<>(),
> >> null,
> >> (EnumerableRel) phyPlan,
> >> EnumerableRel.Prefer.ARRAY);
> >>
> >> I'm confused how the EnumerableProject implement method has this
> >> java.lang.UnsupportedOperationException, and what am I missing. I'm not
> >> quite connecting the dots on this, and how it should look.
> >>
> >> Thank you!
> >> Eric
> >>
> >>

Re: calcite 1.36.0 release procedures

[Stamatis Zampetakis]

Having a gradle task to ensure that class files are valid bytecode
seems like a good idea. It may not be 100% bulletproof since we yet
rely on another tool/library to perform the verification but still
having an extra check will not hurt if it is lightweight.

It makes sense to enforce a certain JDK version/build during the
release. In that case I would suggest also having some gradle logic
perform this validation and not just text in the release instructions.

Automating the release procedure would be great but to do that it is
required to have reproducible builds [1]. See [2] for more background
around this topic.

Best,
Stamatis

[1] https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
[2] https://lists.apache.org/thread/qkstx4o9qw90fxzqcfnp69w33h7vw2yg

On Fri, Jan 26, 2024 at 11:27 PM Guillaume Masse
 wrote:
>
> Hi Julian,
>
> I don't think it's necessary to require JDK > 8. Here is what I propose:
>
> 1) I can add a gradle task to make sure each classfile from the release
> jars are valid bytecode.
>
> 2) I think we need to make sure we compile with the latest build available.
> Per https://lists.apache.org/thread/nrt4ysoc14p20sq23z744jyfqh1bznyh it was
> compiled with 8u371 the 10 of November 2023. At the time the latest build
> of java 8 was 391, so that's 2 builds behind:
>
> 371 > 18 April 2023
> 381 > 18 July 2023
> 391 > 17 October 2023
> 401 > 16 January 2024
>
> I can't be sure that this was the root cause for the broken classfile.
>
> 3) Let's make the release procedure automatic / part of the CI. I can
> assist on this task, but I will need help from commiters since I don't
> have any credentials for maven central.
>
> Guillaume
>
>
> On Fri, Jan 26, 2024 at 2:44 PM Julian Hyde  wrote:
>
> > These JDK bugs seem to be fixed in more recent java versions. Should we
> > mandate that releases are built on a recent JDK (say 17 or 21)? Currently
> > we mandate JDK 8.
> >
> >
> > > On Jan 26, 2024, at 9:03 AM, Ruben Q L  wrote:
> > >
> > > I have the impression that this might be caused by a bug in the JDK, see
> > > similar issues:
> > > https://gitlab.ow2.org/asm/asm/-/issues/317789
> > > https://bugs.openjdk.org/browse/JDK-8144185
> > > https://bugs.openjdk.org/browse/JDK-8187805
> > >
> > >
> > >
> > > On Thu, Dec 21, 2023 at 8:58 PM Julian Hyde  wrote:
> > >
> > >> I think I was mistaken about Guava. I agree with Benchao's analysis
> > >> and #2 seems to be caused by something other than Guava.
> > >>
> > >> On Thu, Dec 21, 2023 at 3:53 AM Benchao Li 
> > wrote:
> > >>>
> > >>> There are two exceptions here:
> > >>>
> > >>> # 1
> > >>> java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds
> > >>> for length 297
> > >>> at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695)
> > >>> at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711)
> > >>> at
> > >> org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777)
> > >>> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929)
> > >>> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> > >>> at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
> > >>> at org.objectweb.asm.ClassReader.accept(ClassReader.java:425)
> > >>> at remapper.bug.RemapperTest.runTest(RemapperTest.java:60)
> > >>> at remapper.bug.RemapperTest.calcite36(RemapperTest.java:36)
> > >>>
> > >>> # 2
> > >>> java.lang.IllegalArgumentException: Invalid end label (must be visited
> > >> first)
> > >>> at
> > >>
> > org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453)
> > >>> at
> > >>
> > org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996)
> > >>> at
> > >>
> > org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757)
> > >>> at
> > >>
> > org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257)
> > >>> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614)
> > >>> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> > >>> at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
> > >>> at org.objectweb.asm.ClassReader.accept(ClassReader.java:425)
> > >>> at remapper.bug.RemapperTest.runTest(RemapperTest.java:53)
> > >>> at remapper.bug.RemapperTest.calcite36WithCheck(RemapperTest.java:39)
> > >>>
> > >>>
> > >>> Sorry that I'm not clear in my previous email. My testing results are
> > >>> all about #1.
> > >>>
> > >>> I tested 1.34.0/1.35.0/1.36.0 with JDK8 on both Linux and Mac for #2,
> > >>> they all reproduced the problem. Hence this is not a problem of
> > >>> platform, and also not a problem of 1.35.0->1.36.0, I guess this is an
> > >>> asm usage problem related to JDK8's output.
> > >>>
> > >>> In summary,
> > >>> for #1 problem, it seems to be a problem of the Guava version we're
> > >>> compiling against,
> > >>> for #2 problem, it seems not to be a problem since both
> > >>> 1.34.0/1.35.0/1.36.0 

Re: Draft: board report for 2023 Q4

[Stamatis Zampetakis]

Looks good thanks for putting it together Benchao!

On Fri, Jan 5, 2024 at 2:41 PM Benchao Li  wrote:
>
> Hello,
>
> Below you can find a draft of this quarter's board report. I plan to
> submit the final version next Tuesday (Jan 9, 2024).
>
> Please let me know if you have any additions or corrections.
>
> Best,
> Benchao Li
> ---
>
> ## Description:
> Apache Calcite is a highly customizable framework for parsing and planning
> queries on data in a wide variety of formats. It allows database-like
> access,
> and in particular a SQL interface and advanced query optimization, for data
> not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building
> local and remote JDBC and ODBC database drivers. Avatica has an independent
> release schedule and its own repository.
>
> ## Issues:
> There are no issues requiring board attention.
>
> ## Membership Data:
> Apache Calcite was founded 2015-10-22 (8 years ago)
> There are currently 74 committers and 27 PMC members in this project.
> The Committer-to-PMC ratio is roughly 5:2.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Benchao Li on 2023-01-27.
> - Hongyu Guo was added as committer on 2023-11-03
> - Runkang He was added as committer on 2023-11-03
> - Lei Shen was added as committer on 2023-11-01
> - Mihai Budiu was added as committer on 2023-10-26
> - Ran Tao was added as committer on 2023-10-28
>
> Following our yearly rotation tradition, the PMC has elected a new
> chair: Benchao Li.
>
> ## Project Activity:
> Apache Calcite 1.36.0 was released on 2023-11-10. It contains
> contributions from 30 contributors, and resolves 125 issues.  The new
> release adds 30 new SQL functions in various libraries such as
> BigQuery and Spark, contains many improvements hardening TABLESAMPLE,
> integrates SQL Logic Test suite, and many more core improvements such
> as the support of recursive WITH and CREATE TABLE ... LIKE DDL.
>
> Apache Calcite Avatica 1.24.0 was released on 2023-12-04. It features
> mostly dependency upgrades with some minor bug fixes and features, and
> contains a breaking change: due to CALCITE-5678, date literals not
> satisfying the Gregorian calendar will be rejected.
>
> Apache Calcite Avatica Go 5.3.0 was released on 2023-12-11. It is a
> maintenance release of Avatica Go including dependency updates and bug
> fixes. This release supports Go 1.20 and 1.21, which are currently the
> versions supported and maintained by the Go team.
>
> ## Community Health:
> The community maintains a healthy status.
>
> Mailing list and JIRA activity has slightly dropped (dev@ -4%, issues@
> -36%, JIRA opened -47%, JIRA closed -33%) due to the holiday of
> Chrismas and New Year. However, commit (+52%) and PRs closed(+27%) has
> increased on the contrary, one reason is we have many commits/PRs
> testing the unstable test problem in CALCITE-6123, another reason is
> some new contributors are used to opening and closeing the PR
> repeatedly.
>
> The number of non-committer (contributor) commits per month:
> +--+---+-+
> | year | month | contributor_commits |
> +--+---+-+
> | 2023 |10 |  35 |
> | 2023 |11 |  21 |
> | 2023 |12 |  13 |
> +--+---+-+
>
> The number of active reviewers per month:
> +--+---+--+
> | year | month | active_reviewers |
> +--+---+--+
> | 2023 |10 |8 |
> | 2023 |11 |   12 |
> | 2023 |12 |6 |
> +--+---+--+
>
> Top reviewers in the last 3 months:
> +--+-+
> | committer| reviews |
> +--+-+
> | Jiajun  |  11 |
> | Julian Hyde|  11 |
> | Mihai Budiu  |   9 |
> +--+-+
>
>
> The number of non-committer commits and active reviewers are slightly
> decreased compared to Q3 due to the holidays, but the number is still
> very good compared to times before 2023 Q2.

Re: FilterableTable query planner question

[Stamatis Zampetakis]

Hey Eric,

When you have a disjunction in the WHERE clause it may not be safe to
push the condition below a join especially when it comes to outer
joins. I suppose that the FilterJoinRule [1] is the place that you
want to check to see if the filter can be pushed below the join and
into the scan.

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/8d9b27f1ace7f975407920cb88806715b1f0ef82/core/src/main/java/org/apache/calcite/rel/rules/FilterJoinRule.java

On Tue, Jan 2, 2024 at 11:22 PM Eric Berryman  wrote:
>
> Hello,
>
> I'm making a FilterableTable with LDAP as a backend.
>
> I noticed in the FilterableTable method:
> public Enumerable scan(DataContext root, List filters)
>
> The filters list is empty if the sql where clause is checking the same
> field in all tables.
> ie.
> select test1.field test2.field
> from test1
> full outer join test2 on test1.id = test2.id
> where test1.field = 'myval' or test2.field = 'myval';
>
> When I do an EXPLAIN PLAN FOR select ...
> I notice the BindableTableScan filters array is empty with both:
> where test1.field = 'myval' or test2.field = 'myval';
> and it has the filter if I remove one:
> where test1.field = 'myval'
>
> Is there an example I could be pointed to help understand the query planner
> here, and hopefully write my implementation such that the filters show up
> as expected for each table?
>
> Thank you!
> Eric

Re: Some questions about planners and rules

[Stamatis Zampetakis]

Hello,

I would say that the default rules are those tested to work well when
Calcite is used as an application through the various adapters etc. If
you are using Calcite as a library for building a new query planner
for another system then you should probably pick the rules from
scratch based on the particular needs of your application. Certainly
many of those default rules would be useful at some optimization
stage.

A bad choice of rules could definitely lead to infinite loops but I
would say that this affects more the HepPlanner which applies all the
rules till it reaches a fix point. If I remember well the
VolcanoPlanner is able to detect that the cost is not improving after
a certain number of rule invocations and stop applying useless
transformations. Nevertheless, a bad choice of rules could lead to
increased optimization time so it should be done with care.

Best,
Stamatis

On Fri, Dec 29, 2023 at 5:08 AM Hongyu Guo  wrote:
>
> Hi devs,
>
> I have recently started to learn about planners and rules in Calcite, and
> as a beginner, I have some questions.
>
>
> 1. I noticed that RelOptUtil#registerDefault Rules provides default rules.
> Can I assume that these rules are the set of rules recommended by Calcite?
> What were these rules selected based on?
>
>
> 2. There are some mutually exclusive rules in the default rules, such as
> PROJECT_ FILTER_TRANSPOSE and FILTER_PROJECT_TRANSPOSE。Will the presence of
> two rules simultaneously result in infinite loops in the planner?
>
>
> Looking forward to the developer's response and discussion!
>
>
> Thank you so much for your time!
>
> Best regards,
>
> Hongyu

Re: [MINOR]

[Stamatis Zampetakis]

The presence of the "minor" keyword in the commit summary is a bit
redundant. I would argue that if the message is precise enough the
person reading it can infer it is minor or not.

Moreover, the minor classification is subjective. Some people consider
minor things that do not change code at all. Others consider minor
things that don't touch production code. The addition of tests and
fixing a broken build is also considered minor by few people. All
these examples are taken from the current commits which landed in
Calcite.

For the reasons above, I feel that we don't really need to include
"minor" in the commit summary but don't feel very strongly about it
anyways.

In the past we agreed that if a change is trivial/minor we don't need
a JIRA id. Although, this is convenient for getting this committed
faster without too much hassle it has some drawbacks.

If at some point in the future we want to attach more information to a
particular commit, this is not possible since we cannot modify the git
history. When there is a JIRA we can always add new comments and
clarifications there even if the ticket is resolved.

Having a JIRA id is also a convenient way and readable way for
referring to particular commits. The commit hash can be used to
identify a commit in Apache main branch but if the same commit is
backported to other forks/branches using the hash becomes more
cumbersome. This is especially useful in downstream projects and forks
for tracking that a specific change landed in various branches.

Maybe in the future we should reconsider the optionality of the JIRA
id in the commit summary. If this happens then [CALCITE-X][MINOR]
would start getting overly long so this may be another argument
against including "minor" in the message.

Best,
Stamatis

On Wed, Jan 3, 2024 at 10:25 AM Ran Tao  wrote:
>
> This format most likely comes from other open source projects.
> If calcite has its own specifications, such as how to set the title for PRs
> that do not require a jira name,
> IMHO, it can be introduced in the contribution doc.
> Commiters can also review PRs according to this specification.
>
> Best Regards,
> Ran Tao
>
>
> Istvan Toth  于2024年1月3日周三 16:11写道：
>
> > Perhaps the square bracket convention ?
> > If the ticket starts with CALICITE-\d+ , then make sure that the JIRA
> > ticket id is between brackets.
> >
> > Also check for Gerrit Change IDs which are often added automatically, and a
> > paint to remove.
> >
> > Istvan
> >
> > On Tue, Jan 2, 2024 at 10:50 PM Tanner Clary  > .invalid>
> > wrote:
> >
> > > I like the [MINOR] prefix because it makes it easy to identify simple
> > > commits (via grep or ctrl+f), the same way [CALCITE-1234] makes it easy
> > to
> > > find commits related to [CALCITE-1234]. I also like that it maintains the
> > > "[...]" styling at the beginning of the commit message.
> > >
> > > Neither of these reasons is strong enough for me to say I oppose, just
> > some
> > > minor (heh) counter-arguments.
> > >
> > > -Tanner
> > >
> > > On Tue, Jan 2, 2024 at 1:05 PM Julian Hyde  wrote:
> > >
> > > > Ralph Waldo Emerson once wrote: “A foolish consistency is the
> > > > hobgoblin of little minds, adored by little statesmen and philosophers
> > > > and divines."
> > > >
> > > > That said, people tend to bring conventions from other projects to
> > > > Calcite, and we end up with chaos. By which I mean, lots of
> > > > self-expression, but no standards, and therefore commit messages that
> > > > have lower information content, and more work for the release manager
> > > > coercing them into a consistent change log.
> > > >
> > > > In Calcite we have not used '[MINOR]' as a prefix to minor commits. If
> > > > it is minor, it doesn't need a jira case, and doesn't need a prefix.
> > > > But a few commits with [MINOR] crept in, starting about a year ago.
> > > > Once or twice, I asked people to remove them, but the PRs had already
> > > > been merged.
> > > >
> > > > Any objections if I add a lint rule to fail the build if the commit
> > > > message contains [MINOR]?
> > > >
> > > > While I'm there, any other standards we should enforce?
> > > >
> > > > Julian
> > > >
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: st...@cloudera.com
> > cloudera.com 
> > [image: Cloudera] 
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image: Cloudera
> > on LinkedIn] 
> > --
> > --
> >

[ANNOUNCE] New Calcite PMC chair: Benchao Li

[Stamatis Zampetakis]

Calcite community members,

I am pleased to announce that we have a new PMC chair and VP as per
our tradition of rotating the chair once a year. I have resigned, and
Benchao Li was duly elected by the PMC and approved unanimously by the
Board.

Please join me in congratulating Benchao!

Best regards,
Stamatis

[jira] [Created] (CALCITE-6162) Add rule(s) to remove joins with constant single tuple relations

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6162:


 Summary: Add rule(s) to remove joins with constant single tuple 
relations
 Key: CALCITE-6162
 URL: https://issues.apache.org/jira/browse/CALCITE-6162
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Stamatis Zampetakis


In various cases SQL users tend to create joins even when it is not really 
necessary. One common pattern is creating joins (or cartesian products) with 
constant relations with exactly one tuple.

*Q1*
Before:
{code:sql}
select e.empno, e.ename, c.* from emp e cross join (select 5, 
current_timestamp) c;
{code}
After:
{code:sql}
select e.empno, e.ename, 5, current_timestamp from emp e;
{code}
*Q2*
Before:
{code:sql}
select e.empno, e.ename, c.t from emp e inner join (select 7934 as ono, 
current_timestamp as t) c on e.empno=c.ono;
{code}
After:
{code:sql}
select e.empno, e.ename, current_timestamp from emp e where e.empno=7934;
{code}
In the queries outlined above the one side of the join is constant and has 
exactly one tuple so the join can be dropped.

In a nutshell the new rule(s) should be able to transform the "Before" to 
"After" for the above queries.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reminder about ASF release approval

[Stamatis Zampetakis]

Hey everyone,

The community is growing and more and more people get involved in
testing and voting for releases, which is great for the project. For
this purpose, I would like to send a brief reminder around the ASF
release policy [1] especially the part about approving releases. I
would encourage all people, especially those with binding votes to go
over the document and check the requirements for casting +1 votes so
that we keep delivering high quality releases that comply with the ASF
guidelines.

Best,
Stamatis

[1] https://www.apache.org/legal/release-policy.html#release-approval

Re: [DISCUSS] Using easy-fix label in JIRA

[Stamatis Zampetakis]

In our doc page we are already suggesting the use of the "newbie"
label [1]. If someone has cycles to go over JIRA issues and label them
accordingly that would be indeed very helpful for newcomers.

A few months ago, I did a quick cross project JIRA search for similar
labels and it appears that "newbie" is the most popular one; I would
suggest continuing using that one.

labels = newbie 7K
labels = "newbie++" 292
labels = "newbie," 4
labels = newbiee  11
labels = newbiew 1
labels = newbe 7
labels = noob 111
labels = easy 158
labels = easy-fix 233
labels = easyfix 2.4K
labels = easy-fix 233
labels = Newcomer 15
labels = beginner 1531
labels = beginner-friendly 24
labels = beginners 45

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/acc087a8a7c1d301ec713965dae782d14d969064/site/develop/index.md?plain=1#L354

On Sun, Dec 10, 2023 at 9:41 AM LakeShen  wrote:
>
> I think that it is useful to help newcomers who want to contribute to
> Calcite to find their first issue.
>
> Best,
> LakeShen
>
> Jiajun Xie  于2023年12月10日周日 15:29写道：
>
> > Hi, community:
> >
> > Recently, more and more newcomers have started to contribute.
> > Improving documentation and adding unit tests are very easy tasks.
> > But some JIRA issues are also easy to fix.
> >
> > # For current contributors
> > We can create or tag some JIRA issues that have referential solutions.
> > - Improve error messages, refer [CALCITE-4265][1].
> > - Add some functions in some libraries, refer [CALCIT-5826][2].
> > - Unparse some functions in some SqlDialects, refer [CALCITE-6150][3].
> > - ...
> >
> > # For future contributors
> > You can filter[4] JIRA issues by an easy-fix label, then submit your PR.
> > I saw that the easy-fix-list[5] of Apache Linkis received many
> > contributions.
> > Hope more people join Calcite.
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-4265
> > [2] https://issues.apache.org/jira/browse/CALCITE-5826
> > [3] https://issues.apache.org/jira/browse/CALCITE-6150
> > [4]
> >
> > https://issues.apache.org/jira/browse/CALCITE-5525?jql=project%20%3D%20CALCITE%20AND%20labels%20%3D%20easy-fix
> > [5] https://github.com/apache/linkis/issues/1161
> >

Re: Failing CI

[Stamatis Zampetakis]

I really appreciate the fact that many people jumped in and helped to
shoot this down and restabilise CI. Many thanks from my side as well
to everyone involved!

Best,
Stamatis

On Thu, Dec 7, 2023 at 12:46 AM Julian Hyde  wrote:
>
> Thank you to everyone who worked on this - Stamatis, Benchao, Hongyu, Jiajun, 
> and maybe some I’ve missed. It sounded like a very tricky bug to diagnose. 
> (And it wasn’t even a Calcite bug in the end!) A solid, reliable CI system is 
> the foundation of high-quality software.
>
> Gian, The cause seems to have been a subtle bug in Druid’s query cache. If 
> you would like us to log a bug against Druid, let us know.
>
> Julian
>
>
> > On Dec 6, 2023, at 8:06 AM, Tanner Clary  
> > wrote:
> >
> > Thanks for fixing this, Stamatis!
> >
> > -Tanner
> >
> > On Wed, Dec 6, 2023 at 8:04 AM Benchao Li  wrote:
> >
> >> An update on this, it has been solved in CALCITE-6123 by Stamatis, the
> >> Druid test is supposed to be stable now, thanks Stamatis for fixing
> >> this and also thanks everyone who helped on this.
> >>
> >> Julian Hyde  于2023年12月2日周六 05:11写道：
> >>>
> >>> CI has been failing intermittently for three weeks. In 9 of the last18
> >> commits to main, the Druid test has failed. See all the red ‘x’ symbols on
> >> https://github.com/apache/calcite/commits/main.
> >>>
> >>> This has a been logged as
> >> https://issues.apache.org/jira/browse/CALCITE-6123, and there has been
> >> plenty of discussion but it hasn’t been resolved. Without functioning CI,
> >> we are flying blind. There is a high likelihood of more breaking changes
> >> going in. What should we do?
> >>>
> >>> I suggest that we close the main branch until CI is fixed.
> >>>
> >>> Julian
> >>>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
>

Re: [VOTE] Release apache-calcite-avatica-go-5.3.0 (release candidate 0)

[Stamatis Zampetakis]

Upgraded Docker version to 24.0.7, build afdd53b and rerun the
built/test checks.

 * Built from git tag and run tests OK
 * Built from source artifacts and run tests OK

Changing my vote to +1 (binding)
Stamatis

On Thu, Dec 7, 2023 at 10:40 AM Stamatis Zampetakis  wrote:
>
> Ubuntu 20.04.6 LTS,
> docker-compose version 1.25.5, build 8a1c60f6
> Docker version 20.10.5, build 55c4c88
>
>  * Checked signatures and checksums OK
>  * Checked diff between repo and artifacts OK
>  * Went over release note OK
>  * Checked README.md (Logged CALCITE-6158)
>  * Checked LICENSE, NOTICE OK
>  * All source files have ASF headers OK
>  * No unexpected binary files OK (find . -type f -exec file {} \; |
> grep -v text)
>  * Built from git tag and run tests KO (Logged CALCITE-6159)
>  * Built from source artifacts and run tests KO (Logged CALCITE-6159)
>
> 0 (binding)
>
> I couldn't build/test the release on my environment (CALCITE-6159)
> thus my vote. I will try updating docker and compose versions as it
> was suggested in the past and will try again.
>
> Best,
> Stamatis
>
> On Mon, Dec 4, 2023 at 5:19 AM Francis Chuang  
> wrote:
> >
> > Hi all,
> >
> > I have created a release for Apache Calcite Avatica Go 5.3.0, release
> > candidate 0.
> >
> > Thanks to everyone who has contributed to this release. The release
> > notes are available here:
> > https://github.com/apache/calcite-avatica-go/blob/v5.3.0-rc0/site/_docs/go_history.md
> >
> > The commit to be voted on:
> > https://gitbox.apache.org/repos/asf?p=calcite-avatica-go.git;a=commit;h=8657d0ac4148dca20f67253e78be222d10cc9c6a
> >
> > The hash is 8657d0ac4148dca20f67253e78be222d10cc9c6a
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-go-5.3.0-rc0/
> >
> > The hashes of the artifacts are as follows:
> > src.tar.gz 4927DE62 7C46BE04 2F8092E6 86D5FE36 A3E0115F 7D1672AB
> > F0B32285 AB437FE53B6347CE 5824E1D9 C37A24D0 8B80088C 132F706D D484F0F8
> > C2961453 941AC048
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/francischuang.asc
> >
> > Instructions for running the test suite is located here:
> > https://github.com/apache/calcite-avatica-go/blob/v5.3.0-rc0/site/develop/avatica-go.md#testing
> >
> > Please vote on releasing this package as Apache Calcite Avatica Go 5.3.0.
> >
> > To run the tests without a Go environment, install docker and docker
> > compose. Then, in the root of the release's directory, run: docker
> > compose run test
> >
> > When the test suite completes, run "docker compose down" to remove and
> > shutdown all the containers.
> >
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Calcite Avatica Go 5.3.0
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
> > Francis

Re: [VOTE] Release apache-calcite-avatica-go-5.3.0 (release candidate 0)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS,
docker-compose version 1.25.5, build 8a1c60f6
Docker version 20.10.5, build 55c4c88

 * Checked signatures and checksums OK
 * Checked diff between repo and artifacts OK
 * Went over release note OK
 * Checked README.md (Logged CALCITE-6158)
 * Checked LICENSE, NOTICE OK
 * All source files have ASF headers OK
 * No unexpected binary files OK (find . -type f -exec file {} \; |
grep -v text)
 * Built from git tag and run tests KO (Logged CALCITE-6159)
 * Built from source artifacts and run tests KO (Logged CALCITE-6159)

0 (binding)

I couldn't build/test the release on my environment (CALCITE-6159)
thus my vote. I will try updating docker and compose versions as it
was suggested in the past and will try again.

Best,
Stamatis

On Mon, Dec 4, 2023 at 5:19 AM Francis Chuang  wrote:
>
> Hi all,
>
> I have created a release for Apache Calcite Avatica Go 5.3.0, release
> candidate 0.
>
> Thanks to everyone who has contributed to this release. The release
> notes are available here:
> https://github.com/apache/calcite-avatica-go/blob/v5.3.0-rc0/site/_docs/go_history.md
>
> The commit to be voted on:
> https://gitbox.apache.org/repos/asf?p=calcite-avatica-go.git;a=commit;h=8657d0ac4148dca20f67253e78be222d10cc9c6a
>
> The hash is 8657d0ac4148dca20f67253e78be222d10cc9c6a
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-go-5.3.0-rc0/
>
> The hashes of the artifacts are as follows:
> src.tar.gz 4927DE62 7C46BE04 2F8092E6 86D5FE36 A3E0115F 7D1672AB
> F0B32285 AB437FE53B6347CE 5824E1D9 C37A24D0 8B80088C 132F706D D484F0F8
> C2961453 941AC048
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/francischuang.asc
>
> Instructions for running the test suite is located here:
> https://github.com/apache/calcite-avatica-go/blob/v5.3.0-rc0/site/develop/avatica-go.md#testing
>
> Please vote on releasing this package as Apache Calcite Avatica Go 5.3.0.
>
> To run the tests without a Go environment, install docker and docker
> compose. Then, in the root of the release's directory, run: docker
> compose run test
>
> When the test suite completes, run "docker compose down" to remove and
> shutdown all the containers.
>
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite Avatica Go 5.3.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Francis

[jira] [Created] (CALCITE-6160) Missing ASF header from go.mod and go.sum files

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6160:


 Summary: Missing ASF header from go.mod and go.sum files
 Key: CALCITE-6160
 URL: https://issues.apache.org/jira/browse/CALCITE-6160
 Project: Calcite
  Issue Type: Bug
  Components: avatica-go
Affects Versions: avatica-go-5.2.0
Reporter: Stamatis Zampetakis
Assignee: Francis Chuang


The go.mod and go.sum files in [avatica-go 
repository|https://github.com/apache/calcite-avatica-go] do not contain the ASF 
header according to the [ASF 
policy|https://www.apache.org/legal/src-headers.html#headers].

I am not familiar with go so maybe it is impossible to add comments in these 
files so in that case I guess it is ok to leave them as is.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-6159) SIGABRT when running tests with docker-compose

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6159:


 Summary: SIGABRT when running tests with docker-compose
 Key: CALCITE-6159
 URL: https://issues.apache.org/jira/browse/CALCITE-6159
 Project: Calcite
  Issue Type: Bug
  Components: avatica-go
Affects Versions: avatica-go-5.2.0
 Environment: docker --version
Docker version 20.10.5, build 55c4c88

docker-compose --version
docker-compose version 1.25.5, build 8a1c60f6
Reporter: Stamatis Zampetakis
Assignee: Francis Chuang


{code:bash}
docker-compose run test
{code}

{noformat}
WARNING: The GOPATH variable is not set. Defaulting to a blank string.
Creating network "apache-calcite-avatica-go-530-src_default" with the default 
driver
Creating apache-calcite-avatica-go-530-src_phoenix_1 ... done
Creating apache-calcite-avatica-go-530-src_hsqldb_1  ... done
?   github.com/apache/calcite-avatica-go/v5/errors  [no test files]
?   github.com/apache/calcite-avatica-go/v5/generic [no test files]
?   github.com/apache/calcite-avatica-go/v5/hsqldb  [no test files]
?   github.com/apache/calcite-avatica-go/v5/internal[no test files]
?   github.com/apache/calcite-avatica-go/v5/message [no test files]
?   github.com/apache/calcite-avatica-go/v5/phoenix [no test files]
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f35ff61bd3c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f35ff61bd3c
stack: frame={sp:0x7fff98c7f800, fp:0x0} stack=[0x7fff98480c80,0x7fff98c7fc90)
0x7fff98c7f700:  0x00c58000  0x 
0x7fff98c7f710:  0x  0x 
0x7fff98c7f720:  0x  0x7fff98c7f948 
0x7fff98c7f730:  0x00c32000  0x7fff98c7f968 
0x7fff98c7f740:  0x0002  0x800e 
0x7fff98c7f750:  0x  0x 
0x7fff98c7f760:  0x  0x 
0x7fff98c7f770:  0x  0x 
0x7fff98c7f780:  0x009a874d  0x0003 
0x7fff98c7f790:  0x00e0dc2c  0x 
0x7fff98c7f7a0:  0x009a91e2  0x0006 
0x7fff98c7f7b0:  0x00e0dc2a  0x 
0x7fff98c7f7c0:  0x009a89fe  0x0004 
0x7fff98c7f7d0:  0x  0x7f35ff689194 
0x7fff98c7f7e0:  0x  0x000d 
0x7fff98c7f7f0:  0x00a79653  0x7f35ff61bd2e 
0x7fff98c7f800: <0x  0x114d148a7e0ab100 
0x7fff98c7f810:  0x0006  0x7f35ff58e740 
0x7fff98c7f820:  0x7fff98c7fad0  0x 
0x7fff98c7f830:  0x00ddde00  0x7f35ff5ccf32 
0x7fff98c7f840:  0x7f35ff764e70  0x7f35ff5b7472 
0x7fff98c7f850:  0x0020  0x7f35ff764703 
0x7fff98c7f860:  0x0d68  0x7f35ff611290 
0x7fff98c7f870:  0x7f35ff7605e0  0x0001 
0x7fff98c7f880:  0x000a  0x7f35ff58e740 
0x7fff98c7f890:  0x  0x00ddde00 
0x7fff98c7f8a0:  0x0006  0x7f35ff612ee9 
0x7fff98c7f8b0:  0x7f35ff764680  0x7f35ff6132f3 
0x7fff98c7f8c0:  0x7f35ff764680  0x000a 
0x7fff98c7f8d0:  0x7f35ff58e740  0x7f35ff60e87a 
0x7fff98c7f8e0:  0x7f35ff764840  0x114d148a7e0ab100 
0x7fff98c7f8f0:  0x7f35ff764840  0x7f35ff764840 
runtime: g 0: unknown pc 0x7f35ff61bd3c
stack: frame={sp:0x7fff98c7f800, fp:0x0} stack=[0x7fff98480c80,0x7fff98c7fc90)
0x7fff98c7f700:  0x00c58000  0x 
0x7fff98c7f710:  0x  0x 
0x7fff98c7f720:  0x  0x7fff98c7f948 
0x7fff98c7f730:  0x00c32000  0x7fff98c7f968 
0x7fff98c7f740:  0x0002  0x800e 
0x7fff98c7f750:  0x  0x 
0x7fff98c7f760:  0x  0x 
0x7fff98c7f770:  0x  0x 
0x7fff98c7f780:  0x009a874d  0x0003 
0x7fff98c7f790:  0x00e0dc2c  0x 
0x7fff98c7f7a0:  0x009a91e2  0x0006 
0x7fff98c7f7b0:  0x00e0dc2a  0x 
0x7fff98c7f7c0:  0x009a89fe  0x0004 
0x7fff98c7f7d0:  0x  0x7f35ff689194 
0x7fff98c7f7e0:  0x  0x000d 
0x7fff98c7f7f0:  0x00a79653  0x7f35ff61bd2e 
0x7fff98c7f800: <0x  0x114d148a7e0ab100 
0x7fff98c7f810:  0x0006  0x7f35ff58e740 
0x7fff98c7f820:  0x7fff98c7fad0  0x 
0x7fff98c7f830:  0x00ddde00  0x7f35ff5ccf32 
0x7fff98c7f840:  0x

[jira] [Created] (CALCITE-6158) No instructions for building/testing the project in README file

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6158:


 Summary: No instructions for building/testing the project in 
README file
 Key: CALCITE-6158
 URL: https://issues.apache.org/jira/browse/CALCITE-6158
 Project: Calcite
  Issue Type: Bug
  Components: avatica-go
Affects Versions: avatica-go-5.2.0
Reporter: Stamatis Zampetakis
Assignee: Francis Chuang


The [avatica-go repository|https://github.com/apache/calcite-avatica-go] and 
subsequently the source distribution do not contain instructions (or links to 
the appropriate place in the documentation) to build and test the project in 
the README file at the root of the project. 

Someone who downloads the released sources and its not familiar with the 
project may have a hard time using it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release Apache Calcite Avatica 1.24.0 (release candidate 1)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 8.1.1

 * Checked signatures and checksums OK
 * Checked diff between repo and artifacts OK
 * Went over release note OK (Minor comments at the end)
 * Checked README, NOTICE, LICENSE OK ([1, 2, 3, 4] should be ported to Avatica)
 * All source files have ASF headers OK (raised [5] to investigate if
additional action are needed for some files without header)
 * No unexpected binary files OK (find . -type f -exec file {} \; |
grep -v text)
 * Checked LICENSE, NOTICE, signature, and checksum for
avatica-core-1.24.0.jar in nexus [8] OK
 * Built from git tag and run tests (./gradlew build) OK
 * Built from source artifacts and run tests (gradle build) OK

In the release notes, I think we should highlight that CALCITE-5678
[6] is a breaking change. We probably also have to update the testing
and compatibility section to reflect what we actually test against.
Sergey pointed out correctly [7] that the description is a bit stale
in Calcite and the same holds for Avatica.

+1 (binding)

Best,
Stamatis

[1] 
https://github.com/apache/calcite/commit/d802314e69f1eb975558ad3fd6013dc05ddaef7a
[2] https://issues.apache.org/jira/browse/CALCITE-6096
[3] https://issues.apache.org/jira/browse/CALCITE-6097
[4] https://issues.apache.org/jira/browse/CALCITE-6099
[5] https://issues.apache.org/jira/browse/CALCITE-6148
[6] https://issues.apache.org/jira/browse/CALCITE-5678
[7] 
https://issues.apache.org/jira/browse/CALCITE-6137?focusedCommentId=17790295=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17790295
[8] 
https://repository.apache.org/content/repositories/orgapachecalcite-1222/org/apache/calcite/avatica/avatica-core/1.24.0/avatica-core-1.24.0.jar


On Thu, Nov 30, 2023 at 6:01 AM Benchao Li  wrote:
>
> +1(binding)
>
> - Verified checksum and signature (OK)
> - Gone through release note (OK)
> - Diffed source release with git repo (OK)
> - Checked files in Nexus (OK)
> - Compile and test sources with JDK8 (OK)
>
> Noted that the tag link
> (https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.24.0-rc1)
> is directed to a non existed page, it should be
> https://github.com/apache/calcite-avatica/tree/avatica-1.24.0-rc1
>
> Istvan Toth  于2023年11月29日周三 15:26写道：
>
> >
> > +1 (non binding)
> >
> > Thanks for making the JDK8 build painless.
> >
> > On Wed, Nov 29, 2023 at 1:21 AM Francis Chuang 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created a build for Apache Calcite Avatica 1.24.0, release
> > > candidate 1.
> > >
> > > Thanks to everyone who has contributed to this release.
> > >
> > > You can read the release notes here:
> > >
> > > https://github.com/apache/calcite-avatica/blob/avatica-1.24.0-rc1/site/_docs/history.md
> > >
> > > The commit to be voted upon:
> > >
> > > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=4c0999ba154915eee201bd7a037e223e9368a97d
> > >
> > > Its hash is 4c0999ba154915eee201bd7a037e223e9368a97d
> > >
> > > Tag:
> > >
> > > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.24.0-rc1
> > >
> > > The artifacts to be voted on are located here:
> > >
> > > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.24.0-rc1
> > > (revision 65633)
> > >
> > > The hashes of the artifacts are as follows:
> > >
> > > 41da104d811b3925cbf19452cbe3acac8271bae188efb21f4eed9738dd13b1e353923f63a5ed73aefb05a1d5004211e0d438520b268c9e5d3f3f6c8f98c3be00
> > > *apache-calcite-avatica-1.24.0-src.tar.gz
> > >
> > > A staged Maven repository is available for review at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachecalcite-1222/org/apache/calcite/
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/francischuang.asc
> > > https://www.apache.org/dist/calcite/KEYS
> > >
> > > To create the jars and test Apache Calcite Avatica: "gradle build
> > > -Prelease -PskipSign".
> > >
> > > If you do not have a Java/Gradle environment available, you can run the
> > > tests using docker. To do so, install docker and docker-compose, then
> > > run "docker-compose run test" from the root of the directory.
> > >
> > > Please vote on releasing this package as Apache Calcite Avatica 1.24.0.
> > >
> > > The vote is open for the next 72 hours and passes if a majority of at
> > > least three +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Calcite Avatica 1.24.0
> > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > [ ] -1 Do not release this package because...
> > >
> > >
> > > Here is my vote:
> > >
> > > +1 (binding)
> > >
> > > Francis
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: st...@cloudera.com
> > cloudera.com 
> > [image: Cloudera] 
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on 

[jira] [Created] (CALCITE-6148) Missing ASF header from some files in source distribution

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6148:


 Summary: Missing ASF header from some files in source distribution
 Key: CALCITE-6148
 URL: https://issues.apache.org/jira/browse/CALCITE-6148
 Project: Calcite
  Issue Type: Bug
  Components: avatica
Affects Versions: avatica-1.24.0
Reporter: Stamatis Zampetakis


The [ASF release 
policy|https://www.apache.org/legal/src-headers.html#asf-source-header-and-copyright-notice-policy]
 requires all files (with few exception) in a distribution to have an ASF  
header.

During the release of Apache Calcite Avatica 1.24.0 I noticed that some files 
in the source distribution do not contain an ASF header.

{code:bash}
grep -RiL "Licensed to the Apache Software Foundation (ASF)" | grep -v 
"LICENSE" | grep -v "NOTICE" | grep -v "README"
{code}

{noformat}
core/src/main/resources/META-INF/services/java.sql.Driver
.gitattributes
metrics-dropwizardmetrics/src/main/resources/META-INF/services/org.apache.calcite.avatica.metrics.MetricsSystemFactory
site/favicon.ico
site/img/logo.png
site/img/feather.png
site/Gemfile.lock
site/_layouts/external.html
site/_layouts/docs.html
site/_layouts/default.html
site/_layouts/news_item.html
site/_layouts/news.html
site/_layouts/page.html
site/_sass/_style.scss
site/_sass/_pygments.scss
site/_sass/_normalize.scss
site/_sass/_gridism.scss
site/_sass/_mixins.scss
site/_sass/_font-awesome.scss
site/css/screen.scss
site/.gitignore
site/_includes/news_contents_mobile.html
site/_includes/footer.html
site/_includes/docs_contents.html
site/_includes/anchor_links.html
site/_includes/docs_contents_mobile.html
site/_includes/docs_ul.html
site/_includes/section_nav.html
site/_includes/primary-nav-items.html
site/_includes/docs_option.html
site/_includes/header.html
site/_includes/news_contents.html
site/_includes/top.html
.ratignore
noop-driver/src/main/resources/META-INF/services/java.sql.Driver
.gitignore
{noformat}

Most of the files listed above should not have an ASF header, and its normal 
that they don't but we may have to enrich the LICENSE file to clarify the 
licensing of some files such as .html in the site directory.

Moreover for files under META-INF/services we can check if it is possible to 
add the ASF header.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Calcite 1.36.0 released

[Stamatis Zampetakis]

Thanks for driving this release Benchao! Well planned and timely executed.
Awesome work!

On Fri, Nov 10, 2023, 2:49 PM Benchao Li  wrote:

> The Apache Calcite team is pleased to announce the release of Apache
> Calcite 1.36.0.
>
> Calcite is a dynamic data management framework. Its cost-based
> optimizer converts queries, represented in relational algebra, into
> executable plans. Calcite supports many front-end languages and
> back-end data engines, and includes an SQL parser and, as a
> sub-project, the Avatica JDBC driver.
>
> This release comes three months after 1.35.0. It includes more than
> 125 resolved issues, comprising of a few new features as well as
> general improvements and bug-fixes. It includes more than 30 new
> functions, various improvements hardening TABLESAMPLE, and also the
> integration of SQL Logic Test suite.
>
> You can start using it in Maven by simply updating your dependency to:
>
> 
> org.apache.calcite
> calcite-core
> 1.36.0
> 
>
> If you'd like to download the source release, you can find it here:
>
> https://calcite.apache.org/downloads/
>
> You can read more about the release (including release notes) here:
>
> https://calcite.apache.org/news/2023/11/10/release-1.36.0/
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at:
>
> https://calcite.apache.org/
>
> Thanks to everyone involved!
>
> Benchao Li, on behalf of the Apache Calcite Team
>
> --
>
> Best,
> Benchao Li
>

Re: [VOTE] Release Apache Calcite 1.36.0 (release candidate 0)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 7.6.1

 * Checked signatures and checksums OK
 * Went over release note OK
 * Checked diff between repo and artifacts OK
 * Checked README, LICENSE, NOTICE files (Logged CALCITE-6095 to CALCITE-6099)
 * Built from git tag and run tests (./gradlew clean build) OK
 * Built from source distribution and run tests OK
 * Checked META-INF content/structure from calcite-core and
calcite-redis artifacts in nexus OK

I have the impression that all LICENSE/NOTICE issues logged under
CALCITE-609[5-9] are minor and can be fixed after the release. If
someone has a different opinion please comment under the respective
ticket and if necessary mark it as blocker.

+1 (binding)

Best,
Stamatis

On Wed, Nov 8, 2023 at 11:26 AM Ran Tao  wrote:
>
> Thanks Benchao for preparing this calcite-1.36 !
>
> 1.check calcite-src link, tag link, calcite-src-tar link, signature link
> OK
>
> 2.check SHA-512 & GPG signature
> OK
>
> 3.download calcite-1.36.0-src.tar.gz, gradle build with macos sonoma 14.0,
> m1 chip, open jdk-11
> OK
>
> 4.run sqlline with some version related sqls
> OK
>
> +1(non-binding) from my side.
>
> Best Regards,
> Ran Tao
>
>
> Benchao Li  于2023年11月8日周三 18:00写道：
>
> > Thank you all for helping verify the release. One small declaration,
> > only PMC voting is binding according to apache guidelines[1].
> >
> > [1] https://www.apache.org/foundation/voting.html#binding-votes
> >
> > Bertil Chapuis  于2023年11月8日周三 17:32写道：
> > >
> > > Thank you for preparing the release.
> > > - Checksum: ok
> > > - Signature: ok
> > > - Build + test: ok
> > >
> > > +1 (binding)
> > >
> > > Notice that the build requires Gradle v7.6.1 and fails with Gradle v8.4.
> > >
> > > Best,
> > >
> > > Bertil
> > >
> > >
> > > > On 8 Nov 2023, at 08:50, Ruben Q L  wrote:
> > > >
> > > > Thanks Benchao for preparing this release.
> > > >
> > > > - Release notes: ok
> > > > - Checksum: ok
> > > > - Signature: ok
> > > > - Diff source release and git repository: ok
> > > > - Build + tests: ok
> > > > - Calcite-based application test suite: ok
> > > >
> > > > +1 (binding)
> > > >
> > > > Best,
> > > > Ruben
> > > >
> > > >
> > > > On Tue, Nov 7, 2023 at 9:32 PM Francis Chuang <
> > francischu...@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks for being RM for this release, Benchao!
> > > >>
> > > >> My vote is: +1 (binding)
> > > >>
> > > >> - Verified GPG signature - OK
> > > >> - Verified SHA512 - OK
> > > >> - Diffed source release and git repository - OK
> > > >> - Checked release notes on tag
> > > >> (
> > > >>
> > https://github.com/apache/calcite/blob/calcite-1.36.0-rc0/site/_docs/history.md
> > )
> > > >>
> > > >> - OK
> > > >> - Checked year and versions in NOTICE, README and HOWTO - OK
> > > >> - Ran tests (gradle check) - OK
> > > >> - Spot checked Nexus artifacts - OK
> > > >>
> > > >> Environment:
> > > >> Eclipse-temurin:19-jammy docker container in WSL2 (Ubuntu 22.04.3) on
> > > >> Windows 11 22h2
> > > >>
> > > >> $ docker version
> > > >> Client: Docker Engine - Community
> > > >>  Cloud integration: v1.0.35+desktop.5
> > > >>  Version:   24.0.6
> > > >>  API version:   1.43
> > > >>  Go version:go1.20.7
> > > >>  Git commit:ed223bc
> > > >>  Built: Mon Sep  4 12:32:16 2023
> > > >>  OS/Arch:   linux/amd64
> > > >>  Context:   default
> > > >>
> > > >> Server: Docker Desktop
> > > >>  Engine:
> > > >>   Version:  24.0.6
> > > >>   API version:  1.43 (minimum version 1.12)
> > > >>   Go version:   go1.20.7
> > > >>   Git commit:   1a79695
> > > >>   Built:Mon Sep  4 12:32:16 2023
> > > >>   OS/Arch:  linux/amd64
> > > >>   Experimental: false
> > > >>  containerd:
> > > >>   Version:  1.6.22
> > > >>   GitCommit:8165feabfdfe38c65b599c4993d227328c231fca
> > > >>  runc:
> > > >>   Version:  1.1.8
> > > >>   GitCommit:v1.1.8-0-g82f18fe
> > > >>  docker-init:
> > > >>   Version:  0.19.0
> > > >>   GitCommit:de40ad0
> > > >>
> > > >> $ gradle -v
> > > >>
> > > >> 
> > > >> Gradle 7.6.1
> > > >> 
> > > >>
> > > >> Build time:   2023-02-24 13:54:42 UTC
> > > >> Revision: 3905fe8ac072bbd925c70ddbf4463341f4b4
> > > >>
> > > >> Kotlin:   1.7.10
> > > >> Groovy:   3.0.13
> > > >> Ant:  Apache Ant(TM) version 1.10.11 compiled on July 10 2021
> > > >> JVM:  19.0.2 (Eclipse Adoptium 19.0.2+7)
> > > >> OS:   Linux 5.15.90.1-microsoft-standard-WSL2 amd64
> > > >>
> > > >> $ java --version
> > > >> openjdk 19.0.2 2023-01-17
> > > >> OpenJDK Runtime Environment Temurin-19.0.2+7 (build 19.0.2+7)
> > > >> OpenJDK 64-Bit Server VM Temurin-19.0.2+7 (build 19.0.2+7, mixed mode,
> > > >> sharing)
> > > >>
> > > >> Francis
> > > >>
> > > >> On 8/11/2023 7:55 am, Julian Hyde wrote:
> > > >>> Oops, 

[jira] [Created] (CALCITE-6099) Missing LICENSE entry for _pygments.scss

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6099:


 Summary: Missing LICENSE entry for _pygments.scss
 Key: CALCITE-6099
 URL: https://issues.apache.org/jira/browse/CALCITE-6099
 Project: Calcite
  Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis


The 
[_pygment.scss|https://github.com/apache/calcite/blob/4c9011c388f826c36463ae7da875ddb525104376/site/_sass/_pygments.scss]
 file is present both in git and source distribution but there is no info about 
its LICENSE.

I don't know the exact provenance of the file but I found a lot of similarities 
with:
[desert.css|https://github.com/StylishThemes/Syntax-Themes/blob/bfeb1da4fbbb895b9c247e0f289b0e945243e94d/pygments/css-other/desert.css].

If that's the true origin of the file then we should examine the respective 
[LICENSE|https://github.com/StylishThemes/Syntax-Themes/blob/bfeb1da4fbbb895b9c247e0f289b0e945243e94d/LICENSE]
 and if necessary include it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-6098) Remove mentions of Jekyll from LICENSE and NOTICE files

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6098:


 Summary: Remove mentions of Jekyll from LICENSE and NOTICE files
 Key: CALCITE-6098
 URL: https://issues.apache.org/jira/browse/CALCITE-6098
 Project: Calcite
  Issue Type: Task
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


The NOTICE file contains the following statement:
{noformat}
The web site includes files generated by Jekyll.{noformat}
 
However, there is nothing in the [LICENSE of Jekyll 
|https://github.com/jekyll/jekyll/blob/3f3a283018a976da11a0bfcc13a20d43d37ee29f/LICENSE]
 that requires such attribution. 

According to the instructions of composing the [NOTICE 
file|https://infra.apache.org/licensing-howto.html#mod-notice] for ASF projects 
we shouldn't add anything in there that is not *legally* required.

Moreover the generated files are not necessary licensed under the same LICENSE 
with the generator. 

JavaCC, ANTLR, and lots of other source generators use a variety of licenses 
but the generated output is not licensed under the same terms. For instance, 
Calcite uses JavaCC, which is licensed under 
[BSD-3|https://github.com/javacc/javacc/blob/master/LICENSE]  but both the 
grammar as well as the generated .java files are AL2.

As long as we are not packaging bits of Jekyll in Calcite there is no need to 
add explicit mentions in LICENSE or NOTICE files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-6097) cobyism/gridism CSS dependency is mispelled in LICENSE

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6097:


 Summary: cobyism/gridism CSS dependency is mispelled in LICENSE
 Key: CALCITE-6097
 URL: https://issues.apache.org/jira/browse/CALCITE-6097
 Project: Calcite
  Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


The website uses  the 
[gridism.css|https://github.com/apache/calcite/blob/4c9011c388f826c36463ae7da875ddb525104376/site/_sass/_gridism.scss]
 style for presentation purposes. The file is present in git and also in the 
source distribution of Calcite so it should have a proper reference in the 
[LICENSE|https://github.com/apache/calcite/blob/4c9011c388f826c36463ae7da875ddb525104376/LICENSE#L186]
 file.

There is an entry in the LICENSE file but it is mispelled: gridsim vs gridism



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-6096) Remove obsolete html5shiv and respond entries from LICENSE

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6096:


 Summary: Remove obsolete html5shiv and respond entries from LICENSE
 Key: CALCITE-6096
 URL: https://issues.apache.org/jira/browse/CALCITE-6096
 Project: Calcite
  Issue Type: Task
  Components: build
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


The [LICENSE 
file|https://github.com/apache/calcite/blob/4c9011c388f826c36463ae7da875ddb525104376/LICENSE#L184C3-L184C26]
 includes the following entries:

* cobyism:html5shiv:3.7.2
* respond:respond:1.4.2

implying that we are bundling the respective dependencies in the project.

These dependencies correspond to minified javascript files that were removed 
completely from the project [some time 
ago|https://github.com/apache/calcite/commit/7f5e9b8b7e6b4afd8e4f21524aa3c4ce8b7ddb61].
 The references are obsolete and must be removed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] State of the project 2023

[Stamatis Zampetakis]

t to convert Postgres regress
> > > tests
> > >
> > https://github.com/postgres/postgres/tree/master/src/test/regress/expectedto
> > > to Quidem tests. These could be slow tests.
> > >
> > > (2) Documentation; Calcite has a steep learning curve. I think that
> > better
> > > documentation could help attract more developers and users. I would like
> > to
> > > propose adding a blog to the Calcite project web page. If every committer
> > > writes two blog posts per year about some Calcite technical aspect, then
> > > the project can quickly accumulate a lot of helpful information. The dev
> > > mailing list does not seem to be well indexed by search engines; we could
> > > even take some answers for questions on the mailing list and turn them
> > into
> > > short articles. I have written a blog post about several Calcite IRs,
> > which
> > > I plan to post in the following days on our company website (I will
> > > actually send a request for review on this list when I do), but it would
> > be
> > > great to cross-post such documents on the Calcite project blog.
> > >
> > > (3) While poking around I have found examples of data structure that are
> > > probably confusing. Some of them have been known to be confusing for a
> > long
> > > time: e,g: RexLiteral.typeName.
> > >
> > https://github.com/apache/calcite/blob/c83ac69111fd9e75af5e3615af29a72284667a4a/core/src/main/java/org/apache/calcite/rex/RexLiteral.java#L214
> > > A comment from 2006 indicates that this field should not exist.
> > >
> > > Another example is https://github.com/apache/calcite/pull/3411, where I
> > > tried to fix up some problems with the type families. Such PRs are
> > > difficult to accept, in the short term they may increase the code entropy
> > > (like this one does, by creating deprecated functions and keeping the old
> > > ones too), and it is very hard to make sure that they don't break
> > anything
> > > for users, and their value is questionable if people do not actually
> > remove
> > > all uses that should be deprecated. This requires a sustained effort to
> > > cleanup; contributors could periodically self-assign a small cleanup
> > task.
> > >
> > > Thank you,
> > > Mihai
> > >
> > > -Original Message-
> > > From: Stamatis Zampetakis
> > > Sent: Sunday, October 22, 2023 9:00 AM
> > > To: dev@calcite.apache.org
> > > Subject: [DISCUSS] State of the project 2023
> > >
> > > Hello community and happy birthday Calcite,
> > >
> > > Eight years ago (22 October 2015) Calcite graduated as a top-level Apache
> > > project [1]. At that point, the community decided to have an annual
> > “state
> > > of the project” discussion, and we arrived at that time of the year.
> > >
> > > We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023 [2],
> > > with probably one more coming before the end of the year. With roughly
> > 300
> > > commits coming in, it is evident that the community did a great job in
> > > fixing numerous bugs, landing new features, and bringing notable
> > > improvements and optimizations to the project. It is worth highlighting
> > the
> > > collective effort that was done for improving the BigQuery dialect where
> > > more than 60 new SQL functions were added.
> > >
> > > Regarding the sub-project Avatica, we had a release at the beginning of
> > > the year (1.23.0) [3] featuring some new configurations for fetchSize and
> > > SSL key stores along with various other improvements and bug fixes. Since
> > > the last release, there have been roughly 10 new commits in the main
> > branch
> > > mostly comprising dependency upgrades and build fixes. As in previous
> > years
> > > the activity is rather low but there are still volunteers regularly
> > > checking and contributing to the repo.
> > >
> > > In terms of community, I think this has been a great year. We see more
> > and
> > > more people participating in email discussions, Jira tickets and Github
> > > PRs. Our list of committers has grown with Istvan Toth, Alex Plehanov,
> > > Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ Banghart, Dan
> > > Zou; and so has our PMC with Benchao Li joining the team. With nine new
> > > committers and one new PMC member till now, it's probably the best year
> > so
> > > far (and it'

Re: How about enable dependabot?

[Stamatis Zampetakis]

Upgrading dependencies is an important topic thanks for starting the
discussion Hongyu.

In terms of tooling, my experience with Dependabot in Apache Hive is
rather negative. Out of 52 PRs [1] raised by the bot 34 [2] are
failing the build/tests. In most cases committers do not follow-up to
resolve the failures and currently there are only 3 [3] that are
merged to master. In Hive, the presence of the bot adds noise and
wastes resources.

Calcite is a simpler project so maybe Dependabot does a better job but
unless there are people who are willing to track and follow up on
those PRs the result may be similar.

Best,
Stamatis

[1] https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+
[2] 
https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+label%3A%22tests+unstable%22%2C%22tests+failed%22
[3] https://github.com/apache/hive/commits?author=dependabot%5Bbot%5D

On Sun, Nov 5, 2023 at 10:03 PM Francis Chuang  wrote:
>
> Perhaps refreshVersions [1] can be used.
>
> [1] https://splitties.github.io/refreshVersions/
>
> On 6/11/2023 7:54 am, Julian Hyde wrote:
> > I agree that we should be trying to stay on the most recent version of
> > our dependencies (with a few exceptions, such as JavaCC). Most of our
> > dependencies are mature libraries, and the latest version is more
> > likely to fix security problems than to introduce bugs.
> >
> > However, I'm not sure that Dependabot is the best way to do it. One,
> > dependabot generates quite a lot of noise (frequent upgrades). Two, we
> > would have to restructure our build files. The best process is
> > probably to do manual upgrades, like
> > https://github.com/apache/calcite/pull/3504, just before each release.
> > Is there a straightforward way to script those upgrades?
> >
> > Julian
> >
> >
> > On Sun, Nov 5, 2023 at 1:09 AM Jiajun Xie  wrote:
> >>
> >> Hi, Hongyu.
> >>
> >> Your idea is great and you also introduced the steps to use it.
> >>
> >> We need more feedback about benefits and risks from calcite users.
> >> # What are the benefits?
> >> - Quickly fix dependency vulnerabilities.
> >> - Balancing the workload of each upgrade(Not 4.0 to 7.x).
> >> - ...
> >>
> >> # What are the risks?
> >> - The latest version may be unstable.
> >> - The burden of upgrading Calcite for users has increased.
> >> - ...
> >>
> >> For me, the risks are acceptable.
> >> I am willing to help you complete this work.
> >>
> >> On Sat, 4 Nov 2023 at 21:04, Hongyu Guo  wrote:
> >>
> >>> Hi all,
> >>>
> >>> Recently, I opened 2 PRs about removing an unused library[1] and bumping
> >>> various libraries[2]. I noticed that many dependencies of calcite are
> >>> outdated. To address this issue, I suggest enabling dependabot[3] to
> >>> automatically open "bump dependency" PRs and make calcite healthier.
> >>>
> >>> If we enable dependabot, what should we do?
> >>>
> >>> - Add `dependabot.yml` to `.github/`. It is straightforward, just follow
> >>> the instructions in the documentation[4].
> >>> - Refactor gradle project files: Dependabot's support for gradle is not
> >>> sufficient as it only reads the text of `build.gradle.kts`, 
> >>> `build.gradle`,
> >>> and `settings.gradle.kts` instead of running gradle. Additionally,
> >>> dependabot can NOT read `gradle.properties`, so we need to refactor the
> >>> gradle project files.
> >>> - Ignore some dependencies: Some dependencies cannot be upgraded. For
> >>> example, I attempted to bump javacc from 4.0 to 7.x, but due to
> >>> incompatibility caused by the large version span, I had to give up. Also,
> >>> we cannot upgrade elasticsearch due to licensing restrictions.
> >>>
> >>> What is your opinion on dependabot?
> >>>
> >>> [1]https://github.com/apache/calcite/pull/3502
> >>> [2]https://github.com/apache/calcite/pull/3504
> >>> [3]
> >>>
> >>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
> >>> [4]
> >>>
> >>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
> >>>
> >>> Best,
> >>> Hongyu
> >>>

Re: [ANNOUNCE] New committer: Hongyu Guo

[Stamatis Zampetakis]

Welcome Hongyu, and thanks a lot for the work you are doing for the project!

Best,
Stamatis

On Fri, Nov 3, 2023 at 3:57 AM Hongyu Guo  wrote:
>
> Thank you for your introduction! I am delighted to join the Calcite
> community, and I am also grateful for your recognition of me.
>
> I have just graduated from university and have been working at Tencent for
> one year. My current work mainly revolves around Calcite.
> Calcite is a powerful framework that we use to parse, validate, and
> optimize SQL.
>
> I really enjoy contributing and communicating in the community, which has
> helped me grow rapidly. I am looking forward to working with the community
> to make Calcite better in the future.
>
> Thanks again,
> Hongyu Guo
>
> On Fri, Nov 3, 2023 at 10:17 AM Dan Zou  wrote:
>
> > Congrats, Hongyu!
> >
> > Best,
> > Dan Zou
> >
> >
> >
> >
> >
> > > 2023年11月2日 22:27，Benchao Li  写道：
> > >
> > > Apache Calcite's Project Management Committee (PMC) has invited Hongyu
> > > Guo to become a committer, and we are pleased to announce that he has
> > > accepted.
> > >
> > > Hongyu's contributions are around the functions, parser, DDL(CREATE
> > > TABLE LIKE, GRANT, REVOKE), and simplication. Apart from the code
> > > contribution, Hongyu is also very active in the mailing list and code
> > > reviews.
> > >
> > > Hongyu, welcome, thank you for your contributions, and we look forward
> > > to your further interactions with the community! If you wish, please
> > > feel free to tell us more about yourself and what you are working on.
> > >
> > > As your first commit, please add yourself to the contributors list [1]
> > > and the community page will re-generate [2].
> > >
> > > Benchao Li (on behalf of the Apache Calcite PMC)
> > >
> > > [1]
> > https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
> > > [2] https://calcite.apache.org/community/#project-members
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> >
> >

Re: [ANNOUNCE] New committer: Runkang He

[Stamatis Zampetakis]

Congratulations Runkang, keep up the good work!

Best,
Stamatis

On Fri, Nov 3, 2023 at 3:57 AM Hongyu Guo  wrote:
>
> Congratulations!
>
> Best,
> Hongyu Guo
>
> On Fri, Nov 3, 2023 at 10:16 AM Dan Zou  wrote:
>
> > Congrats, Runkang!
> >
> > Best,
> > Dan Zou
> >
> >
> >
> >
> >
> > > 2023年11月3日 09:54，Jiajun Xie  写道：
> > >
> > > Congrats, Runkang!
> > >
> > > On Fri, 3 Nov 2023 at 04:46, Francis Chuang 
> > > wrote:
> > >
> > >> Congrats, Runkang!
> > >>
> > >> On 3/11/2023 3:54 am, Tanner Clary wrote:
> > >>> Congratulations, Runkang!
> > >>>
> > >>> Best,
> > >>> Tanner
> > >>>
> > >>> On Thu, Nov 2, 2023 at 9:11 AM Ran Tao  wrote:
> > >>>
> >  Congrats, Runkang!
> > 
> >  Best Regards,
> >  Ran Tao
> > 
> > 
> >  Benchao Li  于2023年11月2日周四 22:27写道：
> > 
> > > Apache Calcite's Project Management Committee (PMC) has invited
> > > Runkang He to become a committer, and we are pleased to announce that
> > > they have accepted.
> > >
> > > Runkang has added many new functions to Hive and Spark libraries, and
> > > various improvements around simplification and sub-query. Apart from
> > > code contributions, Runkang also helps a lot with code reviews.
> > >
> > > Runkang, welcome, thank you for your contributions, and we look
> > > forward to your further interactions with the community! If you wish,
> > > please feel free to tell us more about yourself and what you are
> > > working on.
> > >
> > > As your first commit, please add yourself to the contributors list
> > [1]
> > > and the community page will re-generate [2].
> > >
> > > Benchao Li (on behalf of the Apache Calcite PMC)
> > >
> > > [1]
> > >
> > >> https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
> > > [2] https://calcite.apache.org/community/#project-members
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> > 
> > >>>
> > >>
> >
> >

Re: [DISCUSS] Calcite Jenkins job fails due to Nexus account problem

[Stamatis Zampetakis]

The only job that we have for publishing snapshots is on Jenkins [1].

The problem of missing nexus username/password credentials seems
similar to what we encountered in the past in INFRA-20044.

Best,
Stamatis

[1] https://ci-builds.apache.org/job/Calcite/job/Calcite-snapshots
[2] https://issues.apache.org/jira/browse/INFRA-20044

On Thu, Nov 2, 2023 at 6:29 AM Francis Chuang  wrote:
>
> The ASF release Gradle plugin requires those values to push to nexus.
>
> It seems strange that it's trying to build and publish. Is it trying to
> build a nightly snapshot and publish it to Nexus? That could be why it's
> throwing up all of these errors.
>
> To fix the error, we'll need to open a ticket with Infra to set those
> values, however, it'd be good to check if there's a GitHub Actions
> workflow doing the same thing, to avoid conflicts.
>
>
> On 2/11/2023 4:20 pm, Benchao Li wrote:
> > Hi all,
> >
> > I noticed that recently jobs are all failing[1][2] with the following
> > message, anyone knows why? Is there anything changed in Gradle
> > recently?
> >
> > Please correct the following before proceeding with the release:
> > 1) Project property 'asfNexusUsername' is not specified
> > 2) Project property 'asfNexusPassword' is not specified
> >
> > [1] https://ge.apache.org/s/w56jhscwxw5ei
> > [2] https://ge.apache.org/s/dbni4geb7yl7q

[ANNOUNCE] New committer: Lei Shen

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Lei Shen to
become a committer, and we are pleased to announce that they have accepted.

Lei has added many new optimizer rules to the project enhancing the
query plans for Calcite and downstream projects and pushed various
enhancements around the TABLESAMPLE clause. Apart from code
contributions, Lei is very active in the dev list participating in
discussions and helping a lot with reviews.

Lei, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

Re: System fields

[Stamatis Zampetakis]

In some projects where I was involved and there was the notion of
"system fields" these used to be "special" columns on the table level
that could be used in queries but they were ignored when expanding the
"*".

Implementation wise, I think we introduced a new interface (not
necessarily connected to Table interface) with a single method
(something along the lines isSystemField(int pos)) and then customized
the validator to check the method and act accordingly.
It can be something really similar to how we handle rolled up columns
in expandStar [1].

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/782d327d24c04e2161102b22f8880204462befd4/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L667

On Mon, Oct 30, 2023 at 11:30 PM Julian Hyde  wrote:
>
> That makes sense.
>
> I asked the questions because I have seen several other definitions of 
> ’system fields’ that seemed - to the person asking for them - to be the only 
> possible definition of ’system fields’.
>
> Rather than trying to come up with a grand notion of ‘system field’ I would 
> prefer to focus on specific behaviors. In this case, the desired behavior 
> seems to be a table [1] to declare its row type and somehow indicate that 
> some of those columns should be ignored when expanding “*” or “alias.*”.
>
> One way to accomplish this is modifying the Table interface. Say, add a 
> method ‘boolean includeInStarExpansion(RelDataTypeField field)’. To use this, 
> the schema author would have to change their implementation of Table.
>
> Another way is to extend RelDataType. Drill took this route, adding 
> RelDataType.isDynamicStruct(), when they wanted to support late-binding 
> schema [2]. To use this, the author would probably need to provide a new type 
> factory or type system.
>
> Another way is a global property, say a regular expression for field names to 
> ignore during star expansion. Simple, but rather inflexible, because it would 
> apply to all tables (and queries) regardless of their source.
>
> Whichever way we provide, it would need to be plumbed through to the 
> SqlValidatorImpl.expandStar method. Very likely there would be changes to 
> interface SqlValidatorNamespace. Or we could re-use 
> SqlValidatorNamespace.getRowTypeSansSystemColumns() method, which seems 
> purpose-built for this (because it was!)
>
> Julian
>
> [1] 
> https://calcite.apache.org/javadocAggregate/org/apache/calcite/schema/Table.html
>
> [2] https://issues.apache.org/jira/browse/CALCITE-1150
>
>
> > On Oct 27, 2023, at 3:06 PM, Gian Merlino  wrote:
> >
> > I take it from your reply that there isn't really a "system field" concept 
> > like this in Calcite today, even though there appear to be things that are 
> > kind of similar. So we can discuss what it might look like.
> >
> > Warning: I'm only vaguely familiar with how this works in other systems, 
> > and completely ignorant of whether the SQL standard touches on this. But 
> > that being said, these seem like reasonable answers to your questions:
> >
> >> * Does a system field appear in all tables (relations, including tables 
> >> derived via a query) or only base tables?
> >
> > I only have a need for them in base tables, so I'd say only base tables. My 
> > feeling is that system fields are these quasi-physical things, so they make 
> > sense coming from a table, but they don't really make sense coming from any 
> > arbitrary relation. I have a hard time coming up with something they'd be 
> > useful for given that an optimizer can morph your relations any which way 
> > it chooses.
> >
> > Btw, this makes me consider the case where an optimizer morphs a base table 
> > call (for example it can swap in a materialized view). There's various ways 
> > to deal with that, I suppose. One is to not allow base table calls to be 
> > replaced with something else if their system fields are being referred to. 
> > Another is to have the system fields show up as null if the base table is 
> > replaced. Another is to have the behavior be undefined and let implementors 
> > do whatever they want: they may be null, or they may take on values that 
> > make sense in the context of whatever the new plan is. (Like, perhaps 
> > selecting __filename would refer to a file that backs the materialized 
> > view.) My first thought is that it's best if the behavior is undefined at 
> > Calcite's level, i.e. dependent on the implementation of the particular 
> > optimization and the particular system field. But that's only because I 
> > can't immediately think of a specific behavior that is always going to be 
> > good.
> >
> >> * If there are many system fields (hundreds), how do we keep the plan of a 
> >> manageable size?
> >
> > I don't know of systems out there that have hundreds of system fields, so 
> > this may be a non issue. In Druid we certainly aren't planning to add more 
> > than 1 right now, and perhaps a small handful in the foreseeable future. 
> > Not sure if this is a 

[ANNOUNCE] New committer: Ran Tao

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Ran Tao to
become a committer, and we are pleased to announce that he has accepted.

Ran has implemented numerous improvements and fixes in the past few
months. He has added new SQL functions to the Spark library, addressed
various issues related to casts and type inference, enhanced
documentation, and made various changes with the aim of improving code
quality

Ran, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

[ANNOUNCE] New committer: Mihai Budiu

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Mihai
Budiu to become a committer, and we are pleased to announce that he
has accepted.

Mihai has addressed numerous compile and runtime issues in SQL
functions and has made several significant improvements to the test
infrastructure to enhance code coverage, quality, and stability. In
addition to code contributions, Mihai is highly active on the mailing
list, where he answers questions, assists others, and shares ideas for
future improvements. Mihai has also been writing blogs and giving
talks about Calcite at various events and conferences, showcasing
Calcite in both simple and more advanced use cases, which can greatly
contribute to the growth of the community.

Mihai, welcome, thank you for your contributions, and we look forward
to your further interactions with the community! If you wish, please
feel free to tell us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

Re: Reviewing blog post

[Stamatis Zampetakis]

Great content Mihai, thanks for putting this together!

As Julian suggested the whole content of the blog could be put in a
news entry under the respective page [1].
Alternatively, if it is a link to an external site you could add it
under the external resources section [2].

Many apache projects also have blog namespace [3] for such content.
This is very similar to Calcite's news page so I don't feel strongly
about having a dedicated "blog" page.

Best,
Stamatis

[1] https://calcite.apache.org/news/
[2] https://calcite.apache.org/community/#external-resources
[3] https://infra.apache.org/project-blogs.html

On Tue, Oct 24, 2023 at 2:34 AM Julian Hyde  wrote:
>
> I have reviewed for both branding and technical content, and it looks great. 
> Thank you for writing.
>
> When it’s final, let me know, and I’ll promote it using the ApacheCalcite 
> Twitter account.
>
> If I were writing the post, I’d qualify the statement "The modern way to 
> implement compilers … is to use the visitor pattern” a little. Visitors are 
> certainly one tool you can use, and they make sense in an OO language like 
> Java that has inheritance (therefore single-dispatch) but weak 
> pattern-matching. But they can be cumbersome, difficult to maintain (because 
> you can’t easily pass state around), and inefficient (because you need to 
> walk the whole tree).
>
> In Calcite, there are tools other than visitor; rewrite rules (RelRule) are 
> preferred in many circumstances, since they walk over a small section of the 
> tree, and do not mutate the tree. Metadata allows you to propagate properties 
> (e.g. unique keys) towards the root of the tree and use caching to avoid 
> continually re-computing.
>
> Julian
>
>
> > On Oct 23, 2023, at 5:16 PM,   wrote:
> >
> > Hello all,
> >
> >
> >
> > I am planning to write a series of blog posts about Calcite.
> >
> > I wrote a first one on 3 or the Calcite IR representations.
> >
> > I would appreciate if anyone could vet it:
> > https://www.feldera.com/blog/calcite-irs/
> >
> > I don't want to mislead readers.
> >
> >
> >
> > Julian suggested adding this kind of stuff to "what's new" as well, so if I
> > find a good place I will send a PR.
> >
> >
> >
> > Thank you!
> >
> > Mihai
> >
>

[DISCUSS] State of the project 2023

[Stamatis Zampetakis]

Hello community and happy birthday Calcite,

Eight years ago (22 October 2015) Calcite graduated as a top-level
Apache project [1]. At that point, the community decided to have an
annual “state of the project” discussion, and we arrived at that time
of the year.

We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023
[2], with probably one more coming before the end of the year. With
roughly 300 commits coming in, it is evident that the community did a
great job in fixing numerous bugs, landing new features, and bringing
notable improvements and optimizations to the project. It is worth
highlighting the collective effort that was done for improving the
BigQuery dialect where more than 60 new SQL functions were added.

Regarding the sub-project Avatica, we had a release at the beginning
of the year (1.23.0) [3] featuring some new configurations for
fetchSize and SSL key stores along with various other improvements and
bug fixes. Since the last release, there have been roughly 10 new
commits in the main branch mostly comprising dependency upgrades and
build fixes. As in previous years the activity is rather low but there
are still volunteers regularly checking and contributing to the repo.

In terms of community, I think this has been a great year. We see more
and more people participating in email discussions, Jira tickets and
Github PRs. Our list of committers has grown with Istvan Toth, Alex
Plehanov, Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ
Banghart, Dan Zou; and so has our PMC with Benchao Li joining the
team. With nine new committers and one new PMC member till now, it's
probably the best year so far (and it's not yet ended). Calcite grows
and evolves because of (and thanks to) its community, so I would like
to thank everyone for being part of this family and working together
in a respectful and motivating environment.

It was nice to see our community members giving talks to conferences
such as ApacheCon East Asia, VLDB, and Community over Code presenting
Calcite and/or its application. As in previous years, we had one
online Calcite meetup, which was a great opportunity for the community
to virtually meet and share some interesting presentations, and
hopefully we could add some in-person events in the years to come.

Over the past few years we always had problems with reviewing pull
requests with a lot of weight falling upon a few individuals. In the
last board report for (Q3 2023), the numbers were a bit more
encouraging showing more people involved in reviews. The problem is
not yet solved but we are moving in a promising direction.

To conclude, I will repeat the questions from previous years:
1) What else are we doing well in the project?
2) What areas do we need to do better?

Please take some time to share your thoughts!

Note that this discussion is for everyone, not only for committers /
PMC members; even if you have never sent an email to the dev list
before, now is a good time to do so :)

Finally, it has been a privilege to serve as Calcite's PMC Chair this
year. I have learnt a lot and I am very grateful for the opportunity
that I was given. Following our yearly rotation tradition, I will step
down as Chair by the end of the year, and a new one will have to be
chosen. As we discussed some time ago [4], if you have any suggestion
and you would like to put someone forward as a potential next Chair,
please send an email to priv...@calcite.apache.org. The nomination
period for the new chair will remain open for the next two weeks (till
2023-11-05). The PMC will study all proposals, discuss, and start a
vote soon after 2023-11-05. The change will be effective some time in
December once the resolution is approved by the board.

Best,
Stamatis

[1] https://calcite.apache.org/news/2015/10/22/calcite-graduates/
[2] https://calcite.apache.org/news/
[3] https://calcite.apache.org/avatica/news/
[4] https://lists.apache.org/thread/gplfqs4snr1b6h62cngyvb65sz41z3fk

[jira] [Created] (CALCITE-6053) Upgrade Calcite to Avatica 1.24.0

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-6053:


 Summary: Upgrade Calcite to Avatica 1.24.0
 Key: CALCITE-6053
 URL: https://issues.apache.org/jira/browse/CALCITE-6053
 Project: Calcite
  Issue Type: Task
Reporter: Stamatis Zampetakis


Upgrade Calcite to Avatica 1.24.0 when it becomes available.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: CALCITE-5678/CALCITE-5957: Datetime literal validation

[Stamatis Zampetakis]

Indeed the ideal would be to have some liberty around the supported
datetime formats and it would be great if we find people who want to
contribute in this direction.

However, this discussion is more about converging and taking a
decision around CALCITE-5678 and CALCITE-5957.

The results of the survey are not really conclusive since the
participation was rather low (4 responses). So far everybody agrees
that the SQL standard literal is valid but there are different
opinions about the rest.

Given the input so far, I am leaning towards leaving the repo as it is
for now. This means that CALCITE-5678 remains in place and it is a
breaking change that will come in the next Avatica version and do
nothing for CALCITE-5957.

We can revisit this decision in the future if necessary. Thanks
everyone for your inputs!

Best,
Stamatis

On Sun, Oct 1, 2023 at 9:56 AM stanilovsky evgeny
 wrote:
>
> Thank you, Stamatis.
>
> I vote for direct following the SQL standard, i.e.:
> only strict SQL standard compliant literals
>
> date like 10-1-1 is strange and contain huge field for human errors.
>
> > Hey everyone,
> >
> > CALCITE-5678, which landed recently in Avatica, enforces strict
> > validation of datetime literals and is inline with the SQL standard
> > specification. However, this strict validation also leads to behavior
> > changes (e.g., CALCITE-5957) since some previously "valid" dates are
> > now considered invalid.
> >
> > Roughly four people have expressed an opinion on the aforementioned
> > JIRAs but since this is a change that will likely have wider impact I
> > would prefer to gather some more inputs from the community.
> >
> > I created a very brief anonymous survey [1] with a few sample literals
> > to aid the decision on what Calcite/Avatica should consider valid and
> > invalid from now onwards.
> >
> > I will close the survey in 96 hours from now. Please fill in the
> > survey and/or comment under the respective tickets.
> >
> > Based on the feedback we can opt to do one of the following:
> > * Revert CALCITE-5678 (restore the liberal parsing of datetime literals)
> > * Merge CALCITE-5957 (stricter parsing but leading zeros are optional)
> > * Reject CALCITE-5957 (only strict SQL standard compliant literals)
> >
> > Best,
> > Stamatis
> >
> > [1] https://forms.gle/23f3yVhJCKZJdqQu8

Draft: board report for 2023 Q3

[Stamatis Zampetakis]

Hello,

Below you can find a draft of this quarter's board report. I plan to
submit the final version next Monday (October 9, 2023).

Please let me know if you have any additions or corrections.

Best,
Stamatis
---

## Description:
Apache Calcite is a highly customizable framework for parsing and planning
queries on data in a wide variety of formats. It allows database-like
access,
and in particular a SQL interface and advanced query optimization, for data
not residing in a traditional database.

Avatica is a sub-project within Calcite and provides a framework for
building
local and remote JDBC and ODBC database drivers. Avatica has an independent
release schedule and its own repository.

## Project Status:
Current project status: ongoing
Issues for the board: none

## Membership Data:
Apache Calcite was founded 2015-10-22 (8 years ago)
There are currently 69 committers and 27 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Benchao Li on 2023-01-27.
- TJ Banghart was added as committer on 2023-07-04
- Dan Zou was added as committer on 2023-07-04

## Project Activity:
Apache Calcite 1.35.0 was released on 2023-07-26. It contains contributions
from 36 contributors, and resolves 140 issues. The new release has many
improvements in the BigQuery and Spark dialect bringing in more than 40 SQL
functions. Additionally, it comes with new optimizations for reducing the size
of generated code and more powerful expression simplifications.

On August 18, 2023, Benchao Li and Jiajun Xie represented the Calcite
community at the Apache Con East Asia by giving talks related with the
project.

## Community Health:
The project is healthy. Previously, it was super healthy. The drop is likely
to the fact that the PMC has not grown in the last six months but this will
very likely change in the near future since a lot of our current committers
are very involved with the project and hopefully they will join the PMC
shortly.

The dev list had a 38% in activity in the past quarter, with busiest threads
been as usual those around releases and introduction of new committers. The
16% increase in activity of the JIRA also contributes to the general increase
in traffic of the dev list.

The number of non-committer (contributor) commits per month:
+-+-+-+
|year |month| contributor_commits |
+-+-+-+
| 2023| 7   | 16  |
| 2023| 8   | 32  |
| 2023| 9   | 32  |
+-+-+-+

The number of active reviewers per month:
+-+-+-+
|year |month|  active_reviewers   |
+-+-+-+
| 2023| 7   | 9   |
| 2023| 8   | 9   |
| 2023| 9   | 10  |
+-+-+-+

Top-3 reviewers in the last quarter:
+---+-+
| committer |   reviews   |
+---+-+
| Jiajun  | 15  |
| Julian Hyde  | 13  |
| Benchao Li  | 11  |
+---+-+

The number of non-committer commits has increased by 9% from the last quarter
(73 commits in Q2 vs. 80 commits in Q3) keeping up the good momentum of having
new people contributing to the project.

The average number of active reviewers per month has increased slightly from
the last quarter (7.6 in Q2 vs. 9.3 in Q3) showing that more people are
participating in the review process which is among the main points of the
project.

In the top reviewers, we can observe that things are a bit more balanced in
Q3. The review count per person in the top-3 tier is lower than usual but in
conjunction with the increase in the number of non-committer commits
it shows that
reviews are more evenly distributed among the members of the community.

CALCITE-5678/CALCITE-5957: Datetime literal validation

[Stamatis Zampetakis]

Hey everyone,

CALCITE-5678, which landed recently in Avatica, enforces strict
validation of datetime literals and is inline with the SQL standard
specification. However, this strict validation also leads to behavior
changes (e.g., CALCITE-5957) since some previously "valid" dates are
now considered invalid.

Roughly four people have expressed an opinion on the aforementioned
JIRAs but since this is a change that will likely have wider impact I
would prefer to gather some more inputs from the community.

I created a very brief anonymous survey [1] with a few sample literals
to aid the decision on what Calcite/Avatica should consider valid and
invalid from now onwards.

I will close the survey in 96 hours from now. Please fill in the
survey and/or comment under the respective tickets.

Based on the feedback we can opt to do one of the following:
* Revert CALCITE-5678 (restore the liberal parsing of datetime literals)
* Merge CALCITE-5957 (stricter parsing but leading zeros are optional)
* Reject CALCITE-5957 (only strict SQL standard compliant literals)

Best,
Stamatis

[1] https://forms.gle/23f3yVhJCKZJdqQu8

Re: Existing rules for duplication of filter conditions above joins which should not be pushed down by FilterIntoJoinRule ?

[Stamatis Zampetakis]

Hey Ian,

I don't think there is such a rule in Calcite but you may find similar
ideas in rules present in other projects.

In Hive for instance, there is the HivePreFilteringRule [1, 2] that
pushes "redundantly" some filters below other operations in the tree.
What is inherently problematic with all these rules that introduce
"duplicate" predicates in the plan is that they require some kind of
state to prevent infinite matching. Additionally, you need to pay
attention when you put together rules that are pushing and pulling
since they can interact badly with each other.

Best,
Stamatis

[1] 
https://github.com/apache/hive/blob/57f096c9a73eb92806f2a7cc97f87fabf5d546fe/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
[2] 
https://issues.apache.org/jira/browse/HIVE-9069?focusedCommentId=14534098=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14534098

On Tue, Sep 26, 2023 at 10:10 PM Ian Bertolacci
 wrote:
>
> Hi,
> I was wondering if there exist any rules to duplicate filters which exist 
> above the join, whose effect is dependent on the result of the join and 
> therefore cannot be *pushed* below a join, but could be *duplicated* below 
> the join.
>
> For example: `select … from A LEFT join B on … where B.field is null`
> Here, the best we could do is push the filter condition into the join 
> condition, but not necessarily below it, because the null-ness of the column 
> is partially dependent on the result of the join.
> However, in this case we can duplicate the condition below the join:
> `select … from A LEFT join (select … from B where B.field is null) as B on … 
> where B.field is null`
> This is because the condition because the null-ness of the column is also 
> partially dependent value of the column.
> With both of these filters in place we capture instances of B which are null 
> because the column is null and because there was no match to B
> This (1) reduced the cardinality of that side of the join, and (2) maintained 
> the original intent of the query.
>
> In this example, I use `is null` but we would like to do this for some of our 
> custom comparison operators.
> For these operators, we cannot do push-down (because it would change the 
> intent of the original query) but doing filter duplication should be fine 
> (though we’re still making sure of that).
>
> I figure that this probably doesn’t exist, in which case I’ll probably use 
> FilterIntoJoinRule as a jumping off point.
> Any other suggestions?
>
> Thanks!
> -Ian J. Bertolacci

Re: Running CI Druid tests locally

[Stamatis Zampetakis]

Hello,

There are some pointers on DruidAdapterIT Javadoc [1]. If you have
problems setting up the environment let me know and I will try to
help.

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/509e8c171268a0599d2612b74c608989cb1bc2ab/druid/src/test/java/org/apache/calcite/test/DruidAdapterIT.java#L54

On Wed, Sep 27, 2023 at 2:44 AM Benchao Li  wrote:
>
> It seems that Druid tests need some additional datasets, and a Druid
> deployment[1].
>
> I've never run it locally manually, but it might be done if you
> prepare the same environment as our CI does.
>
> [1] 
> https://github.com/apache/calcite/blob/509e8c171268a0599d2612b74c608989cb1bc2ab/.github/workflows/main.yml#L427-L473
>
>  于2023年9月27日周三 07:15写道：
> >
> > Hello,
> >
> >
> >
> > I submitted a PR https://github.com/apache/calcite/pull/3435 which passes
> > all tests locally, but fails the Druid tests in CI:
> > https://github.com/apache/calcite/actions/runs/6257240352/job/16989292034?pr
> > =3435
> >
> > Does anyone know what I have to do to reproduce these locally?
> >
> >
> >
> > Thank you,
> >
> > Mihai
> >
> >
> >
>
>
> --
>
> Best,
> Benchao Li

Re: gradle build main branch encounters error

[Stamatis Zampetakis]

Looks like a staleness (gradle caches, build dirs) issue. Clean
everything and start from scratch normally it should go away.

Best,
Stamatis


On Fri, Sep 22, 2023 at 9:43 AM Benchao Li  wrote:
>
> I couldn't reproduce the problem you described locally.
>
> And looking at the history of the main branch[1], it seems most of the
> commits' checking is passing.
>
> Could you give more info about the steps and environment so that we
> can reproduce it?
>
> [1] https://github.com/apache/calcite/commits/main
>
> P.F. ZHAN  于2023年9月22日周五 03:44写道：
> >
> > It seems caused by the pr of issue,
> > https://issues.apache.org/jira/browse/CALCITE-5920
> > I revert this pr locally, it works.
> >
> >
> > On Fri, Sep 22, 2023 at 2:54 AM P.F. ZHAN  wrote:
> >
> > > I rebased the newest code, and built the code locally
> > >
> > > * What went wrong:
> > > Execution failed for task ':linq4j:forbiddenApisMain'.
> > > > de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API
> > > calls failed while scanning class
> > > 'org.apache.calcite.linq4j.QueryableRecorder$57':
> > > java.lang.ClassNotFoundException:
> > > org.apache.calcite.linq4j.QueryableDefaults$NonLeafReplayableQueryable
> > > (while looking up details about referenced class
> > > 'org.apache.calcite.linq4j.QueryableRecorder$57')
> > >
> > >
> > >
>
>
>
> --
>
> Best,
> Benchao Li

Re: DECIMAL(2, 3) meaning

[Stamatis Zampetakis]

If we want to argue about if a type is valid or not I don't see a better
place other than the the type factory. I don't know what exactly is the
need here but if type checking should be performed in another place I would
be curious to know more about this use case.

On Tue, Aug 8, 2023, 6:19 PM stanilovsky evgeny 
wrote:

> Stamatis, as i can see this discussion is about values\literals
> validation
> instead of NUMERIC types, not only types at all.
>
> > I would say that type checks for precision/scale etc would fit better
> > inside the RelDataTypeFactory and its respective type system.
> >
> > Best,
> > Stamatis
> >
> > On Mon, Aug 7, 2023 at 12:39 AM  wrote:
> >>
> >> I found this documentation for Oracle DECIMAL data type:
> >>
> https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/Data-Types.html#GUID-75209AF6-476D-4C44-A5DC-5FA70D701B78,
>
> >> which explains what a SCALE > PRECISION should mean.
> >>
> >> > Scale can be greater than precision, most commonly when e notation
> is
> >> used. When scale is greater than precision, the precision specifies
> the
> >> maximum number of significant digits to the right of the decimal
> point.
> >> For example, a column defined as NUMBER(4,5) requires a zero for the
> >> first digit after the decimal point and rounds all values past the
> >> fifth digit after the decimal point.
> >>
> >> Let me ask a related question: in my backend I want to reject such
> >> numbers. What is the right way to do it?
> >> Should this be done in a SqlShuttle? Or should some Validator class be
> >> extended?
> >>
> >> Thank you,
> >> Mihai
> >>
> >> -Original Message-
> >> From: Julian Hyde
> >> Sent: Sunday, August 06, 2023 2:19 PM
> >> To: dev@calcite.apache.org
> >> Subject: Re: DECIMAL(2, 3) meaning
> >>
> >> As I commented in https://issues.apache.org/jira/browse/CALCITE-5901,
> I
> >> don’t think it’s a bug to support behavior beyond what the standard
> >> requires. Which Calcite does, intentionally.
> >>
> >> Julian
> >>
> >> > On Aug 6, 2023, at 08:35, stanilovsky evgeny
> >>  wrote:
> >> >
> >> > Ok, seems like a bug.
> >> > Feel free to fill the issue.
> >> >
> >> >> I have added this test to SqlOperatorTest:
> >> >>
> >> >>f.checkScalar("cast(0.012 as DECIMAL(2, 5))", new
> >> BigDecimal("0.012"),
> >> >>"DECIMAL(2, 5) NOT NULL");
> >> >>
> >> >> and it has passed. That's why I am asking. It should fail, but it
> >> doesn't.
> >> >>
> >> >> Mihai
> >> >>
> >> >> -Original Message-
> >> >> From: stanilovsky evgeny
> >> >> Sent: Friday, August 04, 2023 7:00 AM
> >> >> To: dev@calcite.apache.org
> >> >> Subject: Re: DECIMAL(2, 3) meaning
> >> >>
> >> >> Hello Mihai.
> >> >> A bit older standard describes Precision as : Precision of decimal
> >> floating-point values is a positive value that specifies the number of
> >> significant decimal digits in the mantissa.
> >> >>
> >> >> Thus:
> >> >> cast(0.012 as DECIMAL(3, 3)) - ok
> >> >> cast(0.012 as DECIMAL(2, 3)) - fail
> >> >> cast(0.012 as DECIMAL(1, 3)) - fail
> >> >> cast(0.012 as DECIMAL(2, 5)) - fail
> >> >>
> >> >>
> >> >>> Hello,
> >> >>>
> >> >>>
> >> >>> I notice that Calcite happily accepts decimal type specifications
> >> >>> where the scale is greater than the precision.
> >> >>>
> >> >>> There are quite a few tests with such types.
> >> >>>
> >> >>>
> >> >>> What is the meaning of such types?
> >> >>>
> >> >>>
> >> >>> The SQL 92 standard has this statement on page 109:
> >> >>>
> >> >>>
> >> >>> 15)The  of an  shall not be greater than
> >> >>>
> >> >>>the  of the .
> >> >>>
> >> >>>
> >> >>> Thank you,
> >> >>>
> >> >>> Mihai
>

Re: TimeString

[Stamatis Zampetakis]

Hey Mihai,

I'm not sure to which part of calcite you refer to by saying that it
supports arbitrary precision strings but it is not uncommon for consumers
to use only a single component from calcite (say the parser) so I wouldn't
be surprised if there are implementation gaps in other places.

Best,
Stamatis


On Fri, Aug 11, 2023, 3:07 AM  wrote:

> Why does Calcite support arbitrary precision time-strings and yet makes it
> practically impossible to extract anything but milliseconds?
>
>
>
> Mihai
>
>

Re: DECIMAL(2, 3) meaning

[Stamatis Zampetakis]

I would say that type checks for precision/scale etc would fit better
inside the RelDataTypeFactory and its respective type system.

Best,
Stamatis

On Mon, Aug 7, 2023 at 12:39 AM  wrote:
>
> I found this documentation for Oracle DECIMAL data type: 
> https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/Data-Types.html#GUID-75209AF6-476D-4C44-A5DC-5FA70D701B78,
>  which explains what a SCALE > PRECISION should mean.
>
> > Scale can be greater than precision, most commonly when e notation is used. 
> > When scale is greater than precision, the precision specifies the maximum 
> > number of significant digits to the right of the decimal point. For 
> > example, a column defined as NUMBER(4,5) requires a zero for the first 
> > digit after the decimal point and rounds all values past the fifth digit 
> > after the decimal point.
>
> Let me ask a related question: in my backend I want to reject such numbers. 
> What is the right way to do it?
> Should this be done in a SqlShuttle? Or should some Validator class be 
> extended?
>
> Thank you,
> Mihai
>
> -Original Message-
> From: Julian Hyde
> Sent: Sunday, August 06, 2023 2:19 PM
> To: dev@calcite.apache.org
> Subject: Re: DECIMAL(2, 3) meaning
>
> As I commented in https://issues.apache.org/jira/browse/CALCITE-5901, I don’t 
> think it’s a bug to support behavior beyond what the standard requires. Which 
> Calcite does, intentionally.
>
> Julian
>
> > On Aug 6, 2023, at 08:35, stanilovsky evgeny  
> > wrote:
> >
> > Ok, seems like a bug.
> > Feel free to fill the issue.
> >
> >> I have added this test to SqlOperatorTest:
> >>
> >>f.checkScalar("cast(0.012 as DECIMAL(2, 5))", new BigDecimal("0.012"),
> >>"DECIMAL(2, 5) NOT NULL");
> >>
> >> and it has passed. That's why I am asking. It should fail, but it doesn't.
> >>
> >> Mihai
> >>
> >> -Original Message-
> >> From: stanilovsky evgeny
> >> Sent: Friday, August 04, 2023 7:00 AM
> >> To: dev@calcite.apache.org
> >> Subject: Re: DECIMAL(2, 3) meaning
> >>
> >> Hello Mihai.
> >> A bit older standard describes Precision as : Precision of decimal 
> >> floating-point values is a positive value that specifies the number of 
> >> significant decimal digits in the mantissa.
> >>
> >> Thus:
> >> cast(0.012 as DECIMAL(3, 3)) - ok
> >> cast(0.012 as DECIMAL(2, 3)) - fail
> >> cast(0.012 as DECIMAL(1, 3)) - fail
> >> cast(0.012 as DECIMAL(2, 5)) - fail
> >>
> >>
> >>> Hello,
> >>>
> >>>
> >>> I notice that Calcite happily accepts decimal type specifications
> >>> where the scale is greater than the precision.
> >>>
> >>> There are quite a few tests with such types.
> >>>
> >>>
> >>> What is the meaning of such types?
> >>>
> >>>
> >>> The SQL 92 standard has this statement on page 109:
> >>>
> >>>
> >>> 15)The  of an  shall not be greater than
> >>>
> >>>the  of the .
> >>>
> >>>
> >>> Thank you,
> >>>
> >>> Mihai
>

Re: Easier and more comprehensive testing

[Stamatis Zampetakis]

Hey Mihai,

Indeed the .iq files are mostly end-to-end tests in Calcite. The
Quidem [1] framework is used for running these tests. The Quidem
approach is pretty neat cause we don't have to carry Java boilerplate
code and formatting around which makes the tests much easier to read,
write, inspect history, and copy-paste across projects. Separating
SQL/DDL/DML files in separate non-java files is used in other projects
as well. The most relevant example that comes to mind is Apache Hive
that has ~5K .q files (see cast4.q [2] for an example) and its own
test framework for running those.

End-to-end tests are great but so are unit tests. One does not replace
the other and both are needed in a project. End-to-end tests are much
slower than unit tests so if we have too many of them we will not be
able to run all of them during the build. Again Hive is nice example
on this front cause due to the big number of end-to-end tests
developers cannot run everything locally (it requires more than 24
hours). As a result, lots of PRs are created by running just a few
tests locally and then waiting for the CI to finish. If test failures
are detected, new commits must be pushed, and the whole process is
repeated; the whole cycle takes several days. Unit tests are there to
catch as many problems as possible early on and save us from this
lengthy back and forth.

The end-to-end tests in Calcite may be few compared to other projects
but we should take into account that Calcite is not a complete DBMS.
DBMS that are built using Calcite tend to have many more end-to-end
tests and they are essentially testing Calcite as well.

To conclude, I am strongly in favor of adding new tests especially if
they help to uncover new bugs and problems in the project. For
end-to-end tests, I would probably invest on Quidem (and .iq files) as
far as it concerns Calcite and I would always push for unit tests when
applicable.

Best,
Stamatis

[1] https://github.com/julianhyde/quidem
[2] 
https://github.com/apache/hive/blob/0447b88d8308be669eb9102637a65736ed2bfccf/ql/src/test/queries/clientpositive/cast4.q

On Sat, Jul 29, 2023 at 5:04 AM LakeShen  wrote:
>
> Hi mbudiu,thank you for bringing SqlLogicTest to calcite.
> Now we're working on an optimizer based on Calcite that supports Postgresql
> semantics as a whole. I am very interested in your bringing the Postgres
> test to Calcite and would like to participate in it. Could you please give
> me a complete collection of reference materials?
> The current document information is scattered, some in sql-logic-test
> github and some in sql-to-dbsp-compiler github, which requires everyone to
> read on github by themselves. If you could give me a complete end-to-end
> demo of adding tests, I would really appreciate it. Best, LakeShen
>
>  于2023年7月29日周六 01:04写道：
>
> > The correct link is
> > https://github.com/feldera/dbsp/tree/main/sql-to-dbsp-compiler, but it's
> > not particularly important, I apologize for the broken one.
> >
> > I appreciate you pointing out the Sql Logic Test project, I wrote that
> > code too. But these two projects are almost entirely disjoint. SLT brings a
> > few million pre-written tests. But none of the SLT tests exercises any of
> > the SQL functions. So you need to write many more tests for each *function*
> > implemented, and most contributions to Calcite lately are functions in
> > different dialects.
> >
> > Moreover, while SLT found quite a few bugs, I only had time to file one of
> > them so far. For each failing test I have to figure out (1) whether it's a
> > duplicate, (2) to create a minimal reproduction, (3) to figure out which
> > part of the compiler is faulty in order to understand where to insert the
> > test, (4) create a reproduction tailored for the faulty  component.
> > Reproductions for planner bugs are different than reproductions for library
> > bugs. But some bugs happen only if you combine some planner rules with some
> > functions.
> >
> > With the proposal below you only need a CLI to a database to generate new
> > tests (e.g., BigQuery): you execute write the queries in the CLI, then
> > copy-paste the output into a test. This makes it easier to file bugs, but
> > harder to diagnose them. But it also makes it much easier to write lots of
> > tests, because it enables people without Calcite expertise to write the
> > tests.
> >
> > Mihai
> >
> > -Original Message-
> > From: stanilovsky evgeny
> > Sent: Friday, July 28, 2023 2:43 AM
> > To: dev@calcite.apache.org
> > Subject: Re: Easier and more comprehensive testing
> >
> > Hello, your github link doesn`t open.
> > plz check discussion here in dev list titled:
> >
> > Running Sql Logic Tests for Calcite
> > This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
> > And this is the PR: https://github.com/apache/calcite/pull/3145
> >
> >
> > > Hello,
> > >
> > >
> > > I am working to test our calcite-based compiler
> > > (https://github.com/feldera/dbsp/sql-to-dbsp-compiler), 

Re: [CANCEL][VOTE] Release Apache Calcite 1.35.0 (release candidate 2)

[Stamatis Zampetakis]

Just to clarify that only CALCITE-5865 is a release blocker. All the
other things are just minor that can be fixed after the release.


On Thu, Jul 20, 2023 at 3:12 PM Xiong Duan  wrote:
>
> Hello Calcite Community,
> I'm canceling this vote [VOTE] Release Apache Calcite 1.35.0
> (release candidate 2)
>  because
> of some problem. When the bug is fixed and I will start the round 3 vote
> process.
>
> The detail of the modifications is as follows:
>  1. Put the Spark Functions Release Summary in the right place.
>  2. Minor improvements in release notes.
>  3. CALCITE-5747 broke a few of the test cases in Apache Druid.
>  4. Remove the README and LICENSE irrelevant licensed
> dependencies. (Is there an automated way to do this? Further discussion
> needed)
>
> Thanks a lot for all your review and advice.
>
> Best,
> Xiong Duan

Re: Signing releases using automated release infra

[Stamatis Zampetakis]

If I remember well it is intentional that some part of the process
(svn upload, nexus close, etc.) is not fully automated and explicitly
requires human intervention for legal purposes.

Best,
Stamatis


On Wed, Jul 19, 2023 at 12:13 PM Francis Chuang
 wrote:
>
> Thanks for bringing this to our attention, Stamatis.
>
> This is definitely a huge step forward and something I'd love to see
> implemented for all Calcite projects.
>
> The only downside at the moment is that the artifacts are not
> automatically uploaded to dist.a.o svn server. The way log4j implemented
> their workflow is for the RM to download those artifacts from GitHub and
> upload them manually. This can probably be implemented using a script
> that the RM would run locally on their system, but isn't ideal.
>
> I will reply to that thread to see if there's some possibility of
> automating the uploading of artifacts to svn via CI as well.
>
> For me, the ideal release process would be to completely eliminate all
> mechanical steps in the release process. The ideal release process would
> be for the RM to only deal with the people and project management side
> of things, e.g.
>
> 1. RM opens a PR containing changes to the changelog and dates in the
> NOTICE and LICENSE files, etc.
> 2. RM shepherds PRs and other changes to the release and gets them
> committed in main.
> 3. Contributors can comment on the PR for the release with suggestions,
> etc for the changelog.
> 4. Once the rc is ready, the PR is merged and an rc release is tagged.
> 5. CI picks up the tag and automatically builds and uploads all release
> artifacts. The vote email is automatically generated in CI, which the RM
> sends to the list.
> 6. RM tallys votes and if the vote fails, builds the next RC, starting
> from step 1.
> 7. If the vote passes, RM tags a final release.
> 8. CI picks up the tag and automatically build and signs artifacts.
> 9. RM announces to the list and updates the website announcing the release.
>
> Francis
>
> On 19/07/2023 6:43 pm, Stamatis Zampetakis wrote:
> > Hello,
> >
> > Allowing CI to automate part of the release process is now approved by
> > LEGAL and some projects are already taking advantage of it. For more
> > information check out the respective thread [1].
> >
> > The Calcite release preparation is quite automated already so we may
> > not really need this at this point but sharing the news since it is a
> > topic that has been discussed a lot in ASF the past few years.
> >
> > Best,
> > Stamatis
> >
> > [1] https://lists.apache.org/thread/y5b375054p1yjb2yprnnt16bt4qyccc2

Signing releases using automated release infra

[Stamatis Zampetakis]

Hello,

Allowing CI to automate part of the release process is now approved by
LEGAL and some projects are already taking advantage of it. For more
information check out the respective thread [1].

The Calcite release preparation is quite automated already so we may
not really need this at this point but sharing the news since it is a
topic that has been discussed a lot in ASF the past few years.

Best,
Stamatis

[1] https://lists.apache.org/thread/y5b375054p1yjb2yprnnt16bt4qyccc2

Re: [VOTE] Release Apache Calcite 1.35.0 (release candidate 2)

[Stamatis Zampetakis]

Ubuntu 20.04.6 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 7.6.1

 * Checked signatures and checksums OK
 * Went over release note OK
 * Checked README, LICENSE, NOTICE, and licenses directory (in source
distribution) OK
 * Checked diff between git repo and release sources (diff -qr
calcite-git calcite-src) OK
 * Built from git tag and run tests (./gradlew clean build) OK
 * Built from source artifacts and run tests (gradlew clean build) OK
 * Checked (LICENSE, NOTICE, checksums, signatures) calcite-core jar
artifacts in the staged maven repository OK

The README mentions "or binary distribution of Apache Calcite" but I
don't think we have one; probably we should remove the respective
bits.

The LICENSE file contains 6 MIT licensed dependencies but we are not
bundling those dependencies in the release distribution [1] so I don't
think we need to mention them; same goes for the licenses under the
respective directory. Will open a separate thread to discuss this
further.

+1 (binding)

Best,
Stamatis

[1] https://infra.apache.org/licensing-howto.html#bundled-vs-non-bundled

On Mon, Jul 17, 2023 at 9:24 PM xiong duan  wrote:
>
> Hi all,
>
>
> I have created a build for Apache Calcite 1.35.0, release
>
> candidate 2.
>
>
> Thanks to everyone who has contributed to this release.
>
>
> You can read the release notes here:
>
> https://github.com/apache/calcite/blob/calcite-1.35.0-rc2/site/_docs/history.md
>
>
> The commit to be voted upon:
>
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=2f5635f13a4eb4b89c119fabf25b1d31e0018426
>
>
> Its hash is 2f5635f13a4eb4b89c119fabf25b1d31e0018426
>
>
> Tag:
>
> https://github.com/apache/calcite/tree/calcite-1.35.0-rc2
>
>
> The artifacts to be voted on are located here:
>
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.35.0-rc2
>
> (revision 63046)
>
>
> The hashes of the artifacts are as follows:
>
> b5d837e3725554254bdad0e4f4a63134193a1e3f44bb9b862d9eeeb7f9a5e9bae01985e2f1857874928dc4dbf1f4ca3ae4debacc096470081c249eba699dc74e
>
> *apache-calcite-1.35.0-src.tar.gz
>
>
> A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachecalcite-1213/org/apache/calcite/
>
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/xiong.asc
>
> https://www.apache.org/dist/calcite/KEYS
>
>
> To create the jars and test Apache Calcite: "gradle build"
>
> (requires an appropriate Gradle/JDK installation)
>
>
> Please vote on releasing this package as Apache Calcite 1.35.0.
>
>
> The vote is open for the next 72 hours and passes if a majority of at
>
> least three +1 PMC votes are cast.
>
>
> [ ] +1 Release this package as Apache Calcite 1.35.0
>
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
>
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
>
> +1 (binding)

Re: [VOTE] Release Apache Calcite 1.35.0 (release candidate 0)

[Stamatis Zampetakis]

Thanks for preparing the release Xiong Duan! Unfortunately, I get the
same problem with Ruben with regards to signature verification.

* Verified checksum: OK
* Verified signature: KO

gpg --verify apache-calcite-1.35.0-src.tar.gz.asc
gpg: assuming signed data in 'apache-calcite-1.35.0-src.tar.gz'
gpg: Signature made Fri 14 Jul 2023 10:47:28 CEST
gpg:using RSA key 53B9528B541C85EC
gpg: bad data signature from key 53B9528B541C85EC: Wrong key usage (0x00, 0x2)
gpg: Can't check signature: Wrong key usage

Moreover the URL below is broken:
https://people.apache.org/keys/committer/xiong.asc

-1 (binding)


On Sat, Jul 15, 2023 at 9:39 PM Ruben Q L  wrote:
>
> Thanks Xiong Duan for preparing this release!
>
> - Release notes: ok
> - Checksum: ok
> - Diff source release and git repository: ok
> - Build + tests: ok
> - Calcite-based application test suite: ok
> - Signature: ko?
> I'm getting the following "wrong key usage" error message:
> $   gpg --verify apache-calcite-1.35.0-src.tar.gz.asc
> apache-calcite-1.35.0-src.tar.gz
> ...
> gpg: bad data signature from key 53B9528B541C85EC: Uso de clave
> incorrecto (0x00, 0x2)
>
> Is there something wrong with my machine, or is anybody else getting the
> same error?
>
> Best,
> Ruben
>
>
>
>
> On Fri, Jul 14, 2023 at 12:33 PM xiong duan  wrote:
>
> > Hi all,
> >
> >
> > I have created a build for Apache Calcite 1.35.0, release
> >
> > candidate 0.
> >
> >
> > Thanks to everyone who has contributed to this release.
> >
> >
> > You can read the release notes here:
> >
> > https://s.apache.org/p3rl9
> >
> >
> > The commit to be voted upon:
> >
> > https://s.apache.org/8io6t
> >
> >
> > Its hash is bd4cac4ee30b4c275f5a229eb1704524c3dbc376
> >
> >
> > Tag:
> >
> > https://github.com/apache/calcite/tree/calcite-1.35.0-rc0
> >
> >
> > The artifacts to be voted on are located here:
> >
> > https://s.apache.org/c95f1
> >
> > (revision 62985)
> >
> >
> > The hashes of the artifacts are as follows:
> >
> >
> > b2eb9b25727d1213f889e8d1a547a1f84c66af3821523ebe19a36ae5bc595daa74b4b4584914012bffca0a24be19a523c45dbaf72064d90f50feffc0de8a5e6e
> >
> > *apache-calcite-1.35.0-src.tar.gz
> >
> >
> > A staged Maven repository is available for review at:
> >
> >
> > https://repository.apache.org/content/repositories/orgapachecalcite-1206/org/apache/calcite/
> >
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/xiong.asc
> >
> > https://www.apache.org/dist/calcite/KEYS
> >
> >
> > To create the jars and test Apache Calcite: "gradle build"
> >
> > (requires an appropriate Gradle/JDK installation)
> >
> >
> > Please vote on releasing this package as Apache Calcite 1.35.0.
> >
> >
> > The vote is open for the next 72 hours and passes if a majority of at
> >
> > least three +1 PMC votes are cast.
> >
> >
> > [ ] +1 Release this package as Apache Calcite 1.35.0
> >
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> >
> > [ ] -1 Do not release this package because...
> >
> >
> > Here is my vote:
> >
> >
> > +1 (binding)
> >

Re: Force push to calcite main

[Stamatis Zampetakis]

@Tanner No worries we all did this at some point in time, thanks a lot
for following up!

@Stanilovsky: I prefer keeping force push in place and avoiding messy
reverts that are usually necessary in various situations where we make
a mistake. All commits are archived so there is nothing that we can't
fix (I think).

On Thu, Jul 6, 2023 at 3:45 PM Tanner Clary
 wrote:
>
> Hello,
>
> This was my mistake, my apologies. I will update the hashes. Sorry for any
> inconvenience.
>
> Tanner
>
> On Thu, Jul 6, 2023 at 3:34 AM stanilovsky evgeny <
> estanilovs...@gridgain.com> wrote:
>
> > I already told that community need to vote for prohibit force push.
> >
> > > Hello,
> > >
> > > It appears that there was a force push to main yesterday [1] rewriting
> > > the history for a bunch of commits. I don't know if it was intentional
> > > or not but it seems that now resolved JIRAs (after CALCITE-5810 I
> > > think) are pointing to non-existent commits.
> > >
> > > Can someone please update the JIRA tickets with the correct commit
> > > hashes and also ensure that we didn't lose anything after the rebase?
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] https://lists.apache.org/thread/7jjnbkkh9tv49sjcc5kg2tm7c54tj861
> >

Force push to calcite main

[Stamatis Zampetakis]

Hello,

It appears that there was a force push to main yesterday [1] rewriting
the history for a bunch of commits. I don't know if it was intentional
or not but it seems that now resolved JIRAs (after CALCITE-5810 I
think) are pointing to non-existent commits.

Can someone please update the JIRA tickets with the correct commit
hashes and also ensure that we didn't lose anything after the rebase?

Best,
Stamatis

[1] https://lists.apache.org/thread/7jjnbkkh9tv49sjcc5kg2tm7c54tj861

Re: [DISCUSS] Sharing the load of reviewing PRs

[Stamatis Zampetakis]

Here are some stats around PRs merged in the calcite main branch in
the last quarter [2023-04-01, 2023-07-01). The stats are not 100%
accurate to cover reviews done under PRs/jira etc but clearly show
that we are quite far from what we have been discussing here.

+---+-+
| committer |   reviews   |
+---+-+
| Julian Hyde  | 28  |
| Benchao Li  | 12  |
| rubenada  | 8   |
| Jiajun  | 8   |
| Stamatis Zampetakis  | 3   |
| Tanner Clary <116591231+tancl...@users.noreply.github.com> | 3
|
| Jacky Lau  | 2   |
| Tanner Clary  | 2   |
| NobiGo  | 2   |
| Feng Zhu  | 1   |
| dssysolyatin  | 1   |
| olivrlee <114513231+olivr...@users.noreply.github.com> | 1   |
| Alessandro Solimando  | 1   |
| ILuffZhe <37180946+iluff...@users.noreply.github.com> | 1   |

I would encourage everyone to set some personal goals in terms of
weekly/monthly reviews so that we collectively improve the numbers.

I will send another update in this thread when I submit the report for
the next quarter Q3 to see if we are making progress or not.

Best,
Stamatis

On Thu, Apr 13, 2023 at 6:51 PM Chapuis Bertil  wrote:
>
> +1
>
> I will likely participate in the effort of reviewing PRs when my schedule 
> allows it.
>
> As pointed out in previous comments, some of us have a limited amount of time 
> available and few experience with certain areas of the codebase. One thing we 
> may experiment with is a shadow review process, similar to what is done with 
> shadow program committees at conferences (e.g., S [1]). For instance, I 
> recently reviewed CALCITE-5160 [2] and asked someone else to accept the PR. 
> Frank Zou provided additional feedback, and I learned new things along the 
> way. I think such a mechanism would really help non-committers and new 
> committers to onboard.
>
> [1] https://www.ieee-security.org/TC/SP2021/shadowpc.html
> [2] https://github.com/apache/calcite/pull/2854
>
>
> > On 12 Apr 2023, at 11:51, Stamatis Zampetakis  wrote:
> >
> > For a long time this has been one of the main issues of the project
> > and I am happy to see discussions to address this issue.
> >
> > I would like to mention that as a contributor, I am, and always have
> > been very grateful to people reviewing my work.
> > The fact that I became a committer of this project is mainly due to
> > Julian and Vladimir Sitnikov who reviewed and merged many of my PRs.
> > I would definitely like to help and make other contributors feel the
> > same but I cannot really commit to specific volume and deadlines
> > spanning several months.
> >
> > I have the feeling that we don't need the PR manager role. The
> > assignment work can be done by bots (e.g., [1]) if needed and we
> > already have our quarterly stats for reporting purposes.
> > If we want to put a human behind this then it makes more sense for
> > this person to be the release manager; this should be the one nagging
> > people for advancing the work and moving towards the release.
> >
> > Regarding reviews coming from non-committers, I am not sure it's
> > possible to do the assignment in GitHub. It's not a big deal though;
> > for me a simple comment that I am going to review would be sufficient.
> > Alternatively, we could consider adopting an equivalent workflow in
> > JIRA and potentially introducing a new state "IN REVIEW"; don't think
> > it is necessary.
> > No matter the choice we should ensure that we have a trackable way to
> > recognise "non-committer" reviewers but I think both GitHub (e.g.,
> > "is:pr reviewed-by:julianhyde is:closed") and JIRA offer the necessary
> > filters;
> > Others projects tend to include such info in the commit message we
> > could also opt for this if we deem necessary.
> >
> > As an immediate action I would encourage everyone willing to help to
> > go to the open PRs on GitHub and either assign some PRs to themselves
> > (in case of committers) or leave a comment about their intention
> > (non-committers).
> >
> > In the meantime we can iterate on this till we reach consensus.
> >
> > Best,
> > Stamatis
> >
> > [1] https://github.com/apps/auto-assign
> >
> >
> > On Wed, Apr 12, 2023 at 10:49 AM Ruben Q L  wrote:
> >>
> >> Hello,
> >>
> >> I understand Julian's frustration. We all know that reviewing PRs is a
> >> recurring problem, and it is not the first tim

Draft: board report for 2023 Q2

[Stamatis Zampetakis]

Hello,

Below you can find a draft of this quarter's board report. I plan to
submit it next Tuesday (July 11, 2023).
Please let me know if you have any additions or corrections.

Best regards,
Stamatis
-

## Description:
Apache Calcite is a highly customizable framework for parsing and planning
queries on data in a wide variety of formats. It allows database-like
access,
and in particular a SQL interface and advanced query optimization, for data
not residing in a traditional database.

Avatica is a sub-project within Calcite and provides a framework for
building
local and remote JDBC and ODBC database drivers. Avatica has an independent
release schedule and its own repository.

## Project Status:
Current project status: ongoing
Issues for the board: none

## Membership Data:
Apache Calcite was founded 2015-10-22 (8 years ago)
There are currently 67 committers and 27 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Benchao Li on 2023-01-27.
- Jacky Lau was added as committer on 2023-06-28
- Oliver Lee was added as committer on 2023-06-13
- Tanner Clary was added as committer on 2023-05-25
- Zhe Hu was added as committer on 2023-06-28

## Project Activity:
There were no releases during this quarter. The last release was Apache
Calcite 1.34.0 on 2023-03-14.

We are actively working towards Apache Calcite 1.35.0 and we plan to release
the new version during July 2023.

On 2023-06-29, the Calcite community organised a Virtual key signing party for
expanding the Web of trust and empowering the cryptographic signatures of our
members and future release managers. 4 PMC members and 2 committers attended
the event.

## Community Health:
The project remains super healthy.

The traffic in JIRA, GitHub, issues@, commits, has increased overall, while
dev@ has dropped by 18%. During this quarter there were more discussions and
exchanges under specific issues/tickets shifting the traffic from dev@ to
issues@ and other places.

The number of non-committer (contributor) commits per month:
+-+-+-+
|year |month| contributor_commits |
+-+-+-+
| 2023| 4   | 17  |
| 2023| 5   | 21  |
| 2023| 6   | 35  |
+-+-+-+

The number of active reviewers per month:
+-+-+-+
|year |month|  active_reviewers   |
+-+-+-+
| 2023| 4   | 6   |
| 2023| 5   | 6   |
| 2023| 6   | 11  |
+-+-+-+

Top-3 reviewers in the last quarter:
+---+-+
| committer |   reviews   |
+---+-+
| Julian Hyde  | 28  |
| Benchao Li  | 12  |
| rubenada  | 8   |
| Jiajun  | 8   |
+---+-+

The number of non-committer commits has increased roughly by 43% from the last
quarter (51 commits in Q1 vs. 73 commits in Q2) which is promising for
inviting new committers in the near term.

The number of active reviewers increased slightly compared to the last quarter
with June being the most active month for everyone. The preparation for the
1.35.0 release and the call for action did help in bringing those numbers up
during June.

In terms of sharing the review load, things are a bit better this quarter but
still far from being completely balanced. There have been discussions around
the topic in the last quarter but we obviously need more involvement from the
community members.

[ANNOUNCE] New committer: TJ Banghart

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited TJ Banghart to
become a committer, and we are pleased to announce that they have accepted.

TJ has been contributing to the community for about a year now. They
introduced many new SQL functions for parsing and formatting dates,
times, and timestamps extending the capabilities of the BigQuery
dialect. Furthermore, they pushed various improvements around JSON
serialization and deserialization, JDBC metadata, and lexical
policies.

TJ, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

[ANNOUNCE] New committer: Dan Zou

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Dan Zou to
become a committer, and we are pleased to announce that they have accepted.

Dan has been doing some great work for the project over the past few
months. They implemented and enabled multiple new SQL functions for
BigQuery and MSSQL dialects, fixed some optimization rules, and
improved documentation and test code.

Dan, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

[ANNOUNCE] New committer: Zhe Hu

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Zhe Hu to
become a committer, and we are pleased to announce that they have accepted.

Zhe Hu has been contributing to the project for a while now. They
improved the stability of the ElasticSearch adapter, worked on
supporting new java versions, and landed various enhancements around
CONCAT and CONVERT functions.

Zhe Hu, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

[ANNOUNCE] New committer: Jacky Lau

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Jacky Lau to
become a committer, and we are pleased to announce that they have accepted.

Jacky has started contributing to the project very recently and in a
very short time they landed many notable improvements around ARRAY and
MAP functions with a particular focus on the Spark dialect.

Jacky, welcome, thank you for your contributions, and we look forward to your
further interactions with the community! If you wish, please feel free to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

Re: Virtual key signing party

[Stamatis Zampetakis]

Great let's do one on June 29, 2023, 14:30 UTC+2 [1, 2]. For those who
can't make it in this slot feel free to propose additional ones.

Best,
Stamatis

[1] https://s.apache.org/cn5xq
[2] https://meet.google.com/vwx-mxxz-ibk


On Tue, Jun 27, 2023 at 4:22 AM xiong duan  wrote:
>
> Hi Stamatis,
> Thanks for organizing this party. I am in UTC+8. How about June 29,
> 2023, 20:30 (UTC+8) So the UTC+2 time should be June 29, 2023, 14:30?
> Here is my fingerprint:
>
> pub   rsa4096 2023-06-27 [SC]
>
>   EC78 877D 98B8 CDD4 9613  3DF2 B989 C696 D102 A552
>
> uid   [ultimate] Xiong Duan (CODE SIGNING KEY) 
>
> sub   rsa4096 2023-06-27 [E]
>
> Michael Mior  于2023年6月27日周二 02:19写道：
>
> > I'll attend if possible. My key needs to be updated. (Not compromised, but
> > I lost access to my previous signing key.)
> >
> > pub   rsa4096 2022-06-17 [SC] [expires: 2038-06-13]
> >   EAC5 89C4 44F4 68FE 7E11  3D2D D8D7 2C13 CF2D 8DDB
> > uid   [ultimate] Michael Mior 
> > uid   [ultimate] Michael Mior 
> > uid   [ultimate] Michael Mior 
> > sub   rsa4096 2022-06-17 [E] [expires: 2038-06-13]
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> >
> > On Mon, Jun 26, 2023 at 3:27 AM Stamatis Zampetakis 
> > wrote:
> >
> > > Hello,
> > >
> > > The last virtual key signing party [1] was about a year ago [2]. In
> > > the upcoming releases, we have some people that are serving as release
> > > managers for the first time, thus it may be a good time to hold
> > > another party.
> > >
> > > Since Duan Xiong is the RM for the next release, I would suggest
> > > holding a meeting in a timezone that is convenient for him.
> > >
> > > @Duan: Can you please propose a time slot that works for you?
> > >
> > > I am in UTC+2 and will try to attend if the slot doesn't fall in the
> > > middle of the night (23:00 - 06:00).
> > >
> > > For those that would like to attend, please reply to this thread with
> > > your public
> > > keys' fingerprint (gpg --fingerprint) before the meeting.
> > >
> > > pub   rsa4096 2019-03-15 [SC] [expires: 2027-03-15]
> > >   0474 9577 FD93 4674 B9CD  45C5 D77C 3383 F192 7570
> > > uid   [ultimate] Stamatis Zampetakis 
> > > uid   [ultimate] Stamatis Zampetakis 
> > > sub   rsa4096 2019-03-15 [E] [expires: 2027-03-15]
> > >
> > > To verify your identity, please bring with you a government issued ID
> > > (preferably passport).
> > >
> > > As a reminder of the procedure, have a look at the notes [1] taken by
> > > Francis
> > > during a previous party!
> > >
> > > Anybody can participate (not need to be a committer to the project).
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> > >
> > >
> > http://www.cryptnet.net/fdp/crypto/keysigning_party/en/keysigning_party.html
> > > [2] https://lists.apache.org/thread/4lvch7sfwszbylpzjt11jc9tokm3d0l4
> > > [3] https://gist.github.com/F21/b0e8c62c49dfab267ff1d0c6af39ab84
> > >
> >

Virtual key signing party

[Stamatis Zampetakis]

Hello,

The last virtual key signing party [1] was about a year ago [2]. In
the upcoming releases, we have some people that are serving as release
managers for the first time, thus it may be a good time to hold
another party.

Since Duan Xiong is the RM for the next release, I would suggest
holding a meeting in a timezone that is convenient for him.

@Duan: Can you please propose a time slot that works for you?

I am in UTC+2 and will try to attend if the slot doesn't fall in the
middle of the night (23:00 - 06:00).

For those that would like to attend, please reply to this thread with
your public
keys' fingerprint (gpg --fingerprint) before the meeting.

pub   rsa4096 2019-03-15 [SC] [expires: 2027-03-15]
  0474 9577 FD93 4674 B9CD  45C5 D77C 3383 F192 7570
uid   [ultimate] Stamatis Zampetakis 
uid   [ultimate] Stamatis Zampetakis 
sub   rsa4096 2019-03-15 [E] [expires: 2027-03-15]

To verify your identity, please bring with you a government issued ID
(preferably passport).

As a reminder of the procedure, have a look at the notes [1] taken by Francis
during a previous party!

Anybody can participate (not need to be a committer to the project).

Best,
Stamatis

[1]
http://www.cryptnet.net/fdp/crypto/keysigning_party/en/keysigning_party.html
[2] https://lists.apache.org/thread/4lvch7sfwszbylpzjt11jc9tokm3d0l4
[3] https://gist.github.com/F21/b0e8c62c49dfab267ff1d0c6af39ab84

Re: OSS-Fuzz

[Stamatis Zampetakis]

I had a quick look at the OSS-Fuzz project [1] and I get the
impression that it is not only security oriented but a general
framework for fuzzy testing components.

I am sure that fuzzy testing can uncover many bugs (especially small
ones) so it's worth having I guess. However, receiving notifications
or creating tickets for every problem might be too much. Currently,
it's hard to keep up with JIRAs and PRs created by humans so not sure
if getting more bug reports will really improve the quality of the
project.

For the record, we have some basic fuzzy testing in Calcite already
[2]. Currently it is mostly disabled and not used much but I remember
that it was pretty efficient in identifying problems in Rex-land.

All-in-all good I like the idea but I will probably not have time to
look into every single bug report that comes in from the automation
tool. If it could be configured to run on PRs and "attack" the new
code that is getting in, that would be really helpful and the load
would be more evenly distributed.

Best,
Stamatis

[1] https://github.com/google/oss-fuzz
[2] 
https://github.com/apache/calcite/blob/3f2ae2f4dd2d6b1fab7c3a91e67a6a6d28523298/core/src/test/java/org/apache/calcite/test/fuzzer/RexProgramFuzzyTest.java#L356

On Fri, Jun 16, 2023 at 8:37 PM Michael Mior  wrote:
>
> Thanks for sharing Julian!
>
> Do we *need* to respond to security issues that are uncovered? I certainly
> agree that we *should* if at all possible. But by choosing not to
> participate, we would be choosing not to respond to *all* security issues
> that might only be uncovered via fuzzing. It seems reasonable to me
> (assuming any discovered vulnerabilities can be kept private), that we
> should be free to ignore issues that are uncovered.
>
> --
> Michael Mior
> mm...@apache.org
>
>
> On Fri, Jun 16, 2023 at 2:31 PM Julian Hyde  wrote:
>
> > Someone from Google logged a case offering to add Calcite to the
> > OSS-Fuzz program. (I work for Google but was not aware that we were
> > being considered.)
> >
> > https://issues.apache.org/jira/browse/CALCITE-5781
> >
> > How do people feel about participating in this program?
> >
> > I think that it could improve our security significantly, but it will
> > take work. The fuzzer might generate a lot of false negatives. It
> > might also generate quite a few genuine security issues that we will
> > need to respond to appropriately. As an all-volunteer project it might
> > put a strain on us.
> >
> > Julian
> >

Re: Support Graph Query In Calcite

[Stamatis Zampetakis]

Hello,

We definitely welcome contributions that could improve the graph query
capabilities in Calcite. If something is part of the SQL standard then
it can be contributed to the core SQL parser of Calcite. If it is not
standard but still widely accepted/used it should probably land in the
babel parser. In every other case, we have the adapters and we can
accept new ones if necessary.

In general smaller contributions are easier to review and merge so if
you can divide your work into smaller parts and create JIRA tickets
for each we can reason about them individually.

Other than that the following threads/topics may be somewhat relevant
to what you are doing.

* https://lists.apache.org/thread/q3yx5tmoxzwwh1n8x4oct0vfxcfpcjwg
* https://lists.apache.org/thread/oohfsp1ddyytn8v13btv21trwkbl5y2f
* https://issues.apache.org/jira/browse/CALCITE-3679
* https://calcite.apache.org/docs/reference.html#match_recognize

Best,
Stamatis

On Mon, Jun 19, 2023 at 6:03 AM pzwpzw
 wrote:
>
> Hi everyone, I have noticed that the SQL 2023 have proposed the Graph 
> Property Query support. 
> http://peter.eisentraut.org/blog/2023/04/04/sql-2023-is-finished-here-is-whats-new
> which we can combine the graph query "Match" statement with the sql.
>
> And also the ISO/GQL have proposed the GQL query. We have implement the SQL 
> extend which can combine SQL with the ISO/GQL query in our open source stream 
> graph engine 
> TuGraph-Analytics(https://github.com/TuGraph-family/tugraph-analytics) based 
> on calcite which can combine the SQL process with the graph query in one 
> streaming engine. We can see the  syntax here: 
> https://github.com/TuGraph-family/tugraph-analytics/blob/master/docs/docs-en/application-development/dsl/overview.md
>
> So I would like to know if the Calcite community plans to support graph query 
> language in the future. If possible, we would like to contribute our graph 
> query extension to  the community.
>
> Thanks.

[ANNOUNCE] New committer: Oliver Lee

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Oliver
Lee to become a committer, and we are pleased to announce that they
have accepted.

Oliver started working with us around November 2022 and since then
they contributed multiple SQL functions to the Calcite repository
bringing lots of improvements to the BigQuery dialect. They improved
the extensibility of the SQL validator and pushed various fixes in
RelNode serialization to JSON.

Oliver, welcome, thank you for your contributions, and we look forward
to your further interactions with the community! If you wish, please
feel free to tell us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

[jira] [Created] (CALCITE-5773) Gradle show tasks fails when creating javadocAggregateIncludingTests

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-5773:


 Summary: Gradle show tasks fails when creating 
javadocAggregateIncludingTests
 Key: CALCITE-5773
 URL: https://issues.apache.org/jira/browse/CALCITE-5773
 Project: Calcite
  Issue Type: Bug
  Components: build
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
 Fix For: 1.35.0


Any attempt to display the available Gradle tasks fails when about to create 
the javadocAggregateIncludingTests task.

{noformat}
./gradlew task
{noformat}

{noformat}
> Task :tasks FAILED

Build calcite FAILURE reason:
Execution failed for task ':tasks':

org.gradle.api.internal.tasks.DefaultTaskContainer$TaskCreationException: Could 
not create task ':javadocAggregateIncludingTests'.
at 
org.gradle.api.internal.tasks.DefaultTaskContainer.taskCreationException(DefaultTaskContainer.java:715)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer.access$600(DefaultTaskContainer.java:76)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer$TaskCreatingProvider.domainObjectCreationException(DefaultTaskContainer.java:707)
at 
org.gradle.api.internal.DefaultNamedDomainObjectCollection$AbstractDomainObjectCreatingProvider.tryCreate(DefaultNamedDomainObjectCollection.java:948)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer$TaskCreatingProvider.access$1401(DefaultTaskContainer.java:654)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer$TaskCreatingProvider$1.run(DefaultTaskContainer.java:680)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer$TaskCreatingProvider.tryCreate(DefaultTaskContainer.java:676)
at 
org.gradle.api.internal.DefaultNamedDomainObjectCollection$AbstractDomainObjectCreatingProvider.calculateOwnValue(DefaultNamedDomainObjectCollection.java:929)
at 
org.gradle.api.internal.provider.AbstractMinimalProvider.calculateValue(AbstractMinimalProvider.java:107)
at 
org.gradle.api.internal.provider.Collectors$ElementFromProvider.collectEntries(Collectors.java:100)
at 
org.gradle.api.internal.provider.Collectors$TypedCollector.collectEntries(Collectors.java:334)
at 
org.gradle.api.internal.provider.Collectors$TypedCollector.collectInto(Collectors.java:329)
at 
org.gradle.api.internal.collections.DefaultPendingSource.realize(DefaultPendingSource.java:62)
at 
org.gradle.api.internal.collections.DefaultPendingSource.realizePending(DefaultPendingSource.java:39)
at 
org.gradle.api.internal.collections.SortedSetElementSource.iterator(SortedSetElementSource.java:63)
at 
org.gradle.api.internal.DefaultDomainObjectCollection.iterator(DefaultDomainObjectCollection.java:128)
at 
org.gradle.api.internal.tasks.DefaultTaskContainer.iterator(DefaultTaskContainer.java:620)
at 
org.gradle.api.tasks.diagnostics.internal.SingleProjectTaskReportModel.forTasks(SingleProjectTaskReportModel.java:31)
at 
org.gradle.api.tasks.diagnostics.TaskReportTask.lambda$buildTaskReportModelFor$4(TaskReportTask.java:263)
at 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:378)
at 
org.gradle.api.tasks.diagnostics.TaskReportTask.buildTaskReportModelFor(TaskReportTask.java:262)
at 
org.gradle.api.tasks.diagnostics.TaskReportTask.taskReportModelFor(TaskReportTask.java:246)
at 
org.gradle.api.tasks.diagnostics.TaskReportTask.projectReportModelFor(TaskReportTask.java:214)
at 
org.gradle.api.tasks.diagnostics.TaskReportTask.lambda$computeProjectModels$2(TaskReportTask.java:178)
at org.gradle.internal.Try.ofFailable(Try.java

[ANNOUNCE] New committer: Tanner Clary

[Stamatis Zampetakis]

Apache Calcite's Project Management Committee (PMC) has invited Tanner
Clary to become a committer, and we are pleased to announce that they
have accepted.

In less than 5 months, Tanner has introduced, enabled, and tested over
30 SQL functions in Calcite. They have been a driving force in
improving the BigQuery dialect and by now an expert in library and
parser changes.

Tanner, welcome, thank you for your contributions, and we look forward
to your further interactions with the community! If you wish, please
feel free to tell us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml

[2] https://calcite.apache.org/community/#project-members

Re: Share intermediate data between nodes during plan execution

[Stamatis Zampetakis]

Hey Vladislav,

I see two operators relevant to your use-case:
* Spool [1]
* Correlate [2]

Both allow some form of propagating data from one side of the join to
the other. Check the code and relevant discussions to see how they are
used.

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/core/Spool.java
[2] 
https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/core/Correlate.java

On Wed, May 24, 2023 at 8:32 AM Roman Kondakov
 wrote:
>
> Hi Vladislav,
>
> Can you share more details regarding your use case?
>
> If you want to share the same object between join inputs, you can use
> RelShuttleImpl [1] to traverse the RelNode tree and every time you are
> visiting a Join relation you need to put the object to the stack and
> remove it from there when you are leaving the Join. In this case the
> shared object will be available on the top of the stack for both sides
> of the Join.
>
> [1]
> https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/RelShuttleImpl.java#L49
>
> Thanks,
>
> Roman
>
> On 24.05.2023 05:44, Vladislav Matyushevski wrote:
> > Greetings Calcite team!
> >
> > I`m using the Calcite + Avatica and curious to know:
> > Is there a way how I can share an Object (e.g. Map) between left and right
> > parts of join?
> > Sharing is needed due to the fact that the left part is producing data that
> > must be re-used in the right part.
> > I`m thinking of the following solution:
> > There is a getProvider() method, which returns me CalciteJDBC41Connection.
> > And if I have my own implementation of this class then I`ll be able to
> > extend it and enrich the class with methods I need. However, it
> > doesn't feel like a good idea. I believe there might be a proper solution
> > for this.
> >
> >
> > Thanks in advance,
> > Vladislav
> >

Re: Re: Translate join predicates into filters

[Stamatis Zampetakis]

There is also the EnumerableBatchNestedLoopJoin [1] algorithm which
can batch multiple values into one request. There are also some past
discussions in the list around this concept (for instance [2]).

Best,
Stamatis

[1] 
https://github.com/apache/calcite/blob/9c33d7aeefe082bf5f7be617ef231e1285418a6c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableBatchNestedLoopJoinRule.java
[2] https://lists.apache.org/thread/wtdr5lvycfwz6k89kzry73k4tw0lqxbx

On Tue, May 23, 2023 at 1:50 AM Luca Marchi  wrote:
>
> Hi, thanks for the reply!
>
> Right, the nested loop approach iterates each row, this means that we can not 
> push the references as filter of the `book` table in one request, but one by 
> one.
>
> The second approach sounds promising; can you show us a similar example?
> Thanks!
>
> On 2023/05/19 01:14:49 Julian Hyde wrote:
> > I expect that your table works if you put the filter in the WHERE clause, 
> > e.g.
> >
> >   SELECT *
> >   FROM Books AS b
> >   WHERE b.id  IN (1, 10, 27)
> >
> > and it does so using FilterTableScanRule (which matches a Filter on top of 
> > a Scan of a FilterableTable). But you need a new planner rule that can 
> > convert a Join whose right input is a Scan of a FilterableTable into a 
> > NestedLoopsJoin that dynamically sets the filter for each row from the 
> > left. (In this query, config would be on the left, Books on the right.)
> >
> > There could be a more efficient version that gathers all IDs from the left, 
> > then does one request to the right, and them something like a hash join.
> >
> > Julian
> >
> >
> >
> >
> > > On May 17, 2023, at 8:41 AM, Luca Marchi  wrote:
> > >
> > > Morning everyone,
> > > in our company we are running a POC using Apache Calcite, and we would 
> > > like to collect some feedbacks from you for the scenario mentioned below.
> > >
> > > There is a service API that allows retrieving some `Book`s, and we would 
> > > like to build a table adapter on top of this service; this API
> > > only accepts a set of IDs, and if no IDs are provided, no result is 
> > > returned.
> > >
> > > ```
> > >  interface BookService {
> > >/** Returns the books matching the given IDs.
> > > *
> > > * If not IDs is provided, no result is returned.
> > > */
> > >List findBooksByIds(Set ids);
> > >  }
> > >
> > >  record Book(String id, String title) {};
> > > ```
> > >
> > > A requirement of this table is that it has to support join, and we would 
> > > like to support joining by ID in an efficient way.
> > >
> > > The goal is to define a rule that forces the query planner to always push 
> > > down join predicates into a table scan.
> > >
> > > Given the following `book` table:
> > >
> > > ```java
> > > /** A table which represents books, queryable only by their ID. */
> > > final class BookTable extends AbstractTable implements FilterableTable {
> > >  private final BookService service;
> > >
> > >  BookTable(BookService service) {
> > >this.service = service;
> > >  }
> > >
> > >  @Override
> > >  public RelDataType getRowType(RelDataTypeFactory typeFactory) {
> > >return new RelDataTypeFactory.Builder(typeFactory)
> > >.add("id", SqlTypeName.VARCHAR)
> > >.add("title", SqlTypeName.VARCHAR)
> > >.build();
> > >  }
> > >
> > >  @Override
> > >  public Enumerable scan(DataContext root, List 
> > > filters) {
> > >Set bookIds = getBooksId(filters);
> > >List result = service.findBooksByIds(bookIds)
> > >.stream()
> > >.map(b -> new Object[]{b.id, b.title})
> > >.toList();
> > >
> > >return Linq4j.asEnumerable(result);
> > >  }
> > >
> > >  private static Set getBooksId(List filters) {
> > >if (filters.size() != 1) {
> > >  throw new IllegalArgumentException("Expected one filter to the ID, 
> > > found: %d".formatted(filters.size()));
> > >}
> > >
> > >RexNode filter = filters.get(0);
> > >RexNode leftCondition = ((RexCall) filter).getOperands().get(0);
> > >RexNode rightCondition = ((RexCall) filter).getOperands().get(1);
> > >
> > >if (leftCondition instanceof RexInputRef left
> > >&& rightCondition instanceof RexLiteral right
> > >// The index of the ID column is 1.
> > >&& left.getIndex() == 1) {
> > >  if (filter.isA(SqlKind.EQUALS)) {
> > >String bookId = right.getValue2().toString();
> > >return ImmutableSet.of(bookId);
> > >  }
> > >  if (filter.isA(SqlKind.SEARCH)) {
> > >@SuppressWarnings("unchecked")
> > >Sarg searchArguments = right.getValueAs(Sarg.class);
> > >return searchArguments.rangeSet.asRanges().stream()
> > >.map(Range::lowerEndpoint)
> > >.map(NlsString::getValue)
> > >.collect(toSet());
> > >  }
> > >}
> > >throw new IllegalArgumentException("Unexpected operator, found: 
> > > %s".formatted(filter.getKind()));
> > >  }
> > > }
> > > ```
> > >
> > > The API of the 

[jira] [Created] (CALCITE-5705) Generalize RemoveEmptySingleRule to work with arbitrary relations and pruning configurations

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-5705:


 Summary: Generalize RemoveEmptySingleRule to work with arbitrary 
relations and pruning configurations
 Key: CALCITE-5705
 URL: https://issues.apache.org/jira/browse/CALCITE-5705
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
 Fix For: 1.35.0


Currently 
[RemoveEmptySingleRule|https://github.com/apache/calcite/blob/b0b27e8872b33c5ab203e0e2365d267a594c80be/core/src/main/java/org/apache/calcite/rel/rules/PruneEmptyRules.java#LL285C23-L285C44]
 can only transform {{SingleRel}} relations to empty. However, the logic inside 
the {{matches}} method is at the most part capable of handling any kind of 
relation including those that have multiple children.

By generalizing the {{RemoveEmptySingleRule}} to work with arbitrary relations 
we can refactor other pruning rules such as those created by 
[CorrelateLeftEmptyRuleConfig|https://github.com/apache/calcite/blob/b0b27e8872b33c5ab203e0e2365d267a594c80be/core/src/main/java/org/apache/calcite/rel/rules/PruneEmptyRules.java#L588]
 without the need for creating more classes and duplicating code.

Moreover by changing the constructor to accept {{PruneEmptyRule.Config}} 
instead of {{RemoveEmptySingleRuleConfig}} we can reduce code duplication 
further since configurations such as {{ZeroMaxRowsRuleConfig}} and 
{{SortFetchZeroRuleConfig}} could be modified to create instances of 
{{RemoveEmptySingleRule}}.

This is mainly a refactoring to simplify pruning rules and remove duplicate 
logic. The change is fully backwards compatible.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Rewrite rule to convert self-joins into scans

[Stamatis Zampetakis]

Hi Hanumath,

Thanks for putting this doc together; the examples are helpful and I
find the approach interesting.

CTEs and redundant joins have similarities but I see them as different
problems. CTEs are patterns that appear multiple times in a query but
are not necessarily redundant.

Apart from identifying CTEs I am actually pretty interested in how to
represent them. It may be helpful to add some concrete examples/plans
covering the representation part in the document you shared.

A few weeks ago, I created a doc [1] with some preliminary ideas about
CTEs in the context of Hive covering briefly:
* representation
* identification
* logical -> physical for Hive

It would be great if we can converge in a representation that is
useful for multiple projects and built on top of that. The
identification part is somewhat pluggable and it could be different
depending on the use-case.

Best,
Stamatis

[1] 
https://github.com/zabetak/docs/blob/main/2023-03-09-Hive-cbo-cte-optimization/index.md

On Sun, Apr 30, 2023 at 10:08 PM Hanumath Maduri
 wrote:
>
> Dear Calcite Devs,
>
> I would like to start prototyping the approach for the optimization
> mentioned in (CALCITE-5631
> <https://issues.apache.org/jira/browse/CALCITE-5631>). So I just want to
> follow up on the approach mentioned in the above document.
> Please let me know if this approach seems reasonable to all of you.
>
> @Stamatis and @Julian,  Please let me know if you got a chance to take a
> look at the document and are in-line with the approach.
> Any suggestions/clarifications on this approach would be very helpful.
>
> Thanks,
>
>
> On Mon, Apr 24, 2023 at 7:25 PM Hanumath Maduri 
> wrote:
>
> >
> > Thanks Julian and Stamatis for sharing your thoughts on this optimization.
> > I have jotted down a general approach which could optimize the common
> > subexpression with join elimination use cases in the following document.
> > Please go through this
> > <https://docs.google.com/document/d/1rF6sZOipfXe2UdBViSbIF33FMXyz5Ev_s-q4IMe8gEI/edit?usp=sharing>
> > document and let me know what you think of this approach.
> >
> > In gist the approach mentioned in this document is
> > 1. Finding the common subexpressions
> > 2. Using the subexpression information to eliminate the joins when
> > possible.
> >
> > I did think about implementing this optimization in the decorrelator as
> > well, but this approach was not explored further because it could miss
> > finding non correlated common sub expressions.
> >
> > Let me know if you have any questions or suggestions on this approach.
> >
> > Thanks,
> > Hanu
> >
> >
> >
> >
> > On Wed, Apr 19, 2023 at 11:15 AM Julian Hyde 
> > wrote:
> >
> >> Thank you, Stamatis. This is helpful. I have linked to this email thread
> >> in CALCITE-5631.
> >>
> >> > On Apr 16, 2023, at 2:44 AM, Stamatis Zampetakis 
> >> wrote:
> >> >
> >> > Few quick thoughts about this.
> >> >
> >> > For to the problem of query minimization/redundant joins the simpler
> >> > scenarios that I can think of are the following:
> >> >
> >> > # Scenario A
> >> > select e1.id from emp e1 inner join emp e2 on e1.name = e2.name;
> >> >
> >> > If you know the name column is UNIQUE then you can drop the join on e2.
> >> >
> >> > # Scenario B
> >> > select e.name from emp e inner join dept d on e.dept = d.id;
> >> >
> >> > If you know that e.dept and d.id is a foreign key relationship then
> >> > you can drop the join on dept.
> >> >
> >> > There are probably other cases to define and handle but we should move
> >> > incrementally.
> >> >
> >> > As Julian pointed out, the issue logged  in CALITE-5631 could also be
> >> > addressed by employing common table expression related optimizations.
> >> > CTE optimizations and query minimization are both interesting and
> >> > powerful techniques to reduce the cost of the query (whatever that is
> >> > speed, money, resources, etc).
> >> >
> >> > I would suggest focusing on query minimization first since it is
> >> > pretty well defined and we could come up with solutions much faster
> >> > than CTEs. CTEs usually come up with decisions about materializing or
> >> > not the common expressions which are closer to lower level ("physical
> >> > plan") optimizations.
> >> >
> >> > Most minimization techniques focus on select projec

Re: Authentication and authorizations in Calcite

[Stamatis Zampetakis]

Following up on the authorization part lots of things can be achieved
through the use of views. Views can be used to restrict the columns
and rows that are visible to certain users and it is a common way of
handling permissions.

Here also two other JIRA tickets with discussion around this topic:
https://issues.apache.org/jira/browse/CALCITE-2194
https://issues.apache.org/jira/browse/CALCITE-5292

Best,
Stamatis



On Sat, Apr 29, 2023 at 2:11 AM Julian Hyde  wrote:
>
> I think Calcite should stay out of the authentication business. The container 
> (not Calcite) should authenticate users and convert them to security 
> principals that Calcite knows about. (Avatica does authentication [1] but it 
> just delegates to a provider.)
>
> Regarding authorization. I support adding a grants system to Calcite. Grants 
> could be created via DDL (GRANT, REVOKE commands) or via APIs (like interface 
> Schema) and the validator should enforce them. E.g. principal1 can see table1 
> but is not allowed to see its columns column2 and column3.
>
> If you’re interested please log Jira case(s).
>
> Julian
>
> [1] https://issues.apache.org/jira/browse/CALCITE-643
>
>
> > On Apr 26, 2023, at 9:39 AM, Joachim Bloche  wrote:
> >
> > Hi,
> >
> > I'm discovering Calcite after a customer asked me to enhance a proof of 
> > concept they made with it.
> >
> > I could get most things to work as needed and am very impressed with the 
> > possibilities offered.  But completely stuck on a critical point : I can't 
> > find any hint on how to manage authentication and authorization of users - 
> > sorry if information is available and I somehow missed it, but I did my 
> > best to research it.
> >
> > The customer needs to allow access only to certain schemas or tables (or 
> > columns as a bonus) based on user role.
> >
> > Do you know of any standard way to implement this ? Ideally I would need to 
> > implement it as low as possible in the stack as I'd like to use the same 
> > authorization process for both front-end users and users connecting through 
> > the Avatica JDBC driver.
> >
> > With my still limited knowledge of Calcite, the only "off the shelf" 
> > solution I could find is to create one Calcite model per role and start one 
> > Avatica server per model. Obviously I'm not very proud of this one as it's 
> > not really scalable nor elegant and there has to be a better solution.
> >
> > Any hint to point me in the right direction would be hugely appreciated as 
> > I would need the same for future projects.
> >
> >
> >
> > Of course if some development is needed to implement this in Calcite this 
> > could be an opportunity to contribute to the project.
> >
> >
> >
> > Many thanks in advance and best regards,
> >
> >
> >
> > Joachim
> >
> >
>

Re: Employ bloom filters in joins

[Stamatis Zampetakis]

The topic is really interesting, thanks for sharing your ideas Zoltan!

I see no drawbacks adding the new transformation rule; definitely worth
having! However, adding them to the default rule set or using them in a
cost based decision may require much more work/thinking.

Calcite's built-in cost model is pretty basic and does not account for
parallelism / concurrency etc. Any rule that adds more operations to the
plan is gonna make the cost worse so in the end the new plan may never get
selected.

The proposed rewrite rule is really close to the semi-join reduction
technique. I would say that introducing rules and optimizing queries with
semi-join reducers [1] is a good starting point before moving to more
complicated plans with aggregations and specialized UDFs (bloom/cuckoo
etc). Simpler primitives are also more likely to be adopted by
downstream projects.

Regarding the part of constructing and passing the bloom filter around we
may be able to come up with an alternative design without the use of
additional scans/joins by exploiting correlation variables. I haven't
thought this all the way through but binding things on the left side and
passing them to the right side seems something that could be relevant to
this use-case.

Best,
Stamatis

[1] Integrating semi-join-reducers into state-of-the-art query processors (
https://db.in.tum.de/research/publications/conferences/semijoin.pdf)

On Sat, Apr 29, 2023, 3:54 AM Julian Hyde  wrote:

> It would be great to have such a rule. People who don’t want it can
> disable it; and people who enable it can use a cost function.
>
> Some systems that use Bloom filters (and other probabilistic filters)
> don’t execute the query twice but use a side-channel to send the Bloom
> filter from one scan to the other. For example, suppose that the “dept"
> table is smaller and its scan finishes faster. When the scan has finished,
> it sends the Bloom filter to the “emp" scan, which is still under way. From
> that point, the “emp” scan can eliminate a fraction of its rows because it
> knows that their “deptno” values do not pass the filter.
>
> Julian
>
>
> > On Apr 28, 2023, at 9:01 AM, Zoltan Haindrich  wrote:
> >
> > Hi,
> >
> > I was wondering about the pros and cons of having a Calcite rule which
> could rewrite a join to utilize bloom filters; something like:
> >
> > select e.*
> >   from emp e
> >   join dept d on(e.deptno=d.deptno);
> >   where d.dname='Sales';
> >
> > into something like:
> >
> > select e.*
> >   from (
> >   select e.* from emp e join (
> >   select bloom_sketch(deptno) as sketch from
> dept dname='Sales'
> >   ) dept_agg on (bloom_contains(sketch,e.deptno)
> >   ) e
> >   join dept d on(e.deptno=d.deptno)
> >   where d.dname='Sales';
> >
> > Generally for the original query:
> > * if "dept" is very small a mapjoin is used which is great
> > * or possibly some nested loops with index usages on the big table
> > * but if the execution engine decides to use a non-specialized approach
> like merge-join or hash-join; it may move around a lot of data - and in
> those cases this might be usefull
> >
> > There are systems which handle this by introducing a bloom filter (Hive;
> Spark) and transfer that in the background for the big-table readers - but
> that's outside the scope of the planner. I was wondering if it would be
> beneficial or not to introduce such a rule - so that using this can be a
> cost-based decision during planning.
> >
> > pro:
> > * to enable an engine to support this optimization - it would only need
> to implement a few UDFs
> > * the rule could put the use of this optimization under cost-based
> decision
> >
> > con:
> > * an extra scan of the small table
> > * it adds an extra join + aggregate computation
> >  * exec engine will most likely exploit that its just a single row
> > * I guess without proper stats this could even worsen things
> > * it could put more stress on (join) planning - as it could introduce
> more joins
> >
> > What do you guys think?
> >
> > cheers,
> > Zoltan
> >
> >
>
>

[jira] [Created] (CALCITE-5675) Infer predicates for anti-join

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-5675:


 Summary: Infer predicates for anti-join
 Key: CALCITE-5675
 URL: https://issues.apache.org/jira/browse/CALCITE-5675
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


Enhance {{RelMdPredicates}} to be able to infer predicates for anti-joins. 

Consider the following plans with an anti join between EMP and DEPT tables.
+PulledUpPredicates+
{noformat}
LogicalJoin(condition=[=($7, $8)], joinType=[anti])
  LogicalFilter(condition=[=($1, 'Victor')])
LogicalTableScan(table=[[scott, EMP]])
  LogicalFilter(condition=[=($1, 'CSD')])
LogicalTableScan(table=[[scott, DEPT]])
{noformat}
We can infer that the {{>($1, 'Victor')}} predicate holds on the result of the 
join.

+RightInferredPredicates+
{noformat}
LogicalJoin(condition=[=($7, $8)], joinType=[anti])
  LogicalFilter(condition=[>($7, 10)])
LogicalTableScan(table=[[scott, EMP]])
  LogicalTableScan(table=[[scott, DEPT]])
{noformat}
We can infer that the {{>($0, 10)}} predicate holds on the right relation 
(DEPT).





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CALCITE-5669) Remove trivial correlates from the query plan

[Stamatis Zampetakis (Jira)]

Stamatis Zampetakis created CALCITE-5669:


 Summary: Remove trivial correlates from the query plan
 Key: CALCITE-5669
 URL: https://issues.apache.org/jira/browse/CALCITE-5669
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


Consider the following query correlated query.
{code:sql}
select * from emp e where exists (select 1 from dept where empno=null)
{code}
The query basically returns an empty result cause {{empno=null}} is always 
false.

The plan for the query after applying the sub-query remove rule is shown below:
{noformat}
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], 
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
  LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], 
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
LogicalCorrelate(correlation=[$cor0], joinType=[inner], 
requiredColumns=[{0}])
  LogicalTableScan(table=[[CATALOG, SALES, EMP]])
  LogicalAggregate(group=[{0}])
LogicalProject(i=[true])
  LogicalFilter(condition=[=($cor0.EMPNO, null)])
LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
{noformat}
After applying the reduce expressions rule the filter with the correlated 
condition will become false and the resulting plan would be the following.
{noformat}
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], 
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
  LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], 
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
LogicalCorrelate(correlation=[$cor0], joinType=[inner], 
requiredColumns=[{0}])
  LogicalTableScan(table=[[CATALOG, SALES, EMP]])
  LogicalAggregate(group=[{0}])
LogicalProject(i=[true])
  LogicalValues(tuples=[[]])
{noformat}
Observe that now we have a {{LogicalCorrelate}} but there is no real 
correlation in the plan since the correlation variable on the right side 
disappeared. Depending on how rules are applied and which rules are used 
similar "trivial" correlates may appear.

The goal of this ticket is to provide the means to get rid of them.

One option would be to add a new rule (e.g., {{CorrelateToJoinRule}}) which 
detects that a correlate does not have correlations in the right side and turn 
the correlation to a join; then we could employ other existing rules (such as 
PruneEmptyRules) for joins and remove the newly created join altogether.

Another option, would be to introduce new pruning rule(s) for correlate 
(similar to those for joins) that will remove the correlate when its input is 
an empty values expression.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Disable JIRA worklog for GitHub PRs

[Stamatis Zampetakis]

Thanks everyone for the feedback. The change is now merged!

On Wed, Apr 19, 2023 at 4:59 PM Dan Zou  wrote:
>
> +1, it would reduce a lot of noise.
>
> Best,
> Dan Zou
>
>
>
>
>
> > 2023年4月19日 19:37，Benchao Li  写道：
> >
> > +1,
> >
> > One thing that may affect current workflow is sometimes I only watch the
> > Jira, and will get notified from the PR notification. If we are going to
> > disable it, I need to watch the PR too to get the notification for cases
> > that I'm interested in both the Jira discussion and PR comments. But that
> > won't be a big problem for me.
> >
> > Michael Mior  于2023年4月19日周三 18:43写道：
> >
> >> +1 from me as well
> >>
> >> On Wed, Apr 19, 2023, 04:19 Stamatis Zampetakis  wrote:
> >>
> >>> Hello,
> >>>
> >>> Everything that happens in a GitHub PR creates a worklog entry under
> >>> the respective JIRA ticket.
> >>> For every worklog entry we receive a notification from j...@apache.org
> >>> when we are watching an issue. The worklog entry and email
> >>> notification usually appear messy.
> >>>
> >>> Moreover, if we are watching the GitHub PR we are going to get a
> >>> notification from notificati...@github.com which has the same content
> >>> with the JIRA worklog entry and is much more readable.
> >>>
> >>> Finally, the PR notification is also going to
> >>> comm...@calcite.apache.org so those who are subscribed to that list
> >>> will get the same notification three times.
> >>>
> >>> Personally, I never read the JIRA worklog notifications and I largely
> >>> prefer those from notificati...@github.com.
> >>>
> >>> How do you feel about disabling the worklog entries in JIRA coming
> >>> from GitHub PRs?
> >>>
> >>> For archiving purposes, the notifications already go to commits@ so we
> >>> don't lose anything from disabling the worklog entries. On the
> >>> contrary, I find that this would reduce the noise and redundancy on
> >>> our inboxes.
> >>>
> >>> Concretely this is what I have in mind in terms of change:
> >>> https://github.com/apache/calcite/pull/3166
> >>>
> >>> Best,
> >>> Stamatis
> >>>
> >>
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>

[DISCUSS] Disable JIRA worklog for GitHub PRs

[Stamatis Zampetakis]

Hello,

Everything that happens in a GitHub PR creates a worklog entry under
the respective JIRA ticket.
For every worklog entry we receive a notification from j...@apache.org
when we are watching an issue. The worklog entry and email
notification usually appear messy.

Moreover, if we are watching the GitHub PR we are going to get a
notification from notificati...@github.com which has the same content
with the JIRA worklog entry and is much more readable.

Finally, the PR notification is also going to
comm...@calcite.apache.org so those who are subscribed to that list
will get the same notification three times.

Personally, I never read the JIRA worklog notifications and I largely
prefer those from notificati...@github.com.

How do you feel about disabling the worklog entries in JIRA coming
from GitHub PRs?

For archiving purposes, the notifications already go to commits@ so we
don't lose anything from disabling the worklog entries. On the
contrary, I find that this would reduce the noise and redundancy on
our inboxes.

Concretely this is what I have in mind in terms of change:
https://github.com/apache/calcite/pull/3166

Best,
Stamatis

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[Stamatis Zampetakis]

Hey Mihai,

Thanks for starting this discussion!

Let's focus on the first question for now:

Q1: Should the new slt module under PR-3145 [1] become part of Calcite
repo or get its own?

For those who have not followed the discussion under the CALCITE-5615
[2] let me try to summarize a few things as per my understanding;
Mihai can amend/correct things if necessary.

The new slt module resembles a port of sqllogictest utility [3] to
Java. It can parse and understand the test-script format used in
sqllogictest and can run this scripts over JDBC compliant databases.
It also accounts for extensions for Java engines without a JDBC
interface.

>From my perspective, the code in [1] could perfectly stand on its own
in a separate repo; there are already ports of sqllogictest in other
languages such as Rust [4] and the latter appears to be quite popular.
The sqllocitest parser/runner presents some similarities with the
Quidem [5] executor that we are using for certain tests in Calcite.
The Quidem project has its own repo although we are making use of it
in Calcite.
If it becomes a separate repo then the test scripts could also become
part of the project making it more self-contained.

On the other hand, we already have a testkit module in Calcite so
bringing in new modules for testing purposes is relevant so why not
slt as well. If it becomes part of Calcite it can get more visibility
and facilitate maintenance since more people would be able to review
and merge changes (not only Mihai).

Since we are talking about a new module I would like to see some more
people share their opinion on the topic before I continue the review.

Best,
Stamatis

[1] https://github.com/apache/calcite/pull/3145
[2] https://issues.apache.org/jira/browse/CALCITE-5615
[3] https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
[4] https://github.com/risinglightdb/sqllogictest-rs
[5] https://github.com/julianhyde/quidem



On Sat, Apr 15, 2023 at 11:31 AM Michael Mior  wrote:
>
> Very cool! One approach could be to add set these tests to run periodically
> (daily/weekly) as opposed to being part of the CI pipeline. That way we
> still have a mechanism to keep tabs on bugs but the whole build isn't
> slow/broken until this is fixed.
>
> On Fri, Apr 14, 2023, 15:20 Mihai Budiu  wrote:
>
> > Hello all,
> >
> > I have submitted a PR for Calcite with a standalone executable that runs
> > the Sql Logic Test suite of 7+ million tests from sqlite.
> >
> > This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
> > And this is the PR: https://github.com/apache/calcite/pull/3145
> >
> > As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
> > general framework in Java to run these tests on any JDBC compliant
> > executor. So a question is whether this belongs to the Calcite project, or
> > some place else. sqlite is a C project, I didn't see any Java in their
> > source tree.
> >
> > Please note that SQLite is in the public domain, so their licensing terms
> > are not an obstacle to using the test scripts.
> >
> > The submitted code runs Calcite in its default configuration, but the
> > intent is for other projects that build Calcite-based compilers to be able
> > to test them by subclassing the "TestExecutors". In our own project (
> > https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
> > and we are not using the JDBC API.
> >
> > The testsuite does find bugs in Calcite, both crashes and incorrect
> > results. So I think it's usefulness is not debated.
> >
> > The second question is about the packaging of this program; right now it
> > has a main() entry point and it prints the results to stderr for human
> > consumption and triage. It is not clear to me how it should be inserted in
> > a CI infrastructure, since running all 7 million tests could take a long
> > time. One possible extension would be to have the program generate a
> > regression test for Calcite for each bug it finds, but I haven't
> > implemented this feature yet (and many failures could be due to the same
> > bug). But even that mode would not naturally integrate in a CI
> > infrastructure.
> >
> > A simple possibility is for me to just publish the code as an independent
> > project on github with an MIT license (the code is derived from our
> > MIT-licensed project) and just advertise it to the Calcite community.
> >
> > I would very much appreciate guidance.
> >
> > Mihai Budiu
> >

Re: Rewrite rule to convert self-joins into scans

[Stamatis Zampetakis]

Few quick thoughts about this.

For to the problem of query minimization/redundant joins the simpler
scenarios that I can think of are the following:

# Scenario A
select e1.id from emp e1 inner join emp e2 on e1.name = e2.name;

If you know the name column is UNIQUE then you can drop the join on e2.

# Scenario B
select e.name from emp e inner join dept d on e.dept = d.id;

If you know that e.dept and d.id is a foreign key relationship then
you can drop the join on dept.

There are probably other cases to define and handle but we should move
incrementally.

As Julian pointed out, the issue logged  in CALITE-5631 could also be
addressed by employing common table expression related optimizations.
CTE optimizations and query minimization are both interesting and
powerful techniques to reduce the cost of the query (whatever that is
speed, money, resources, etc).

I would suggest focusing on query minimization first since it is
pretty well defined and we could come up with solutions much faster
than CTEs. CTEs usually come up with decisions about materializing or
not the common expressions which are closer to lower level ("physical
plan") optimizations.

Most minimization techniques focus on select project join (SPJ)
queries so I guess we would have to do some preprocessing to bring the
plan in this format (only Scan, Project, Filter, and Join operators)
before applying the rule. It would be a separate planning phase
combining a bunch of existing rules followed by some new which is
inline with what Julian was saying about bottom-up unification.

The new rule could be something similar to LoptOptimizeJoinRule that
operates on a MultiJoin. I haven't checked if the MultiJoin operator
is sufficient to express an SPJ query but I think the general idea of
grouping joins together seems to be a promising direction for writing
new rules.

Best,
Stamatis

On Sun, Apr 16, 2023 at 2:27 AM Julian Hyde  wrote:
>
> Ian Bertolacci recently logged
> https://issues.apache.org/jira/browse/CALCITE-5631, to convert
>
>   select
>  (select numarrayagg(C5633_203) from T893 where C5633_586 = T895.id),
>  (select numarrayagg(C5633_170) from T893 where C5633_586 = T895.id)
>   from T895
>
> into
>
>   select agg.agg1,
>   agg.agg2
>   from T895
>   left join (
> select C5633_586,
> numarrayagg(C5633_203) as agg1,
> numarrayagg(C5633_170) as agg2
> from T893
> where C5633_586 is not null
> group by C5633_586) as agg
>   on agg.C5633_586 = T895.id
>
> This seems to me an interesting and important problem. But it's also a
> hard problem, and it's not clear to me which approach is the best.
> Does anyone have any ideas for how to approach it?
>
> Also, we could use more example queries that illustrate the general
> pattern.  (Preferably in terms of simple databases such as EMP and
> DEPT.)
>
> In Calcite rewrite rules (RelRule) are usually the preferred approach.
> Because the common relational expressions scans can be an arbitrary
> distance apart in the RelNode tree, RelRule doesn't seem suitable.
>
> There seem to be some similarities to algorithms to use materialized
> views, which use bottom-up unification.
>
> Ian's original query actually has correlated scalar sub-queries rather
> than explicit joins. Would it be better to target common sub-queries
> rather than joins?
>
> Lastly, there are similarities with the WinMagic algorithm, which
> converts correlated sub-queries into window aggregates. Is that a
> useful direction? (My implementation of measures in CALCITE-4496
> naturally creates correlated scalar sub-queries that can be inlined in
> the enclosing query if simple, or converted to window aggregates if
> more complex.)
>
> Julian

Re: [DISCUSS] Sharing the load of reviewing PRs

[Stamatis Zampetakis]

For a long time this has been one of the main issues of the project
and I am happy to see discussions to address this issue.

I would like to mention that as a contributor, I am, and always have
been very grateful to people reviewing my work.
The fact that I became a committer of this project is mainly due to
Julian and Vladimir Sitnikov who reviewed and merged many of my PRs.
I would definitely like to help and make other contributors feel the
same but I cannot really commit to specific volume and deadlines
spanning several months.

I have the feeling that we don't need the PR manager role. The
assignment work can be done by bots (e.g., [1]) if needed and we
already have our quarterly stats for reporting purposes.
If we want to put a human behind this then it makes more sense for
this person to be the release manager; this should be the one nagging
people for advancing the work and moving towards the release.

Regarding reviews coming from non-committers, I am not sure it's
possible to do the assignment in GitHub. It's not a big deal though;
for me a simple comment that I am going to review would be sufficient.
Alternatively, we could consider adopting an equivalent workflow in
JIRA and potentially introducing a new state "IN REVIEW"; don't think
it is necessary.
No matter the choice we should ensure that we have a trackable way to
recognise "non-committer" reviewers but I think both GitHub (e.g.,
"is:pr reviewed-by:julianhyde is:closed") and JIRA offer the necessary
filters;
Others projects tend to include such info in the commit message we
could also opt for this if we deem necessary.

As an immediate action I would encourage everyone willing to help to
go to the open PRs on GitHub and either assign some PRs to themselves
(in case of committers) or leave a comment about their intention
(non-committers).

In the meantime we can iterate on this till we reach consensus.

Best,
Stamatis

[1] https://github.com/apps/auto-assign


On Wed, Apr 12, 2023 at 10:49 AM Ruben Q L  wrote:
>
> Hello,
>
> I understand Julian's frustration. We all know that reviewing PRs is a
> recurring problem, and it is not the first time we discuss potential
> solutions, see e.g. the discussion a year ago [1] (also started by Julian)
> where several ideas were mentioned: automatic assignment, emulate the RM
> process onto the reviewing process (quite similar to the current proposal),
> ...but in the end no change was implemented, and the problem remains.
>
> I agree that something must be done in order to revert the situation.
>
> In my opinion, one of the main factors is that the vast majority of PRs
> (even the ones that get merged) are never assigned. This lack of assignee
> can be seen as if the PR is "no-one's responsibility", so we should try
> somehow to assign each PR to someone, and make that person accountable for
> the PR's progression.
> I think we could try Julian's idea of having a pool of reviewers and a PR
> manager (taken from the pool, rotating this position every month or every
> two months). Personally, I would not set hard deadlines (e.g. something
> must be done within 3 days), because we are all volunteers and, even if we
> are all trying to do our best here, it may happen that a certain week we
> are busy with other personal or professional stuff. In the end, I think it
> should be the PR manager's responsibility to ping the assigned reviewer if
> a PR is not progressing after a reasonable period of time, ask them for an
> update, maybe even involve a different reviewer / re-assign the PR as a
> last resource.
>
> Of course, it must remain clear that, even if we implement this approach,
> anyone is still free (and encouraged) to participate in any PR review. Even
> if someone is not the assigned reviewer, they can chime in and contribute
> to the review nevertheless.
> Also, I think another sensible rule would be: if someone from the reviewers
> pool submits a PR, the PR manager will need to assign it to a different
> person.
>
> One last comment, I have the impression that with this initiative we would
> be moving towards a "better done than perfect" approach. Calcite is a vast
> project, with many different modules, and it could happen (it *will*
> happen) that a certain reviewer gets assigned a PR concerning a part of the
> project that they are not familiar with. Of course, one way of becoming
> (progressively) familiar with unknown parts of the project is by reviewing
> this type of PRs, but that takes time. I guess it would be possible for the
> assignee to try to ping and involve other reviewers with more experience in
> that area, but at the end of the day, it would be the assignee's
> responsibility to review and merge some piece of code that might be a bit
> obscure to them. This might lead to suboptimal or even incorrect code being
> inadvertently merged into the main branch. This is not a catastrophe (it
> can already happen with the current approach), and we will detect and
> correct these 

Draft: board report for 2023 Q1

[Stamatis Zampetakis]

Hello,

Below you can find a draft of this quarter's board report. I plan to
submit it next Tuesday (April 11, 2023).
Please let me know if you have any additions or corrections.

Best regards,
Stamatis
-

## Description:
Apache Calcite is a highly customizable framework for parsing and planning
queries on data in a wide variety of formats. It allows database-like
access,
and in particular a SQL interface and advanced query optimization, for data
not residing in a traditional database.

Avatica is a sub-project within Calcite and provides a framework for
building
local and remote JDBC and ODBC database drivers. Avatica has an independent
release schedule and its own repository.

## Issues:
There are no issues requiring board attention.

## Membership Data:
Apache Calcite was founded 2015-10-22 (7 years ago)
There are currently 63 committers and 27 PMC members in this project.
The Committer-to-PMC ratio is 7:3.

Community changes, past quarter:
- Benchao Li was added to the PMC on 2023-01-27
- Alex Plehanov was added as committer on 2023-01-06
- Jiajun Xie was added as committer on 2023-02-10
- Istvan Toth was added as committer on 2023-01-25

## Project Activity:
Apache Calcite 1.34.0 was released on 2023-03-14. It contains contributions
from 18 contributors, and resolves 34 issues. It’s worth highlighting the
introduction of QUALIFY clause (CALCITE-5268), which facilitates filtering the
results of window functions. Among other improvements and fixes, it adds
roughly 15 new functions in BigQuery library for handling dates, times, and
timestamps.

Apache Calcite 1.33.0 was released on 2023-02-06. It contains contributions
from 33 contributors, and resolves 107 issues. It’s worth highlighting the
support for custom time frames (CALCITE-5155), the new MEASURE type and
AGGREGATE aggregate function (CALCITE-5105) as well as the many improvements
to the BigQuery dialect (CALCITE-5180).

Apache Calcite Avatica 1.23.0 was released on 2023-01-19. It fixes bugs in
Statement.getUpdateCount(), ResultSet.getObject; and supports HTTP_BAD_REQUEST
and configuring fetch size and SSL key-store type. Also, there are various
improvements to DateTimeUtils and ByteString.

On 2023-03-15, there was an online meetup of the Calcite community with
approximately 50 participants joining the call. We had three very interesting
presentations around adding measures in SQL, incremental view maintenance for
streaming engines, and debugging planner issues, followed by open discussion.
The videos from the meetup as well as the slides were published online shortly
after the event.

## Community Health:
The project remains super healthy and there is a general increase in traffic
almost in every aspect of the project (dev@, JIRA, and GitHub).

The most notable increase in traffic was for dev mailing list (46%). One
factor that led to this increase is the addition of four new members in the
project and the traditional welcoming emails that usually come along. Another
one, would probably be the various discussion/votes around releases; we had
more this quarter compared to our usual cadence. Last there were also some
changes in the CI (SonarCloud intergration) that sparked some additional
exchanges.

The number of non-committer (contributor) commits per month:
+-+-+-+
|year |month| contributor_commits |
+-+-+-+
| 2023| 1   | 20  |
| 2023| 2   | 15  |
| 2023| 3   | 17  |
+-+-+-+

The number of active reviewers per month:
+-+-+-+
|year |month|  active_reviewers   |
+-+-+-+
| 2023| 1   | 4   |
| 2023| 2   | 5   |
| 2023| 3   | 4   |
+-+-+-+

Top-3 reviewers in the last quarter:
+---+-+
| committer |   reviews   |
+---+-+
| Julian Hyde  | 34  |
| Benchao Li  | 9   |
| Stamatis Zampetakis  | 4   |
+---+-+

The number of non-commiter commits has almost doubled in the last quarter
which is a good thing. Unfortunately the number of active reviewers is
still pretty low and few individuals namely Julian Hyde, and Benchao Li
are pulling almost all the weight of reviewing contributors work.

Re: CI requiring approval for external contributors

[Stamatis Zampetakis]

It's true that I've never seen a fake PR so far in the ASF repos but
at the same time I am not following every single PR.
I can understand the approval for first time contributors but keeping
the approval for everyone seems a bit too much.

Best,
Stamatis

On Wed, Mar 29, 2023 at 5:04 AM 王鹏  wrote:
>
> +1 for this. I think it's very friendly to create PR for contributors.
>
> Dan Zou  于2023年3月29日周三 10:13写道：
>
> > +1 for this, it is something out of my expectation when I created my first
> > PR in calcite and informed that approval is required.
> >
> > Best,
> > Dan Zou
> >
> >
> >
> >
> >
> > > 2023年3月28日 21:37，Benchao Li  写道：
> > >
> > > I'm +1 for changing it back.
> > >
> > > I was also thinking about starting a discussion about this before, but I
> > > didn't know how to achieve it. Thanks Julian for sharing this with us.
> > >
> > > Julian Hyde  于2023年3月28日周二 20:57写道：
> > >
> > >> (Forwarding from the Druid list for discussion.)
> > >>
> > >> Julian
> > >>
> > >> Begin forwarded message:
> > >>
> > >>> From: Gian Merlino 
> > >>> Date: March 28, 2023 at 1:24:42 AM CDT
> > >>> To: d...@druid.apache.org
> > >>> Subject: CI requiring approval for external contributors
> > >>> Reply-To: d...@druid.apache.org
> > >>>
> > >>> Recently, ASF GitHub repos had their defaults for GitHub Actions
> > >> changed to
> > >>> "always require approval for external contributors". In Slack, Karan
> > >>> pointed out that Airflow has recently submitted a ticket to have that
> > >>> changed back: https://issues.apache.org/jira/browse/INFRA-24200. IMO,
> > we
> > >>> should do the same. I don't think we have a problem with fake PRs, but
> > we
> > >>> can always improve our responsiveness to contributors from outside the
> > >>> project! Every little bit helps, including running CI automatically.
> > >>>
> > >>> If others have opinions on this, let me know. I'd like to raise our own
> > >>> ticket to change our default.
> > >>>
> > >>> Gian
> > >>
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> >
> >

Re: The question about the Apache Calcite 2.0's progress

[Stamatis Zampetakis]

Hi LakeShen,

The last discussion about this topic was back in 2021 [1].

Apart from cleanup there is nothing major to be done as part of 2.0 so
there is no big rush to get it out.

Best,
Stamatis

[1] https://lists.apache.org/thread/p6svy4hps193g4knhsksv7psx0osgvq7

On Wed, Mar 29, 2023 at 8:51 AM LakeShen  wrote:
>
> Hi community,
>
> Now I am reading the Calcite source code,and I find that there are a lot of
> code comments  which are marked to be removed before 2.0.What is the status
> of Apache Calcite 2.0？Where can I see Apache Calcite 2.0 progress?
>
> Best,
> LakeShen.

Re: Rules to squash redundant join sides?

[Stamatis Zampetakis]

Hey Ian,

I think you are referring to the problem of query minimization and at
the moment we don't have any such rules in Calcite but it would be a
valuable contribution.
Apart from the algorithm itself, it might be necessary to introduce
some new metadata provider for PK-FK relationships otherwise dropping
a join may remove duplicates and change the semantics of the plan.

You can find some previous discussion here [1].

Best,
Stamatis

[1] https://lists.apache.org/thread/vsq59yhfj4glf7mgpf9n6j255myhs0so

On Thu, Mar 30, 2023 at 1:27 AM Ian Bertolacci
 wrote:
>
> Howdy,
> Is there a collection of rules which squash a tree of binary joins if the 
> same side of each join is mergeable?
>
> For example:
> ```
> 201:LogicalProject(P4=[$70], P5=[$72])
> └─ 199:LogicalJoin(condition=[=($0, $71)], joinType=[left])
>├─ 190:LogicalProject(...)
>|  └─ 188:LogicalJoin(condition=[=($0, $70)], joinType=[left])
>| ├─ 164:QueryTableScan(table=[[Query, T123]])
>| └─ 186:LogicalAggregate(group=[{0}], EXPR$0=[ARRAY_AGG($1)])
>|└─ 184:LogicalProject(C5633_586=[$85], C5633_170=[$12])
>|   └─ 182:LogicalFilter(condition=[IS_NOT_NULL($85)])
>|  └─ 165:QueryTableScan(table=[[QUERY, T893]])
>└─ 197:LogicalAggregate(group=[{0}], EXPR$0=[ARRAY_AGG($1)])
>   └─ 195:LogicalProject(C5633_586=[$85], C5633_203=[$45])
>  └─ 193:LogicalFilter(condition=[IS_NOT_NULL($85)])
> └─ 172:QueryTableScan(table=[[QUERY, T893]])
> ```
>
> Can be simplified to one join as something like:
> ```
> 201:LogicalProject(P4=[$lhs+1)], P5=[$lhs+2])
> └─ 188:LogicalJoin(condition=[=($0, $70)], joinType=[left])
>├─ 164:QueryTableScan(table=[[Query, T123]])
>└─ 186:LogicalAggregate(group=[{0}], EXPR$0=[ARRAY_AGG($1), 
> EXPR$0=[ARRAY_AGG($2)])
>   └─ 184:LogicalProject(C5633_586=[$85], C5633_170=[$12], C5633_203=[$45])
>  └─ 182:LogicalFilter(condition=[IS_NOT_NULL($85)])
> └─ 165:QueryTableScan(table=[[QUERY, T893]])
> ```
>
>
> I spent some time trying applying various CoreRules, but didn’t immediately 
> see anything I wanted.
> I figure that there exists some set of existing rules which when applied 
> together would accomplish what we want here.
>
> The joins arise from correlated subqueries, but are also synthetically 
> generated, so it is not as simple as asking the user to manually do a single 
> join.
>
> Thanks!
> -Ian Bertolacci

Re: [DISCUSS] Calcite Release Managers

[Stamatis Zampetakis]

Thanks Sergey and Benchao for volunteering.

So far we have:

1.35.0 Duan Xiong
1.36.0 Benchao Li
1.37.0 Sergey Nuyanzin
1.38.0
1.39.0

Regarding the timing we would like to release every 2-3 months but the
exact dates are most of the time flexible. It's up to the release manager
to start the "Towards Calcite 1.X.Y" thread and propose the dates more
convenient to them.

Best,
Stamatis

On Tue, Mar 21, 2023, 10:52 AM Sergey Nuyanzin  wrote:

> Thanks for the starting discussion Stamatis,
>
> depending on timeline of releases I would volunteer to help here with one
> of them
>
> On Tue, Mar 21, 2023 at 10:32 AM Benchao Li  wrote:
>
> > As I've said in another ML thread[1], I will volunteer to do one.
> >
> > [1] https://lists.apache.org/thread/dcf6o3mdzflh7brsklgoy7fd1njy8jjz
> >
> > Stamatis Zampetakis  于2023年3月21日周二 17:01写道：
> >
> > > Hi everyone,
> > >
> > > In the last year or so we had the following people helping out with
> > > releasing Calcite.
> > >
> > > 1.30.0 Liya Fan
> > > 1.31.0 Adrei Sereda
> > > 1.32.0 Julian Hyde
> > > 1.33.0 Jess Balint
> > > 1.34.0 Stamatis Zampetakis
> > >
> > > Big thanks to everyone!
> > >
> > > Now we should see who can help out with the next releases.
> > >
> > > @Duan Xiong: Are you still available for 1.35.0?
> > > Any other volunteers for 1.36.0 to 1.39.0?
> > >
> > > 1.35.0 Duan Xiong
> > > 1.36.0
> > > 1.37.0
> > > 1.38.0
> > > 1.39.0
> > >
> > > Best,
> > > Stamatis
> > >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>
>
> --
> Best regards,
> Sergey
>

[DISCUSS] Calcite Release Managers

[Stamatis Zampetakis]

Hi everyone,

In the last year or so we had the following people helping out with
releasing Calcite.

1.30.0 Liya Fan
1.31.0 Adrei Sereda
1.32.0 Julian Hyde
1.33.0 Jess Balint
1.34.0 Stamatis Zampetakis

Big thanks to everyone!

Now we should see who can help out with the next releases.

@Duan Xiong: Are you still available for 1.35.0?
Any other volunteers for 1.36.0 to 1.39.0?

1.35.0 Duan Xiong
1.36.0
1.37.0
1.38.0
1.39.0

Best,
Stamatis

Re: [DISCUSS] Apache Calcite Meetup March 2023

[Stamatis Zampetakis]

All the slides and videos from the meetup are now online. The links
for the talks can be found on the website [1].

Best,
Stamatis

[1] https://calcite.apache.org/community/#talks

On Thu, Mar 16, 2023 at 11:28 AM Stamatis Zampetakis  wrote:
>
> I would like to thank everyone who attended, supported, and helped
> organize this meetup with of-course a special mention for our
> presenters Julian, Mihai, and Alessandro!
>
> I had a great time yesterday evening and learned many new things along
> the way. From some personal feedback I got, I know that there were
> many people who really appreciated the presentations and the open
> discussion.
>
> If there is more feedback about improvements, changes, or things that
> you would like to see in a next meetup don't hesitate to share here or
> privately.
>
> I will post the videos online on youtube once I finish some small
> post-processing.
> Then I will update the talks section in the website [1] with links to
> the slides, videos, and other relevant information.
>
> Best,
> Stamatis
>
> [1] https://calcite.apache.org/community/#talks
>
> On Wed, Mar 15, 2023 at 6:08 PM Stamatis Zampetakis  wrote:
> >
> > The meetup starts in ~50minutes from now.
> >
> > You can join by clicking on the following link:
> > https://cloudera.zoom.us/j/99307471076?pwd=NS8wWWdocHVDWktXaTVCdUdYY3ZiUT09
> >
> > Best,
> > Stamatis
> >
> > On Tue, Mar 14, 2023 at 1:39 AM Stamatis Zampetakis  
> > wrote:
> > >
> > > Hi all,
> > >
> > > The link for the zoom meeting is now available in the event details on 
> > > meetup [1] and will share it also in this thread a few hours before the 
> > > event starts.
> > >
> > > The event starts on March 15, 2023 18:00 UTC [2].
> > >
> > > Note that there is a bug in meetup.com and some pages display a wrong 
> > > time for the event (problems observed for people in CET); if in doubt use 
> > > [2] as the source of truth.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] 
> > > https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
> > > [2] 
> > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=Apache+Calcite+meetup=20230315T18
> > >
> > > On Wed, Feb 8, 2023 at 5:17 PM Stamatis Zampetakis  
> > > wrote:
> > >>
> > >> Thanks Mihai for proposing to give a talk! Looking forward to it.
> > >>
> > >> I just created and announced the event on meetup [1]. The agenda may 
> > >> change but hopefully the date/time will remain the same.
> > >>
> > >> Apriori we will use a regular Zoom meeting to host the event (like last 
> > >> time) but if people have other preferences let me know. I will share the 
> > >> respective link here and in the meetup platform as we approach the date.
> > >>
> > >> Best,
> > >> Stamatis
> > >>
> > >> [1] 
> > >> https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
> > >>
> > >> On Tue, Feb 7, 2023 at 8:46 PM Julian Hyde  
> > >> wrote:
> > >>>
> > >>> That would be very interesting. (I see Frank McSherry among your 
> > >>> co-authors and everything Frank does is very interesting.)
> > >>>
> > >>>
> > >>> > On Feb 7, 2023, at 11:13 AM, Mihai Budiu  
> > >>> > wrote:
> > >>> >
> > >>> > Hello,
> > >>> >
> > >>> > I would like to propose the following presentation for a meetup.
> > >>> >
> > >>> > Thank you,
> > >>> > Mihai Budiu
> > >>> >
> > >>> > -
> > >>> >
> > >>> > Building a streaming incremental view maintenance engine with Calcite
> > >>> >
> > >>> > Speaker: Mihai Budiu - VMware Research
> > >>> >
> > >>> > Duration: 20-40 minutes
> > >>> >
> > >>> > The DBSP open-source project
> > >>> > (https://github.com/vmware/database-stream-processor) is a Rust
> > >>> > runtime that unifies seamlessly streaming queries and incremental view
> > >>> > maintenance (IVM).  The SQL to DBSP compiler is an open-source
> > >>> >

Re: [DISCUSS] Apache Calcite Meetup March 2023

[Stamatis Zampetakis]

I would like to thank everyone who attended, supported, and helped
organize this meetup with of-course a special mention for our
presenters Julian, Mihai, and Alessandro!

I had a great time yesterday evening and learned many new things along
the way. From some personal feedback I got, I know that there were
many people who really appreciated the presentations and the open
discussion.

If there is more feedback about improvements, changes, or things that
you would like to see in a next meetup don't hesitate to share here or
privately.

I will post the videos online on youtube once I finish some small
post-processing.
Then I will update the talks section in the website [1] with links to
the slides, videos, and other relevant information.

Best,
Stamatis

[1] https://calcite.apache.org/community/#talks

On Wed, Mar 15, 2023 at 6:08 PM Stamatis Zampetakis  wrote:
>
> The meetup starts in ~50minutes from now.
>
> You can join by clicking on the following link:
> https://cloudera.zoom.us/j/99307471076?pwd=NS8wWWdocHVDWktXaTVCdUdYY3ZiUT09
>
> Best,
> Stamatis
>
> On Tue, Mar 14, 2023 at 1:39 AM Stamatis Zampetakis  wrote:
> >
> > Hi all,
> >
> > The link for the zoom meeting is now available in the event details on 
> > meetup [1] and will share it also in this thread a few hours before the 
> > event starts.
> >
> > The event starts on March 15, 2023 18:00 UTC [2].
> >
> > Note that there is a bug in meetup.com and some pages display a wrong time 
> > for the event (problems observed for people in CET); if in doubt use [2] as 
> > the source of truth.
> >
> > Best,
> > Stamatis
> >
> > [1] 
> > https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
> > [2] 
> > https://www.timeanddate.com/worldclock/fixedtime.html?msg=Apache+Calcite+meetup=20230315T18
> >
> > On Wed, Feb 8, 2023 at 5:17 PM Stamatis Zampetakis  
> > wrote:
> >>
> >> Thanks Mihai for proposing to give a talk! Looking forward to it.
> >>
> >> I just created and announced the event on meetup [1]. The agenda may 
> >> change but hopefully the date/time will remain the same.
> >>
> >> Apriori we will use a regular Zoom meeting to host the event (like last 
> >> time) but if people have other preferences let me know. I will share the 
> >> respective link here and in the meetup platform as we approach the date.
> >>
> >> Best,
> >> Stamatis
> >>
> >> [1] 
> >> https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
> >>
> >> On Tue, Feb 7, 2023 at 8:46 PM Julian Hyde  wrote:
> >>>
> >>> That would be very interesting. (I see Frank McSherry among your 
> >>> co-authors and everything Frank does is very interesting.)
> >>>
> >>>
> >>> > On Feb 7, 2023, at 11:13 AM, Mihai Budiu  
> >>> > wrote:
> >>> >
> >>> > Hello,
> >>> >
> >>> > I would like to propose the following presentation for a meetup.
> >>> >
> >>> > Thank you,
> >>> > Mihai Budiu
> >>> >
> >>> > -
> >>> >
> >>> > Building a streaming incremental view maintenance engine with Calcite
> >>> >
> >>> > Speaker: Mihai Budiu - VMware Research
> >>> >
> >>> > Duration: 20-40 minutes
> >>> >
> >>> > The DBSP open-source project
> >>> > (https://github.com/vmware/database-stream-processor) is a Rust
> >>> > runtime that unifies seamlessly streaming queries and incremental view
> >>> > maintenance (IVM).  The SQL to DBSP compiler is an open-source
> >>> > (https://github.com/vmware/sql-to-dbsp-compiler) SQL compiler, based
> >>> > on Calcite, that targets the DBSP runtime.  The compiler has passed
> >>> > more than 7 million SQL tests from SQLLogicTest (with 3 tests
> >>> > failing).  We will present the core design of the IVM engine and the
> >>> > way we use Calcite to compile SQL to streaming computations.  We
> >>> > discuss a few syntactic and sematic differences between SQL dialects
> >>> > that we have identified.  Should Calcite support a notion of an "input
> >>> > SQL dialect", that would allow it to emulate behaviors that differ
> >>> > between other established SQL engines?
> >>&

Re: [DISCUSS] Apache Calcite Meetup March 2023

[Stamatis Zampetakis]

The meetup starts in ~50minutes from now.

You can join by clicking on the following link:
https://cloudera.zoom.us/j/99307471076?pwd=NS8wWWdocHVDWktXaTVCdUdYY3ZiUT09

Best,
Stamatis

On Tue, Mar 14, 2023 at 1:39 AM Stamatis Zampetakis  wrote:
>
> Hi all,
>
> The link for the zoom meeting is now available in the event details on meetup 
> [1] and will share it also in this thread a few hours before the event starts.
>
> The event starts on March 15, 2023 18:00 UTC [2].
>
> Note that there is a bug in meetup.com and some pages display a wrong time 
> for the event (problems observed for people in CET); if in doubt use [2] as 
> the source of truth.
>
> Best,
> Stamatis
>
> [1] 
> https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
> [2] 
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=Apache+Calcite+meetup=20230315T18
>
> On Wed, Feb 8, 2023 at 5:17 PM Stamatis Zampetakis  wrote:
>>
>> Thanks Mihai for proposing to give a talk! Looking forward to it.
>>
>> I just created and announced the event on meetup [1]. The agenda may change 
>> but hopefully the date/time will remain the same.
>>
>> Apriori we will use a regular Zoom meeting to host the event (like last 
>> time) but if people have other preferences let me know. I will share the 
>> respective link here and in the meetup platform as we approach the date.
>>
>> Best,
>> Stamatis
>>
>> [1] 
>> https://www.meetup.com/apache-calcite/events/291489488?utm_medium=referral_campaign=share-btn_savedevents_share_modal_source=link
>>
>> On Tue, Feb 7, 2023 at 8:46 PM Julian Hyde  wrote:
>>>
>>> That would be very interesting. (I see Frank McSherry among your co-authors 
>>> and everything Frank does is very interesting.)
>>>
>>>
>>> > On Feb 7, 2023, at 11:13 AM, Mihai Budiu  
>>> > wrote:
>>> >
>>> > Hello,
>>> >
>>> > I would like to propose the following presentation for a meetup.
>>> >
>>> > Thank you,
>>> > Mihai Budiu
>>> >
>>> > -
>>> >
>>> > Building a streaming incremental view maintenance engine with Calcite
>>> >
>>> > Speaker: Mihai Budiu - VMware Research
>>> >
>>> > Duration: 20-40 minutes
>>> >
>>> > The DBSP open-source project
>>> > (https://github.com/vmware/database-stream-processor) is a Rust
>>> > runtime that unifies seamlessly streaming queries and incremental view
>>> > maintenance (IVM).  The SQL to DBSP compiler is an open-source
>>> > (https://github.com/vmware/sql-to-dbsp-compiler) SQL compiler, based
>>> > on Calcite, that targets the DBSP runtime.  The compiler has passed
>>> > more than 7 million SQL tests from SQLLogicTest (with 3 tests
>>> > failing).  We will present the core design of the IVM engine and the
>>> > way we use Calcite to compile SQL to streaming computations.  We
>>> > discuss a few syntactic and sematic differences between SQL dialects
>>> > that we have identified.  Should Calcite support a notion of an "input
>>> > SQL dialect", that would allow it to emulate behaviors that differ
>>> > between other established SQL engines?
>>> >
>>> >
>>> >> -Original Message-
>>> >> From: Alessandro Solimando 
>>> >> Sent: Monday, February 6, 2023 12:00 AM
>>> >> To: dev@calcite.apache.org
>>> >> Subject: Re: [DISCUSS] Apache Calcite Meetup March 2023
>>> >>
>>> >> !! External Email
>>> >>
>>> >> Hello everyone,
>>> >> the proposed date and time is fine with me and I'd be happy to hear 
>>> >> Julian's
>>> >> talk.
>>> >>
>>> >> In addition, together with Stamatis, we could present how the extended
>>> >> logging capabilities introduced via CALCITE-4704
>>> >> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
>>> >> ues.apache.org%2Fjira%2Fbrowse%2FCALCITE-
>>> >> 4704=05%7C01%7Cmbudiu%40vmware.com%7Cf608d2711bd54ca8856
>>> >> b08db0818514c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C6381
>>> >> 12672763787844%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>>> >> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
>>> >> ata=94li