Re: RelNode semantic evolution vs adapter implementations

[Vladimir Ozerov]

Practically, Apache Calcite doesn’t have public API. Real projects usually
override not only public extension points, but also internals, such as
SqlValidatorImpl or VolcanoPlanner.

Given the flexibility that Apache Calcite provides, this lack of public API
surface is not necessarily a bad thing, since you may change almost
everything without forking the project. On the other hand, almost any
change to any part of Calcite code base may cause regressions in downstream
projects.

I am not sure there is an ideal way of versioning rels because even if you
do so, some direct or indirect changes (metadata, rules, validator, etc)
may cause changes to plans around the given RelNode in the downstream
project.

Maybe instead of providing versions manually, we may expose some sort of
generated signatures for different components - rels, rules, metadata
handlers, golden plans, etc. Then, we may provide the ability to verify
expected and actual signatures, e.g., using some code generation: user
invokes a command with the list of interesting classes, and gets the source
code of the test that he includes into his project and CI. Now, when the
user migrates to the new version, tests for changed entities will fail, and
user will investigate the associated changes.

The main difference s from the original proposal:
1. Wider scope, because validation of rels is often not sufficient.
2. Automation, because as implement or you cannot always predict the effect
of your changes (e.g., metadata).

WDYT?

Вт, 14 дек. 2021 г. в 01:49, Konstantin Orlov :

> > The case there is that "correlation variables" are added to the Logical*
> nodes (e.g. LogicalFilter, LogicalProject).
>
> BTW, "correlation variables" were added to a LogicalFilter in CALCITE-816.
> So what is wrong with CALCITE-3183?
>
> вт, 14 дек. 2021 г. в 01:24, Konstantin Orlov :
>
> > Vladimir, could you please clarify in what way the PR#2623 changes
> > the semantics?
> >
> > The correlated project is already possible in the master. The
> MongoProject
> > already discards variablesSet (simply because it's currently not stored
> > for
> > project node) and either fails or returns invalid results. This behavior
> > (alas,
> > incorrect) will be preserved after this patch.
> >
> > пн, 13 дек. 2021 г. в 17:55, Vladimir Sitnikov <
> > sitnikov.vladi...@gmail.com>:
> >
> >> Hi,
> >>
> >> It turns out Calcite's nodes can change semantics without notification
> for
> >> the end-users.
> >>
> >> Here are the release notes for Calcite 1.21, and it says **nothing**
> like
> >> "Ensure you handle Filter#variablesSet in case you implement Filter or
> in
> >> case you transform LogicalFilter in your rules"
> >> https://calcite.apache.org/news/2019/09/11/release-1.21.0/
> >>
> >> On top of that, the in-core adapters fail to handle that properly. For
> >> example, MongoFilter discards Filter#variablesSet.
> >>
> >> Can we please stop changing the semantics of RelNodes or can we have a
> >> better way to detect the changes in the client code?
> >>
> >> What if we add a "version" property to the corresponding RelNodes, and
> we
> >> increment it every time the semantic changes?
> >> Then client APIs could be coded like "ok, I'm prepared to handle Project
> >> v4, and Filter v5" (e.g. make "version" required when registering a
> rule),
> >> and there will be a runtime error in case Calcite generates Filter v6 in
> >> runtime.
> >>
> >> ---
> >>
> >> Sample case:
> >> CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
> >> CALCITE-4913 Correlated variables in a select list are not deduplicated
> >>
> >> The case there is that "correlation variables" are added to the Logical*
> >> nodes (e.g. LogicalFilter, LogicalProject).
> >> Unfortunately, that change is hard to notice: there's no compilation
> >> failure, and there's no runtime error.
> >>
> >> The old client code just discards "correlation variables", and I guess
> it
> >> would result in wrong results or something like that.
> >> Silent wrong results is a really sad outcome from the database.
> >>
> >> CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
> >> guess it would in the same hidden semantic loss.
> >>
> >> Vladimir
> >>
> >
> >
> > --
> > Regards,
> > Konstantin Orlov
> >
>
>
> --
> Regards,
> Konstantin Orlov
>

Re: JIRA perm issues?

[Jacques Nadeau]

I was but apparently I have two identities :D

The strange part is that the identity that didn't have any permissions has
a profile listed that includes all the relevant/expected groups...

On Mon, Dec 13, 2021 at 6:46 PM Chunwei Lei  wrote:

> I am able to resolve things in jira. Do you log in?
>
>
> Best,
> Chunwei
>
>
> On Tue, Dec 14, 2021 at 8:17 AM Jacques Nadeau  wrote:
>
> > Hey all, I am no longer able to resolve things in jira (across multiple
> > projects). Are others seeing the same thing? The buttons are just gone
> and
> > most fields are no longer editable.
> >
> > thanks,
> > Jacques
> >
>

Re: JIRA perm issues?

[Chunwei Lei]

I am able to resolve things in jira. Do you log in?


Best,
Chunwei


On Tue, Dec 14, 2021 at 8:17 AM Jacques Nadeau  wrote:

> Hey all, I am no longer able to resolve things in jira (across multiple
> projects). Are others seeing the same thing? The buttons are just gone and
> most fields are no longer editable.
>
> thanks,
> Jacques
>

Re: [DISCUSS] Do we need a Avatica release before Calcite 1.29.0 release?

[Julian Hyde]

Correct.

On Mon, Dec 13, 2021 at 5:24 PM Rui Wang  wrote:
>
> Ok to confirm, as https://issues.apache.org/jira/browse/CALCITE-4937 is
> marked as resolved, there is no blocker from Avatica side for 1.29.0
> release?
>
>
> -Rui
>
> On Fri, Dec 10, 2021 at 1:47 PM Jacques Nadeau  wrote:
>
> > Given the conversation around the log4j cve, it seems like we may need an
> > Avatica release afterall...
> >
> > Let's see how that discussion resolves.
> >
> > On Thu, Dec 9, 2021 at 12:47 PM Rui Wang  wrote:
> >
> > > Ack thanks!
> > >
> > > If there is no objection until the end of this week I will just go for
> > the
> > > 1.29.0 release process.
> > >
> > >
> > > -Rui
> > >
> > > On Thu, Dec 9, 2021 at 11:40 AM Josh Elser  wrote:
> > >
> > > > CALCITE-4152 is a big performance improvement to Avatica with
> > > > SPNEGO/Kerberos, but not at all about correctness/datatypes that
> > Calcite
> > > > needs.
> > > >
> > > > I'd agree with Julian.
> > > >
> > > > On 12/8/21 7:03 PM, Julian Hyde wrote:
> > > > > Probably not.
> > > > >
> > > > > * 4cf769f8e - (HEAD -> master, origin/master, origin/HEAD) Disable
> > > > Travis job that uses Calcite master until [CALCITE-4877] is fixed (4
> > > weeks
> > > > ago) 
> > > > > * 434cf002f - [CALCITE-4877] Make the exception information more
> > > > explicit for instantiate plugin. (5 weeks ago) 
> > > > > * 57c254f24 - [CALCITE-4840] Make README easier to scan (6 weeks ago)
> > > > 
> > > > > * c3a919234 - [CALCITE-4152] Upgrade Avatica to use the configurable
> > > > SPNEGO Jetty implementation (6 weeks ago) 
> > > > > * 95c1eb06e - [CALCITE-4828] Standard exception console output (7
> > weeks
> > > > ago) 
> > > > > * e462b09d4 - [CALCITE-4837] FLOOR/CEIL for DECADE, CENTURY,
> > MILLENIUM
> > > > return wrong results (8 weeks ago) 
> > > > > * ef277ff34 - Update website for Avatica 1.19.0 release (8 weeks ago)
> > > > 
> > > > >
> > > > >> On Dec 8, 2021, at 2:37 PM, Rui Wang  wrote:
> > > > >>
> > > > >> Hi community,
> > > > >>
> > > > >> Do you think we need an Avatica release before Calcite 1.29.0?
> > > > >>
> > > > >>
> > > > >> -Rui
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] Do we need a Avatica release before Calcite 1.29.0 release?

[Rui Wang]

Ok to confirm, as https://issues.apache.org/jira/browse/CALCITE-4937 is
marked as resolved, there is no blocker from Avatica side for 1.29.0
release?


-Rui

On Fri, Dec 10, 2021 at 1:47 PM Jacques Nadeau  wrote:

> Given the conversation around the log4j cve, it seems like we may need an
> Avatica release afterall...
>
> Let's see how that discussion resolves.
>
> On Thu, Dec 9, 2021 at 12:47 PM Rui Wang  wrote:
>
> > Ack thanks!
> >
> > If there is no objection until the end of this week I will just go for
> the
> > 1.29.0 release process.
> >
> >
> > -Rui
> >
> > On Thu, Dec 9, 2021 at 11:40 AM Josh Elser  wrote:
> >
> > > CALCITE-4152 is a big performance improvement to Avatica with
> > > SPNEGO/Kerberos, but not at all about correctness/datatypes that
> Calcite
> > > needs.
> > >
> > > I'd agree with Julian.
> > >
> > > On 12/8/21 7:03 PM, Julian Hyde wrote:
> > > > Probably not.
> > > >
> > > > * 4cf769f8e - (HEAD -> master, origin/master, origin/HEAD) Disable
> > > Travis job that uses Calcite master until [CALCITE-4877] is fixed (4
> > weeks
> > > ago) 
> > > > * 434cf002f - [CALCITE-4877] Make the exception information more
> > > explicit for instantiate plugin. (5 weeks ago) 
> > > > * 57c254f24 - [CALCITE-4840] Make README easier to scan (6 weeks ago)
> > > 
> > > > * c3a919234 - [CALCITE-4152] Upgrade Avatica to use the configurable
> > > SPNEGO Jetty implementation (6 weeks ago) 
> > > > * 95c1eb06e - [CALCITE-4828] Standard exception console output (7
> weeks
> > > ago) 
> > > > * e462b09d4 - [CALCITE-4837] FLOOR/CEIL for DECADE, CENTURY,
> MILLENIUM
> > > return wrong results (8 weeks ago) 
> > > > * ef277ff34 - Update website for Avatica 1.19.0 release (8 weeks ago)
> > > 
> > > >
> > > >> On Dec 8, 2021, at 2:37 PM, Rui Wang  wrote:
> > > >>
> > > >> Hi community,
> > > >>
> > > >> Do you think we need an Avatica release before Calcite 1.29.0?
> > > >>
> > > >>
> > > >> -Rui
> > > >
> > >
> >
>

Build failed in Jenkins: Calcite » Calcite-snapshots #9

[Apache Jenkins Server]

See 


Changes:

[Julian Hyde] [CALCITE-4937] Upgrade Calcite to Avatica 1.20

[Julian Hyde] [CALCITE-4938] Upgrade SQLLine to 1.12.0


--
[...truncated 322.27 KB...]

> Task :plus:test
 42.8sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery06()
 24.5sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery14()
 42.9sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery01()
 25.4sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery12()
 23.0sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery20()
 43.3sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery10()
 24.9sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery15()
 43.6sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery03()
 43.7sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery08()
 43.8sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery04()
 23.7sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery18()
 44.1sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery09()
 24.5sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery17()
 25.8sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery21()

> Task :redis:test
 49.3sec,2 completed,   0 failed,   0 skipped, 
org.apache.calcite.adapter.redis.RedisAdapterCaseBase
WARNING  52.2sec,3 completed,   0 failed,   1 
skipped, Gradle Test Run :redis:test

> Task :plus:test
 49.7sec, org.apache.calcite.adapter.tpch.TpchTest 
> testQuery05()

> Task :elasticsearch:test
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
dateCat()
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
integerCat()
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
cat1Cat2Cat3()
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
cat1Cat3()
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projection2()
  3.0sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projection()
  3.1sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
cat1Cat2()
  3.1sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
aggregationWithCast()
  3.2sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
simpleProjectionNoScripting()
  3.2sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projection3()
  3.2sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
anyValue()
  3.3sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
cat2()
  3.3sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
anyValueWithOtherAgg()
  3.3sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
all()
  3.4sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projectionStringLiteral()
  3.4sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
cat1()
  3.4sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
approximateCountDistinct()
  3.4sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projectionStringLiteralAndColumn()
  3.7sec, 
org.apache.calcite.adapter.elasticsearch.AggregationTest > 
countStar()
WARNING  43.1sec,   14 completed,   0 failed,   1 
skipped, org.apache.calcite.adapter.elasticsearch.AggregationTest
  3.7sec, 
org.apache.calcite.adapter.elasticsearch.Projection2Test > 
projectionWithIdField()
 43.1sec,7 completed,   0 failed,   0 skipped, 

Re: Automatic maven snapshot releases

[Jacques Nadeau]

Josh, I've enabled the basic integration tests in my updated patch:
https://github.com/apache/calcite/runs/4513702440?check_suite_focus=true

LMK if you think that is sufficient.

Thanks

On Mon, Dec 13, 2021 at 4:05 PM Jacques Nadeau  wrote:

> I added a blurb about snapshot releases in the howto in the updated PR.
> LMK your thoughts.
>
> https://github.com/apache/calcite/pull/2641/files#
>
> On Mon, Dec 13, 2021 at 11:34 AM Julian Hyde  wrote:
>
>> Jacques,
>>
>> Your 4 points are convincing. Snapshots (used in CI) will help people
>> stay on the main branch, and therefore will increase engagement.
>>
>> It would be useful if there were some guidance in the HOWTO on using
>> snapshots. (Don't release based on snapshots; snapshots may change or
>> disappear without notice; snapshots are not Apache licensed; do run
>> your CI against snapshots, but remember that if they fail, there are
>> now TWO places to look for the failure.)
>>
>> Julian
>>
>>
>> On Sun, Dec 12, 2021 at 4:11 PM Jacques Nadeau 
>> wrote:
>> >
>> > Good questions, Julian.
>> >
>> > Some thoughts, in random order. I think snapshot releases have the
>> ability
>> > to:
>> >
>> > 1. Reduce the likelihood of long-lived forks
>> > Right now, the easiest way one has to produce snapshot artifacts is to
>> set
>> > up a separate process around a forked repo. Once this process exists,
>> it is
>> > a slippery slope towards starting to rely on those artifacts for not
>> only
>> > development but also releases.
>> >
>> > 2. Reduces the likelihood of fork-causing events
>> > If one's dev branch builds against master snapshot artifacts, one is
>> more
>> > likely to notice a disruptive change sooner. This then allows one to
>> start
>> > a conversation about the disruption as early as possible post merge (in
>> a
>> > perfect world, it would be noticed pre-merge but that's a separate
>> topic).
>> > If one waits for releases to test new features and confirm if they are
>> > disruptive, this makes adaptation harder. The developer who added the
>> > disruptive change is less likely to be available and thinking about the
>> > same issues and the pressure to push a release will make this extra
>> > painful. This kind of situation increases the likelihood of hard forks.
>> > I've experienced this first hand several times and hope to never repeat
>> it
>> > again.
>> >
>> > 3. Reduces the likelihood of accidental breaking changes (or awkward
>> > situations)
>> > Similar to 2 above, if downstream projects depend on released versions
>> > during development, it increases the likelihood of a breaking change
>> being
>> > accidentally introduced. And on the opposite side, the introduction of
>> new
>> > changes may turn out to be problematic or challenging. Catching those
>> items
>> > before release decreases the likelihood of introducing an API and then
>> > having to back it out afterwards (and dealing with compat implications).
>> > With snapshot artifacts, I believe people are more likely to catch these
>> > problems sooner.
>> >
>> > 4. Increase interest/effort towards releases
>> > Reasonable people won't release against a snapshot release (my $0.02).
>> If
>> > these people develop more often against Calcite master (as opposed to
>> their
>> > own fork), they are more likely to push for a release to help them
>> > accommodate their own releasing needs.
>> >
>> > Of course, these are all just my own theories about this stuff. I
>> > unfortunately don't have any numbers to back this up. Nonetheless, I
>> think
>> > these benefits outweigh associated costs or potential downsides.
>> >
>> > Jacques
>> >
>> >
>> >
>> >
>> > On Sun, Dec 12, 2021 at 2:08 PM Julian Hyde 
>> wrote:
>> >
>> > > I have no problems with the technical side of this. And I am
>> supportive of
>> > > one-off snapshots, e.g. a snapshot for testing made a week or two
>> ahead of
>> > > a release.
>> > >
>> > > But there are some downsides with regular, automated snapshots, and I
>> am
>> > > concerned that projects and companies will become dependent on them,
>> for
>> > > the following reasons:
>> > >
>> > > 1. Snapshots can change without notice. (Bugs may be introduced, APIs
>> > > removed without notice.) This is mostly the problem for the downstream
>> > > project — in other words, caveat emptor — but may cause work for
>> > > committers. It’s certainly a bad idea to base a release on a snapshot.
>> > >
>> > > 2. Snapshots are not releases. The Apache license does not apply to
>> code
>> > > that has not been released.
>> > >
>> > > 3. If people get too comfortable using snapshots, there will be fewer
>> > > resources to make official releases.
>> > >
>> > > Is the additional convenience worth these downsides? Just asking. I’m
>> > > prepared to be persuaded.
>> > >
>> > > Julian
>> > >
>> > >
>> > >
>> > > > On Dec 12, 2021, at 10:23 AM, Jacques Nadeau 
>> wrote:
>> > > >
>> > > > Hey All,
>> > > >
>> > > > I've been wanting to do more testing against master for integration
>> 

JIRA perm issues?

[Jacques Nadeau]

Hey all, I am no longer able to resolve things in jira (across multiple
projects). Are others seeing the same thing? The buttons are just gone and
most fields are no longer editable.

thanks,
Jacques

Re: Automatic maven snapshot releases

[Jacques Nadeau]

I added a blurb about snapshot releases in the howto in the updated PR. LMK
your thoughts.

https://github.com/apache/calcite/pull/2641/files#

On Mon, Dec 13, 2021 at 11:34 AM Julian Hyde  wrote:

> Jacques,
>
> Your 4 points are convincing. Snapshots (used in CI) will help people
> stay on the main branch, and therefore will increase engagement.
>
> It would be useful if there were some guidance in the HOWTO on using
> snapshots. (Don't release based on snapshots; snapshots may change or
> disappear without notice; snapshots are not Apache licensed; do run
> your CI against snapshots, but remember that if they fail, there are
> now TWO places to look for the failure.)
>
> Julian
>
>
> On Sun, Dec 12, 2021 at 4:11 PM Jacques Nadeau  wrote:
> >
> > Good questions, Julian.
> >
> > Some thoughts, in random order. I think snapshot releases have the
> ability
> > to:
> >
> > 1. Reduce the likelihood of long-lived forks
> > Right now, the easiest way one has to produce snapshot artifacts is to
> set
> > up a separate process around a forked repo. Once this process exists, it
> is
> > a slippery slope towards starting to rely on those artifacts for not only
> > development but also releases.
> >
> > 2. Reduces the likelihood of fork-causing events
> > If one's dev branch builds against master snapshot artifacts, one is more
> > likely to notice a disruptive change sooner. This then allows one to
> start
> > a conversation about the disruption as early as possible post merge (in a
> > perfect world, it would be noticed pre-merge but that's a separate
> topic).
> > If one waits for releases to test new features and confirm if they are
> > disruptive, this makes adaptation harder. The developer who added the
> > disruptive change is less likely to be available and thinking about the
> > same issues and the pressure to push a release will make this extra
> > painful. This kind of situation increases the likelihood of hard forks.
> > I've experienced this first hand several times and hope to never repeat
> it
> > again.
> >
> > 3. Reduces the likelihood of accidental breaking changes (or awkward
> > situations)
> > Similar to 2 above, if downstream projects depend on released versions
> > during development, it increases the likelihood of a breaking change
> being
> > accidentally introduced. And on the opposite side, the introduction of
> new
> > changes may turn out to be problematic or challenging. Catching those
> items
> > before release decreases the likelihood of introducing an API and then
> > having to back it out afterwards (and dealing with compat implications).
> > With snapshot artifacts, I believe people are more likely to catch these
> > problems sooner.
> >
> > 4. Increase interest/effort towards releases
> > Reasonable people won't release against a snapshot release (my $0.02). If
> > these people develop more often against Calcite master (as opposed to
> their
> > own fork), they are more likely to push for a release to help them
> > accommodate their own releasing needs.
> >
> > Of course, these are all just my own theories about this stuff. I
> > unfortunately don't have any numbers to back this up. Nonetheless, I
> think
> > these benefits outweigh associated costs or potential downsides.
> >
> > Jacques
> >
> >
> >
> >
> > On Sun, Dec 12, 2021 at 2:08 PM Julian Hyde 
> wrote:
> >
> > > I have no problems with the technical side of this. And I am
> supportive of
> > > one-off snapshots, e.g. a snapshot for testing made a week or two
> ahead of
> > > a release.
> > >
> > > But there are some downsides with regular, automated snapshots, and I
> am
> > > concerned that projects and companies will become dependent on them,
> for
> > > the following reasons:
> > >
> > > 1. Snapshots can change without notice. (Bugs may be introduced, APIs
> > > removed without notice.) This is mostly the problem for the downstream
> > > project — in other words, caveat emptor — but may cause work for
> > > committers. It’s certainly a bad idea to base a release on a snapshot.
> > >
> > > 2. Snapshots are not releases. The Apache license does not apply to
> code
> > > that has not been released.
> > >
> > > 3. If people get too comfortable using snapshots, there will be fewer
> > > resources to make official releases.
> > >
> > > Is the additional convenience worth these downsides? Just asking. I’m
> > > prepared to be persuaded.
> > >
> > > Julian
> > >
> > >
> > >
> > > > On Dec 12, 2021, at 10:23 AM, Jacques Nadeau 
> wrote:
> > > >
> > > > Hey All,
> > > >
> > > > I've been wanting to do more testing against master for integration
> > > > purposes and right now that requires private builds. As such, I
> opened
> > > > CALCITE-4934 [1] to add support for automatic snapshot builds
> deployed to
> > > > the Apache snapshot Maven repository on master merges. As noted in
> the
> > > > ticket, this used to be the case many years ago (CALCITE-351 enabled
> it).
> > > > The right bits are now enabled in GitHub to make 

[jira] [Created] (CALCITE-4938) Upgrade SQLLine to 1.12.0

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4938:


 Summary: Upgrade SQLLine to 1.12.0
 Key: CALCITE-4938
 URL: https://issues.apache.org/jira/browse/CALCITE-4938
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Upgrade SQLLine to 1.12.0. Among [other 
things|https://github.com/julianhyde/sqlline/blob/main/HISTORY.md#1120-2021-11-07],
 SQLLine 1.12 can run on 
[Apple/M1|https://github.com/julianhyde/sqlline/issues/444].



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[ANNOUNCE] Apache Calcite Avatica 1.20.0 released

[Julian Hyde]

The Apache Calcite team is pleased to announce the release of Apache
Calcite Avatica 1.20.0.

Avatica is a framework for building database drivers. Avatica defines
a wire API and serialization mechanism for clients to communicate with
a server as a proxy to a database. The reference Avatica client and
server are implemented in Java and communicate over HTTP. Avatica is a
sub-project of Apache Calcite.

Apache Calcite Avatica 1.20.0 upgrades Log4j2 to version 2.15.0 (to
address CVE-2021-44228), and makes the SPNEGO protocol much more
efficient.

For a full list of changes, please see the release notes:

  https://calcite.apache.org/avatica/docs/history.html#v1-20-0

The release is available here:

  https://calcite.apache.org/avatica/downloads/avatica.html

We welcome your help and feedback. For more information on how to
report problems and get involved, visit the project website at:

  https://calcite.apache.org/avatica/

or the Apache Calcite project website:

  https://calcite.apache.org/

Thanks to everyone involved. The Log4j issue only became known on
Friday, and a half dozen Calcite committers and PMC members worked
over the weekend to create a release to address this issue. We take
your security seriously!

Julian Hyde, on behalf of the Apache Calcite team

Re: [RESULT] [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Jacques Nadeau]

Thanks for sheparding this through Julian

On Mon, Dec 13, 2021, 11:52 AM Julian Hyde  wrote:

> Thanks to everyone who has tested the release candidate and given
> their comments and votes.
>
> A special thank you to all of you who worked over the weekend on this,
> fixing bugs, making release candidates, and reviewing and voting on
> the release candidates. The Log4j RCE issue has been a testing time
> for many Apache projects, and I am proud that we have risen to the
> challenge. I know that our users will be grateful that we take
> security issues seriously.
>
> The tally is as follows.
>
> 6 binding +1s:
> Francis
> Josh
> Jacques
> Julian
> Michael
> Stamatis
>
> 1 non-binding +1s:
> Alessandro
>
> No 0s or -1s.
>
> Therefore I am delighted to announce that the proposal to release
> Apache Calcite Avatica 1.20.0 has passed.
>
> Thanks everyone. We’ll now roll out the release. The announcement will
> follow shortly, but for those in a hurry to upgrade, I suspect that
> the artifacts will be on Maven Central in less than two hours from
> now.
>
> Julian
>
>
>
> On Dec 13, 2021, at 7:40 AM, Josh Elser  wrote:
>
> (sorry for the belated..) +1 (binding)
>
> * xsums/sigs are good
> * Can build from source
> * CALCITE-4395 a non-blocker, agreed
> * Git commit looks to be in order
>
> On 12/12/21 9:45 PM, Julian Hyde wrote:
>
> Hi all,
> Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
> candidate 0, and then canceled it due to CALCITE-4935. (See
> https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
> subject of this email ends "(release candidate 0) (second vote)".
> The artifacts are the same, but we have new information, namely the
> discussion about whether CALCITE-4935 is a show-stopper. My judgment,
> as release manager, that it is not a show-stopper, and therefore
> release candidate 0 is good. I have restarted the vote because I would
> like to give you all a chance to weigh in.
> This vote does NOT carry forward votes from the previous thread. If
> you voted yesterday, please vote again.
> The remainder of this email is the same as yesterday's email, except
> "The vote is open for the next 18 hours".
> You can read the release notes here:
>
> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html
> The commit to be voted upon:
>
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452
> Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452
> Tag:
>
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0
> The artifacts to be voted on are located here:
>
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
> (revision 51334)
> The hashes of the artifacts are as follows:
>
> 2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
> *apache-calcite-avatica-1.20.0-src.tar.gz
> A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
> https://www.apache.org/dist/calcite/KEYS
> To create the jars and test Apache Calcite Avatica: "gradle build
> -Prelease -PskipSign".
> If you do not have a Java/Gradle environment available, you can run the
> tests
> using docker. To do so, install docker and docker-compose, then run
> "docker-compose run test" from the root of the directory.
> Please vote on releasing this package as Apache Calcite Avatica 1.20.0.
> The vote is open for the next 18 hours and passes if a majority of at
> least three +1 PMC votes are cast. (The voting period is shorter than
> usual.)
> [ ] +1 Release this package as Apache Calcite Avatica 1.20.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
> Here is my vote:
> +1 (binding)
>

Re: RelNode semantic evolution vs adapter implementations

[Konstantin Orlov]

> The case there is that "correlation variables" are added to the Logical*
nodes (e.g. LogicalFilter, LogicalProject).

BTW, "correlation variables" were added to a LogicalFilter in CALCITE-816.
So what is wrong with CALCITE-3183?

вт, 14 дек. 2021 г. в 01:24, Konstantin Orlov :

> Vladimir, could you please clarify in what way the PR#2623 changes
> the semantics?
>
> The correlated project is already possible in the master. The MongoProject
> already discards variablesSet (simply because it's currently not stored
> for
> project node) and either fails or returns invalid results. This behavior
> (alas,
> incorrect) will be preserved after this patch.
>
> пн, 13 дек. 2021 г. в 17:55, Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com>:
>
>> Hi,
>>
>> It turns out Calcite's nodes can change semantics without notification for
>> the end-users.
>>
>> Here are the release notes for Calcite 1.21, and it says **nothing** like
>> "Ensure you handle Filter#variablesSet in case you implement Filter or in
>> case you transform LogicalFilter in your rules"
>> https://calcite.apache.org/news/2019/09/11/release-1.21.0/
>>
>> On top of that, the in-core adapters fail to handle that properly. For
>> example, MongoFilter discards Filter#variablesSet.
>>
>> Can we please stop changing the semantics of RelNodes or can we have a
>> better way to detect the changes in the client code?
>>
>> What if we add a "version" property to the corresponding RelNodes, and we
>> increment it every time the semantic changes?
>> Then client APIs could be coded like "ok, I'm prepared to handle Project
>> v4, and Filter v5" (e.g. make "version" required when registering a rule),
>> and there will be a runtime error in case Calcite generates Filter v6 in
>> runtime.
>>
>> ---
>>
>> Sample case:
>> CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
>> CALCITE-4913 Correlated variables in a select list are not deduplicated
>>
>> The case there is that "correlation variables" are added to the Logical*
>> nodes (e.g. LogicalFilter, LogicalProject).
>> Unfortunately, that change is hard to notice: there's no compilation
>> failure, and there's no runtime error.
>>
>> The old client code just discards "correlation variables", and I guess it
>> would result in wrong results or something like that.
>> Silent wrong results is a really sad outcome from the database.
>>
>> CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
>> guess it would in the same hidden semantic loss.
>>
>> Vladimir
>>
>
>
> --
> Regards,
> Konstantin Orlov
>


-- 
Regards,
Konstantin Orlov

Re: RelNode semantic evolution vs adapter implementations

[Konstantin Orlov]

Vladimir, could you please clarify in what way the PR#2623 changes
the semantics?

The correlated project is already possible in the master. The MongoProject
already discards variablesSet (simply because it's currently not stored for
project node) and either fails or returns invalid results. This behavior
(alas,
incorrect) will be preserved after this patch.

пн, 13 дек. 2021 г. в 17:55, Vladimir Sitnikov :

> Hi,
>
> It turns out Calcite's nodes can change semantics without notification for
> the end-users.
>
> Here are the release notes for Calcite 1.21, and it says **nothing** like
> "Ensure you handle Filter#variablesSet in case you implement Filter or in
> case you transform LogicalFilter in your rules"
> https://calcite.apache.org/news/2019/09/11/release-1.21.0/
>
> On top of that, the in-core adapters fail to handle that properly. For
> example, MongoFilter discards Filter#variablesSet.
>
> Can we please stop changing the semantics of RelNodes or can we have a
> better way to detect the changes in the client code?
>
> What if we add a "version" property to the corresponding RelNodes, and we
> increment it every time the semantic changes?
> Then client APIs could be coded like "ok, I'm prepared to handle Project
> v4, and Filter v5" (e.g. make "version" required when registering a rule),
> and there will be a runtime error in case Calcite generates Filter v6 in
> runtime.
>
> ---
>
> Sample case:
> CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
> CALCITE-4913 Correlated variables in a select list are not deduplicated
>
> The case there is that "correlation variables" are added to the Logical*
> nodes (e.g. LogicalFilter, LogicalProject).
> Unfortunately, that change is hard to notice: there's no compilation
> failure, and there's no runtime error.
>
> The old client code just discards "correlation variables", and I guess it
> would result in wrong results or something like that.
> Silent wrong results is a really sad outcome from the database.
>
> CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
> guess it would in the same hidden semantic loss.
>
> Vladimir
>


-- 
Regards,
Konstantin Orlov

[jira] [Created] (CALCITE-4937) Upgrade Calcite to Avatica 1.20

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4937:


 Summary: Upgrade Calcite to Avatica 1.20
 Key: CALCITE-4937
 URL: https://issues.apache.org/jira/browse/CALCITE-4937
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Upgrade Calcite to Avatica 1.20. 1.20 was [just 
released|https://calcite.apache.org/avatica/news/2021/12/13/release-1.20.0/].

We will need to change the variables that were set in CALCITE-4877 to work 
around incompatible changes in Avatica output.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: RelNode semantic evolution vs adapter implementations

[Vladimir Sitnikov]

>practice I fear that
>it will introduce a lot of red tape and false negatives

I am afraid "wrong results and corrupted data in database" is way worse
than "recompiling in case of Calcite update".

>I don't recall
>any feedback (positive or otherwise) about how we describe breaking
>changes

CALCITE-3183 was breaking in 1.21, and it was NOT listed in release notes
at all.
Everybody makes mistakes, so I would avoid using "release notes" as the
only way to convey the breaking changes.
In the ideal world, there should be a compilation failure (or runtime
failure) in case of semantic change.

>Maybe we can improve how we report breaking changes

I'm afraid people would miss the notifications.
They can upgrade across several calcite versions, so they might miss
"one release inside of 1.21...1.28 has a significant breaking change"

If we increased the major version number, then they would anticipate the
change.
However, the thing is that "the change in Project" does not necessarily
break all consumers,
and it breaks only those consumers that inspect Project contents and/or
write new methods, etc.



>CALCITE-3183 Trimming method for Filter

In that case, the new field was added to LogicalFilter, so custom
implementations of Filter nodes (e.g. MongoFilter)
had no chance to notice the change.

>CALCITE-4913 Correlated variables in a select list

In the current PR 2623, the extra field is added to Project.java, however,
old constructor is kept intact, so even classes that extend Project don't
notice the semantics difference.

My takeaway is that:
1) It looks like we should refrain from adding fields to "Logical*" rels
without adding the corresponding fields to base classes.
2) It looks like we want to change the constructor signature, so the user's
code breaks if they extended Project.
As an alternative option is to keep old constructors and mark them
as @Deprecated, however, that deprecation would mean
"you get wrong results unless you support the new constructor argument
somehow".

3) Unfortunately, the planner rules do not care which signature the
constructors have, so the planner rules
won't notice the semantic difference.
So I would suggest we add per-relnode "version" (or "feature set"), so the
planner rule could declare "I know what Filter up to v5 means".

We do not change node semantics often, however, when we do we want to avoid
accidental "wrong results".

Vladimir

[RESULT] [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Julian Hyde]

Thanks to everyone who has tested the release candidate and given
their comments and votes.

A special thank you to all of you who worked over the weekend on this,
fixing bugs, making release candidates, and reviewing and voting on
the release candidates. The Log4j RCE issue has been a testing time
for many Apache projects, and I am proud that we have risen to the
challenge. I know that our users will be grateful that we take
security issues seriously.

The tally is as follows.

6 binding +1s:
Francis
Josh
Jacques
Julian
Michael
Stamatis

1 non-binding +1s:
Alessandro

No 0s or -1s.

Therefore I am delighted to announce that the proposal to release
Apache Calcite Avatica 1.20.0 has passed.

Thanks everyone. We’ll now roll out the release. The announcement will
follow shortly, but for those in a hurry to upgrade, I suspect that
the artifacts will be on Maven Central in less than two hours from
now.

Julian



On Dec 13, 2021, at 7:40 AM, Josh Elser  wrote:

(sorry for the belated..) +1 (binding)

* xsums/sigs are good
* Can build from source
* CALCITE-4395 a non-blocker, agreed
* Git commit looks to be in order

On 12/12/21 9:45 PM, Julian Hyde wrote:

Hi all,
Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
candidate 0, and then canceled it due to CALCITE-4935. (See
https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
subject of this email ends "(release candidate 0) (second vote)".
The artifacts are the same, but we have new information, namely the
discussion about whether CALCITE-4935 is a show-stopper. My judgment,
as release manager, that it is not a show-stopper, and therefore
release candidate 0 is good. I have restarted the vote because I would
like to give you all a chance to weigh in.
This vote does NOT carry forward votes from the previous thread. If
you voted yesterday, please vote again.
The remainder of this email is the same as yesterday's email, except
"The vote is open for the next 18 hours".
You can read the release notes here:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html
The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452
Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452
Tag:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0
The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
(revision 51334)
The hashes of the artifacts are as follows:
2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
*apache-calcite-avatica-1.20.0-src.tar.gz
A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jhyde.asc
https://www.apache.org/dist/calcite/KEYS
To create the jars and test Apache Calcite Avatica: "gradle build
-Prelease -PskipSign".
If you do not have a Java/Gradle environment available, you can run the
tests
using docker. To do so, install docker and docker-compose, then run
"docker-compose run test" from the root of the directory.
Please vote on releasing this package as Apache Calcite Avatica 1.20.0.
The vote is open for the next 18 hours and passes if a majority of at
least three +1 PMC votes are cast. (The voting period is shorter than
usual.)
[ ] +1 Release this package as Apache Calcite Avatica 1.20.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...
Here is my vote:
+1 (binding)

Re: RelNode semantic evolution vs adapter implementations

[Julian Hyde]

Vladimir's points are correct in theory, but in practice I fear that
it will introduce a lot of red tape and false negatives, and not yield
benefits in proportion to that effort.

Maybe we can improve how we report breaking changes. I don't recall
any feedback (positive or otherwise) about how we describe breaking
changes in our release notes. So it's difficult to know how we can do
better.

Julian

On Mon, Dec 13, 2021 at 9:57 AM Alessandro Solimando
 wrote:
>
> Hi,
> what you mention really sounds like breaking changes, so they should be
> treated as such and be explicitly mentioned in the release notes.
>
> Whenever possible, these behaviours should also be covered by unit tests,
> in order to limit the chance of changes sneaking in silently.
>
> Concerning the versioning of RelNode semantics, do you expect "consuming"
> code to throw on higher semantics versions than expected?
> I see the general idea behind your proposal but I fail to imagine where
> this kind of version check should happen, in order not to pollute too much
> the "consumer" code.
>
> Best regards,
> Alessandro
>
> On Mon, 13 Dec 2021 at 15:55, Vladimir Sitnikov 
> wrote:
>
> > Hi,
> >
> > It turns out Calcite's nodes can change semantics without notification for
> > the end-users.
> >
> > Here are the release notes for Calcite 1.21, and it says **nothing** like
> > "Ensure you handle Filter#variablesSet in case you implement Filter or in
> > case you transform LogicalFilter in your rules"
> > https://calcite.apache.org/news/2019/09/11/release-1.21.0/
> >
> > On top of that, the in-core adapters fail to handle that properly. For
> > example, MongoFilter discards Filter#variablesSet.
> >
> > Can we please stop changing the semantics of RelNodes or can we have a
> > better way to detect the changes in the client code?
> >
> > What if we add a "version" property to the corresponding RelNodes, and we
> > increment it every time the semantic changes?
> > Then client APIs could be coded like "ok, I'm prepared to handle Project
> > v4, and Filter v5" (e.g. make "version" required when registering a rule),
> > and there will be a runtime error in case Calcite generates Filter v6 in
> > runtime.
> >
> > ---
> >
> > Sample case:
> > CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
> > CALCITE-4913 Correlated variables in a select list are not deduplicated
> >
> > The case there is that "correlation variables" are added to the Logical*
> > nodes (e.g. LogicalFilter, LogicalProject).
> > Unfortunately, that change is hard to notice: there's no compilation
> > failure, and there's no runtime error.
> >
> > The old client code just discards "correlation variables", and I guess it
> > would result in wrong results or something like that.
> > Silent wrong results is a really sad outcome from the database.
> >
> > CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
> > guess it would in the same hidden semantic loss.
> >
> > Vladimir
> >

Re: Automatic maven snapshot releases

[Julian Hyde]

Jacques,

Your 4 points are convincing. Snapshots (used in CI) will help people
stay on the main branch, and therefore will increase engagement.

It would be useful if there were some guidance in the HOWTO on using
snapshots. (Don't release based on snapshots; snapshots may change or
disappear without notice; snapshots are not Apache licensed; do run
your CI against snapshots, but remember that if they fail, there are
now TWO places to look for the failure.)

Julian


On Sun, Dec 12, 2021 at 4:11 PM Jacques Nadeau  wrote:
>
> Good questions, Julian.
>
> Some thoughts, in random order. I think snapshot releases have the ability
> to:
>
> 1. Reduce the likelihood of long-lived forks
> Right now, the easiest way one has to produce snapshot artifacts is to set
> up a separate process around a forked repo. Once this process exists, it is
> a slippery slope towards starting to rely on those artifacts for not only
> development but also releases.
>
> 2. Reduces the likelihood of fork-causing events
> If one's dev branch builds against master snapshot artifacts, one is more
> likely to notice a disruptive change sooner. This then allows one to start
> a conversation about the disruption as early as possible post merge (in a
> perfect world, it would be noticed pre-merge but that's a separate topic).
> If one waits for releases to test new features and confirm if they are
> disruptive, this makes adaptation harder. The developer who added the
> disruptive change is less likely to be available and thinking about the
> same issues and the pressure to push a release will make this extra
> painful. This kind of situation increases the likelihood of hard forks.
> I've experienced this first hand several times and hope to never repeat it
> again.
>
> 3. Reduces the likelihood of accidental breaking changes (or awkward
> situations)
> Similar to 2 above, if downstream projects depend on released versions
> during development, it increases the likelihood of a breaking change being
> accidentally introduced. And on the opposite side, the introduction of new
> changes may turn out to be problematic or challenging. Catching those items
> before release decreases the likelihood of introducing an API and then
> having to back it out afterwards (and dealing with compat implications).
> With snapshot artifacts, I believe people are more likely to catch these
> problems sooner.
>
> 4. Increase interest/effort towards releases
> Reasonable people won't release against a snapshot release (my $0.02). If
> these people develop more often against Calcite master (as opposed to their
> own fork), they are more likely to push for a release to help them
> accommodate their own releasing needs.
>
> Of course, these are all just my own theories about this stuff. I
> unfortunately don't have any numbers to back this up. Nonetheless, I think
> these benefits outweigh associated costs or potential downsides.
>
> Jacques
>
>
>
>
> On Sun, Dec 12, 2021 at 2:08 PM Julian Hyde  wrote:
>
> > I have no problems with the technical side of this. And I am supportive of
> > one-off snapshots, e.g. a snapshot for testing made a week or two ahead of
> > a release.
> >
> > But there are some downsides with regular, automated snapshots, and I am
> > concerned that projects and companies will become dependent on them, for
> > the following reasons:
> >
> > 1. Snapshots can change without notice. (Bugs may be introduced, APIs
> > removed without notice.) This is mostly the problem for the downstream
> > project — in other words, caveat emptor — but may cause work for
> > committers. It’s certainly a bad idea to base a release on a snapshot.
> >
> > 2. Snapshots are not releases. The Apache license does not apply to code
> > that has not been released.
> >
> > 3. If people get too comfortable using snapshots, there will be fewer
> > resources to make official releases.
> >
> > Is the additional convenience worth these downsides? Just asking. I’m
> > prepared to be persuaded.
> >
> > Julian
> >
> >
> >
> > > On Dec 12, 2021, at 10:23 AM, Jacques Nadeau  wrote:
> > >
> > > Hey All,
> > >
> > > I've been wanting to do more testing against master for integration
> > > purposes and right now that requires private builds. As such, I opened
> > > CALCITE-4934 [1] to add support for automatic snapshot builds deployed to
> > > the Apache snapshot Maven repository on master merges. As noted in the
> > > ticket, this used to be the case many years ago (CALCITE-351 enabled it).
> > > The right bits are now enabled in GitHub to make this effortless and I've
> > > confirmed that this looks like it is working correctly [2][3].
> > >
> > > Before merging I wanted to make sure that people were comfortable with
> > this
> > > addition since it changes build infra. The PR is pretty trivial [4] and
> > > modeled off other Apache projects.
> > >
> > > Lastly, note that snapshots are *not* Apache Releases and shouldn't be
> > > presented as such in docs, etc.
> > >
> > > Please 

Re: Automatic maven snapshot releases

[Jacques Nadeau]

I could add a basic build & test. Right now it is just a build and publish.

Basic unit test suite seems like it would cover most common cases (where a
rebase after pre-commit test causes a problem). Do you think it should be
more thorough? Less?

On Mon, Dec 13, 2021 at 9:52 AM Josh Elser  wrote:

> (top-posting to avoid extending the "how" divergence)
>
> Looks great and thanks for your care to avoid advertising them as
> "releases".
>
> Only thing I was going to mention was about whether or not there's a
> minimally "acceptable" test suite which can reliably run against master
> before the new job would run. I know folks have been chasing this for
> some time.
>
> On 12/12/21 1:23 PM, Jacques Nadeau wrote:
> > Hey All,
> >
> > I've been wanting to do more testing against master for integration
> > purposes and right now that requires private builds. As such, I opened
> > CALCITE-4934 [1] to add support for automatic snapshot builds deployed to
> > the Apache snapshot Maven repository on master merges. As noted in the
> > ticket, this used to be the case many years ago (CALCITE-351 enabled it).
> > The right bits are now enabled in GitHub to make this effortless and I've
> > confirmed that this looks like it is working correctly [2][3].
> >
> > Before merging I wanted to make sure that people were comfortable with
> this
> > addition since it changes build infra. The PR is pretty trivial [4] and
> > modeled off other Apache projects.
> >
> > Lastly, note that snapshots are *not* Apache Releases and shouldn't be
> > presented as such in docs, etc.
> >
> > Please raise your hand if you have any concerns.
> >
> > Thanks,
> > Jacques
> >
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-4934
> > [2]
> https://github.com/apache/calcite/runs/4499040456?check_suite_focus=true
> > [3]
> >
> https://repository.apache.org/content/groups/snapshots/org/apache/calcite/calcite-core/1.29.0-SNAPSHOT/
> > [4] https://github.com/apache/calcite/pull/2641
> >
>

Re: RelNode semantic evolution vs adapter implementations

[Alessandro Solimando]

Hi,
what you mention really sounds like breaking changes, so they should be
treated as such and be explicitly mentioned in the release notes.

Whenever possible, these behaviours should also be covered by unit tests,
in order to limit the chance of changes sneaking in silently.

Concerning the versioning of RelNode semantics, do you expect "consuming"
code to throw on higher semantics versions than expected?
I see the general idea behind your proposal but I fail to imagine where
this kind of version check should happen, in order not to pollute too much
the "consumer" code.

Best regards,
Alessandro

On Mon, 13 Dec 2021 at 15:55, Vladimir Sitnikov 
wrote:

> Hi,
>
> It turns out Calcite's nodes can change semantics without notification for
> the end-users.
>
> Here are the release notes for Calcite 1.21, and it says **nothing** like
> "Ensure you handle Filter#variablesSet in case you implement Filter or in
> case you transform LogicalFilter in your rules"
> https://calcite.apache.org/news/2019/09/11/release-1.21.0/
>
> On top of that, the in-core adapters fail to handle that properly. For
> example, MongoFilter discards Filter#variablesSet.
>
> Can we please stop changing the semantics of RelNodes or can we have a
> better way to detect the changes in the client code?
>
> What if we add a "version" property to the corresponding RelNodes, and we
> increment it every time the semantic changes?
> Then client APIs could be coded like "ok, I'm prepared to handle Project
> v4, and Filter v5" (e.g. make "version" required when registering a rule),
> and there will be a runtime error in case Calcite generates Filter v6 in
> runtime.
>
> ---
>
> Sample case:
> CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
> CALCITE-4913 Correlated variables in a select list are not deduplicated
>
> The case there is that "correlation variables" are added to the Logical*
> nodes (e.g. LogicalFilter, LogicalProject).
> Unfortunately, that change is hard to notice: there's no compilation
> failure, and there's no runtime error.
>
> The old client code just discards "correlation variables", and I guess it
> would result in wrong results or something like that.
> Silent wrong results is a really sad outcome from the database.
>
> CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
> guess it would in the same hidden semantic loss.
>
> Vladimir
>

Re: Automatic maven snapshot releases

[Josh Elser]

(top-posting to avoid extending the "how" divergence)

Looks great and thanks for your care to avoid advertising them as 
"releases".


Only thing I was going to mention was about whether or not there's a 
minimally "acceptable" test suite which can reliably run against master 
before the new job would run. I know folks have been chasing this for 
some time.


On 12/12/21 1:23 PM, Jacques Nadeau wrote:

Hey All,

I've been wanting to do more testing against master for integration
purposes and right now that requires private builds. As such, I opened
CALCITE-4934 [1] to add support for automatic snapshot builds deployed to
the Apache snapshot Maven repository on master merges. As noted in the
ticket, this used to be the case many years ago (CALCITE-351 enabled it).
The right bits are now enabled in GitHub to make this effortless and I've
confirmed that this looks like it is working correctly [2][3].

Before merging I wanted to make sure that people were comfortable with this
addition since it changes build infra. The PR is pretty trivial [4] and
modeled off other Apache projects.

Lastly, note that snapshots are *not* Apache Releases and shouldn't be
presented as such in docs, etc.

Please raise your hand if you have any concerns.

Thanks,
Jacques


[1] https://issues.apache.org/jira/browse/CALCITE-4934
[2] https://github.com/apache/calcite/runs/4499040456?check_suite_focus=true
[3]
https://repository.apache.org/content/groups/snapshots/org/apache/calcite/calcite-core/1.29.0-SNAPSHOT/
[4] https://github.com/apache/calcite/pull/2641

Re: Automatic maven snapshot releases

[Vladimir Sitnikov]

>Where is -Pasf defined/documented?

https://calcite.apache.org/docs/howto.html#making-a-snapshot

Vladimir

Re: Druid test hangs?

[Jacques Nadeau]

I wonder if we can create a simple shell script that runs a jstack once an
hour (starting after one hour) and then run it using
https://github.com/psxpaul/gradle-execfork-plugin? Since none of our jobs
run an hour, most of the time it wouldn't do anything. In the cases where
the job hung, we'd hopefully get a jstack.


On Mon, Dec 13, 2021 at 12:17 AM Stamatis Zampetakis 
wrote:

> If there is a systematic way to do it I would be interested to know.
>
> In the past, when I encountered similar hangs in CI what I ended-up doing
> is adding debugging commits in the PR with a thread printing stack traces
> of other threads at some intervals.
>
> Best,
> Stamatis
>
> On Sun, Dec 12, 2021 at 7:00 PM Jacques Nadeau  wrote:
>
> > It could be infra but I'm wondering if it is some kind of concurrency
> bug.
> >
> > Anyone know if there is a straightforward way to add a secondary process
> in
> > a github workflow that takes a jstack after an hour or something (if the
> > tests run that long). Trying to jump on an instance when this happens and
> > do this manually sounds like an effort in frustration.
> >
> > I guess another option would be to modify the druid job to provide info
> on
> > tests that are running so that we can see if it always locks on the same
> > test.
> >
> > On Sat, Dec 11, 2021 at 11:39 PM Alessandro Solimando <
> > alessandro.solima...@gmail.com> wrote:
> >
> > > I started noticing that intermittently around a month ago, I had a
> quick
> > > look back then but I could not pinpoint the root cause.
> > >
> > > I don't think it is expected, and I guess it comes from test infra
> setup
> > > rather than the Calcite code itself.
> > >
> > > Il Dom 12 Dic 2021, 05:43 Jacques Nadeau  ha
> > scritto:
> > >
> > > > I see a couple of recent builds with Druid tests hanging. Is that a
> > > normal
> > > > thing or something that has started recently.
> > > >
> > > > Examples:
> > > >
> > https://github.com/apache/calcite/runs/4487013505?check_suite_focus=true
> > > >
> > >
> >
> https://github.com/jacques-n/calcite/runs/4494836558?check_suite_focus=true
> > > >
> > >
> >
>

Re: Automatic maven snapshot releases

[Jacques Nadeau]

Let's keep the "should" conversation separate from the "how" conversation.

*On should: *
I understand that it worked in the past and then stopped working. I still
think it is appropriate to confirm people are comfortable with enabling it
again.

*On how:*
Vladimir, the github workflow job is configured to use ASF provided
credentials, as standard per other ASF projects that push using Github
workflows. My notes in the original post called out that this PR already
works on the ASF repo. (The INFRA ticket linked to
https://issues.apache.org/jira/browse/CALCITE-4934,
https://issues.apache.org/jira/browse/INFRA-22601 enabled the credentials.)

Where is -Pasf defined/documented? What are the credentials it is using
(I'm assuming it is an env variable)?





On Sun, Dec 12, 2021 at 9:36 PM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:

> Jacques, which credentials are you going to supply to publish the snapshots
> to the ASF repository from GitHub?
>
> While I agree with moving towards GitHub actions, I would like to minimize
> the use of ASF credentials in non-ASF environments.
>
> Luckily, ASF infra supplies the relevant credentials for publishing to ASF
> Nexus,
> so a mere "./gradlew -Pasf publish" at ASF Jenkins is enough to publish the
> snapshots.
>
> Stamatis>In the past we used to have Jenkins jobs doing this
>
> The job was there, so we don't really need to discuss or vote on that.
> I've created the corresponding Jenkins job here:
> https://ci-builds.apache.org/job/Calcite/job/Calcite-snapshots/
>
> It pulls GitHub every 5 minutes.
> I am not sure if GitHub webhook integration would trigger the build (it
> might need a config from INFRA), however, it is possible as well.
>
> Vladimir
>

Re: [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Josh Elser]

(sorry for the belated..) +1 (binding)

* xsums/sigs are good
* Can build from source
* CALCITE-4395 a non-blocker, agreed
* Git commit looks to be in order

On 12/12/21 9:45 PM, Julian Hyde wrote:

Hi all,

Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
candidate 0, and then canceled it due to CALCITE-4935. (See
https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
subject of this email ends "(release candidate 0) (second vote)".

The artifacts are the same, but we have new information, namely the
discussion about whether CALCITE-4935 is a show-stopper. My judgment,
as release manager, that it is not a show-stopper, and therefore
release candidate 0 is good. I have restarted the vote because I would
like to give you all a chance to weigh in.

This vote does NOT carry forward votes from the previous thread. If
you voted yesterday, please vote again.

The remainder of this email is the same as yesterday's email, except
"The vote is open for the next 18 hours".


You can read the release notes here:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html

The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452

Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452

Tag:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
(revision 51334)

The hashes of the artifacts are as follows:
2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
*apache-calcite-avatica-1.20.0-src.tar.gz

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jhyde.asc
https://www.apache.org/dist/calcite/KEYS

To create the jars and test Apache Calcite Avatica: "gradle build
-Prelease -PskipSign".

If you do not have a Java/Gradle environment available, you can run the tests
using docker. To do so, install docker and docker-compose, then run
"docker-compose run test" from the root of the directory.

Please vote on releasing this package as Apache Calcite Avatica 1.20.0.

The vote is open for the next 18 hours and passes if a majority of at
least three +1 PMC votes are cast. (The voting period is shorter than usual.)

[ ] +1 Release this package as Apache Calcite Avatica 1.20.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Re: [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Michael Mior]

+1 (binding) Checked release notes and hashes, compiled and ran tests.

Thanks Julian!

--
Michael Mior
mm...@apache.org

Le dim. 12 déc. 2021 à 21:45, Julian Hyde  a écrit :
>
> Hi all,
>
> Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
> candidate 0, and then canceled it due to CALCITE-4935. (See
> https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
> subject of this email ends "(release candidate 0) (second vote)".
>
> The artifacts are the same, but we have new information, namely the
> discussion about whether CALCITE-4935 is a show-stopper. My judgment,
> as release manager, that it is not a show-stopper, and therefore
> release candidate 0 is good. I have restarted the vote because I would
> like to give you all a chance to weigh in.
>
> This vote does NOT carry forward votes from the previous thread. If
> you voted yesterday, please vote again.
>
> The remainder of this email is the same as yesterday's email, except
> "The vote is open for the next 18 hours".
>
>
> You can read the release notes here:
> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html
>
> The commit to be voted upon:
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452
>
> Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452
>
> Tag:
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
> (revision 51334)
>
> The hashes of the artifacts are as follows:
> 2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
> *apache-calcite-avatica-1.20.0-src.tar.gz
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
> https://www.apache.org/dist/calcite/KEYS
>
> To create the jars and test Apache Calcite Avatica: "gradle build
> -Prelease -PskipSign".
>
> If you do not have a Java/Gradle environment available, you can run the tests
> using docker. To do so, install docker and docker-compose, then run
> "docker-compose run test" from the root of the directory.
>
> Please vote on releasing this package as Apache Calcite Avatica 1.20.0.
>
> The vote is open for the next 18 hours and passes if a majority of at
> least three +1 PMC votes are cast. (The voting period is shorter than usual.)
>
> [ ] +1 Release this package as Apache Calcite Avatica 1.20.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)

RelNode semantic evolution vs adapter implementations

[Vladimir Sitnikov]

Hi,

It turns out Calcite's nodes can change semantics without notification for
the end-users.

Here are the release notes for Calcite 1.21, and it says **nothing** like
"Ensure you handle Filter#variablesSet in case you implement Filter or in
case you transform LogicalFilter in your rules"
https://calcite.apache.org/news/2019/09/11/release-1.21.0/

On top of that, the in-core adapters fail to handle that properly. For
example, MongoFilter discards Filter#variablesSet.

Can we please stop changing the semantics of RelNodes or can we have a
better way to detect the changes in the client code?

What if we add a "version" property to the corresponding RelNodes, and we
increment it every time the semantic changes?
Then client APIs could be coded like "ok, I'm prepared to handle Project
v4, and Filter v5" (e.g. make "version" required when registering a rule),
and there will be a runtime error in case Calcite generates Filter v6 in
runtime.

---

Sample case:
CALCITE-3183 Trimming method for Filter rel uses wrong traitSet
CALCITE-4913 Correlated variables in a select list are not deduplicated

The case there is that "correlation variables" are added to the Logical*
nodes (e.g. LogicalFilter, LogicalProject).
Unfortunately, that change is hard to notice: there's no compilation
failure, and there's no runtime error.

The old client code just discards "correlation variables", and I guess it
would result in wrong results or something like that.
Silent wrong results is a really sad outcome from the database.

CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I
guess it would in the same hidden semantic loss.

Vladimir

Re: [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Alessandro Solimando]

Hi everyone,
my vote is +1 (non-binding) after checking the RC as follows:

- Checked hash and signature: OK
- Checked release notes: OK
- Checked the release tag with right commit: OK
- Checking the tag and the tar.gz content: OK
  diff -qr apache-calcite-avatica-1.20.0-src ~/git/calcite-avatica:
.trash spurious folder as noticed by Stamatis
- Visual inspection of the links provided in the email: OK
- ./gradlew build (after copying the wrapper and content of "gradle" folder
from the git clone) --> OK

The details about JVMs, gradle, and OS versions are as follows:

$ java -version (8.0.292.hs-adpt)
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

$ sw_vers
ProductName: macOS
ProductVersion: 11.6
BuildVersion: 20G165

$ ./gradlew -version

Gradle 6.8.1


Build time:   2021-01-22 13:20:08 UTC
Revision: 31f14a87d93945024ab7a78de84102a3400fa5b2

Kotlin:   1.4.20
Groovy:   2.5.12
Ant:  Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:  1.8.0_292 (AdoptOpenJDK 25.292-b10)
OS:   Mac OS X 10.16 x86_64

Best regards,
Alessandro

On Mon, 13 Dec 2021 at 13:43, Stamatis Zampetakis  wrote:

> Apologies for holding back the first vote due to CALCITE-4935, I misjudged
> the situation.
>
> Ubuntu 20.04.3 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 6.8.3
>
>  * Checked signatures and checksums OK
>  * Went over release note OK
>  * Built from git tag and run tests (./gradlew clean build) OK
>  * Built from source artifacts and run tests (gradle clean build) OK
>  * Checked diff between repo and artifacts OK
> There is an empty .trash directory in sources (and not in git) but it's
> minor and shouldn't block the release.
>
> +1 (binding)
>
> Best,
> Stamatis
>
> On Mon, Dec 13, 2021 at 5:01 AM Francis Chuang 
> wrote:
>
> > +1 (binding) as per my vote in the previous vote thread.
> >
> > Francis
> >
> > On 13/12/2021 2:05 pm, Jacques Nadeau wrote:
> > > +1 (binding).
> > >
> > > RC0 looks good to me (built/tested/verified) and I agree that
> > CALCITE-4935
> > > is not a release blocker.
> > >
> > > On Sun, Dec 12, 2021 at 6:45 PM Julian Hyde  wrote:
> > >
> > >> Hi all,
> > >>
> > >> Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
> > >> candidate 0, and then canceled it due to CALCITE-4935. (See
> > >> https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
> > >> subject of this email ends "(release candidate 0) (second vote)".
> > >>
> > >> The artifacts are the same, but we have new information, namely the
> > >> discussion about whether CALCITE-4935 is a show-stopper. My judgment,
> > >> as release manager, that it is not a show-stopper, and therefore
> > >> release candidate 0 is good. I have restarted the vote because I would
> > >> like to give you all a chance to weigh in.
> > >>
> > >> This vote does NOT carry forward votes from the previous thread. If
> > >> you voted yesterday, please vote again.
> > >>
> > >> The remainder of this email is the same as yesterday's email, except
> > >> "The vote is open for the next 18 hours".
> > >>
> > >>
> > >> You can read the release notes here:
> > >>
> > >>
> >
> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html
> > >>
> > >> The commit to be voted upon:
> > >>
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452
> > >>
> > >> Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452
> > >>
> > >> Tag:
> > >>
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0
> > >>
> > >> The artifacts to be voted on are located here:
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
> > >> (revision 51334)
> > >>
> > >> The hashes of the artifacts are as follows:
> > >>
> > >>
> >
> 2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
> > >> *apache-calcite-avatica-1.20.0-src.tar.gz
> > >>
> > >> A staged Maven repository is available for review at:
> > >>
> > >>
> >
> https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/
> > >>
> > >> Release artifacts are signed with the following key:
> > >> https://people.apache.org/keys/committer/jhyde.asc
> > >> https://www.apache.org/dist/calcite/KEYS
> > >>
> > >> To create the jars and test Apache Calcite Avatica: "gradle build
> > >> -Prelease -PskipSign".
> > >>
> > >> If you do not have a Java/Gradle environment available, you can run
> the
> > >> tests
> > >> using docker. To do so, install docker and docker-compose, then run
> > >> "docker-compose run test" from the root of the directory.
> > >>
> 

Re: [VOTE] Release apache-calcite-avatica-1.20.0 (release candidate 0) (second vote)

[Stamatis Zampetakis]

Apologies for holding back the first vote due to CALCITE-4935, I misjudged
the situation.

Ubuntu 20.04.3 LTS, jdk1.8.0_261, Gradle wrapper, Gradle 6.8.3

 * Checked signatures and checksums OK
 * Went over release note OK
 * Built from git tag and run tests (./gradlew clean build) OK
 * Built from source artifacts and run tests (gradle clean build) OK
 * Checked diff between repo and artifacts OK
There is an empty .trash directory in sources (and not in git) but it's
minor and shouldn't block the release.

+1 (binding)

Best,
Stamatis

On Mon, Dec 13, 2021 at 5:01 AM Francis Chuang 
wrote:

> +1 (binding) as per my vote in the previous vote thread.
>
> Francis
>
> On 13/12/2021 2:05 pm, Jacques Nadeau wrote:
> > +1 (binding).
> >
> > RC0 looks good to me (built/tested/verified) and I agree that
> CALCITE-4935
> > is not a release blocker.
> >
> > On Sun, Dec 12, 2021 at 6:45 PM Julian Hyde  wrote:
> >
> >> Hi all,
> >>
> >> Yesterday I started a vote for Apache Calcite Avatica 1.20.0, release
> >> candidate 0, and then canceled it due to CALCITE-4935. (See
> >> https://issues.apache.org/jira/browse/CALCITE-4935.) Note that the
> >> subject of this email ends "(release candidate 0) (second vote)".
> >>
> >> The artifacts are the same, but we have new information, namely the
> >> discussion about whether CALCITE-4935 is a show-stopper. My judgment,
> >> as release manager, that it is not a show-stopper, and therefore
> >> release candidate 0 is good. I have restarted the vote because I would
> >> like to give you all a chance to weigh in.
> >>
> >> This vote does NOT carry forward votes from the previous thread. If
> >> you voted yesterday, please vote again.
> >>
> >> The remainder of this email is the same as yesterday's email, except
> >> "The vote is open for the next 18 hours".
> >>
> >>
> >> You can read the release notes here:
> >>
> >>
> https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html
> >>
> >> The commit to be voted upon:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=8f8377551e7b07dbc6119ecf196942bc99834452
> >>
> >> Its hash is 8f8377551e7b07dbc6119ecf196942bc99834452
> >>
> >> Tag:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.20.0-rc0
> >>
> >> The artifacts to be voted on are located here:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.20.0-rc0
> >> (revision 51334)
> >>
> >> The hashes of the artifacts are as follows:
> >>
> >>
> 2de57fb1823237ec2d73f2e26059b6a7f5f51ec8f6dcfdcb2eb0b562cf4ef141c4a0a8c430610815e85b0b1c1c60245062cbb075c6dc72a44a4f543a6813bff2
> >> *apache-calcite-avatica-1.20.0-src.tar.gz
> >>
> >> A staged Maven repository is available for review at:
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachecalcite-1122/org/apache/calcite/
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/jhyde.asc
> >> https://www.apache.org/dist/calcite/KEYS
> >>
> >> To create the jars and test Apache Calcite Avatica: "gradle build
> >> -Prelease -PskipSign".
> >>
> >> If you do not have a Java/Gradle environment available, you can run the
> >> tests
> >> using docker. To do so, install docker and docker-compose, then run
> >> "docker-compose run test" from the root of the directory.
> >>
> >> Please vote on releasing this package as Apache Calcite Avatica 1.20.0.
> >>
> >> The vote is open for the next 18 hours and passes if a majority of at
> >> least three +1 PMC votes are cast. (The voting period is shorter than
> >> usual.)
> >>
> >> [ ] +1 Release this package as Apache Calcite Avatica 1.20.0
> >> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> >> [ ] -1 Do not release this package because...
> >>
> >>
> >> Here is my vote:
> >>
> >> +1 (binding)
> >>
> >
>

Re: how to represent and optimize a query over partitioned storage

[Alessandro Solimando]

Thank you Jacques and Jing Zhang for the code pointers!

On Mon, 13 Dec 2021 at 03:58, Jing Zhang  wrote:

> Hi,
> partition pruner is a common requirement.
> You could provide a rule which push partition evaluated by filter condition
> into a TableScan.
> There are two ways to ensure the converted result would be chosen in the
> final plan.
> 1. If use CBO planner, you could override the `getRowCount ` by merging all
> the row count of matched partitions into total row count. After apply the
> rule, the total row count must be less than before.
> 2. You could use HBO planner which I thought is proper for this case,
> because we could always get benifite  from partition pruner.
>
> Apache Flink also provides Partitioner pruner, you could find it's
> implementation here [1]. Hope it helps.
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushPartitionIntoTableSourceScanRule.java
>
> Best,
> Jing Zhang
>
> Jacques Nadeau  于2021年12月13日周一 01:57写道：
>
> > I don't have a great pointer to an intelligent arbitrary filter coster.
> The
> > default one in Calcite isn't great (last I checked) as it considers more
> > filters to be equivalent to more reduction. This means it dramatically
> > overestimates set reduction. In Dremio, in simple cases we had to apply
> an
> > upper limit on reduction.
> >
> > For partitions specifically, we actually apply the condition to the
> > underlying partition details to get an accurate new cost. You can see the
> > base class of the pruning here:
> >
> >
> >
> https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/logical/partition/PruneScanRuleBase.java
> >
> > In this code, the problem is decomposed into first doing search argument
> > pruning (sarg) followed by arbitrary expression pruning (using an
> > interpreter).
> >
> > Sorry I don't have a great pointer to a high-quality selectivity
> estimator
> > in OSS that uses more advanced stats. Maybe someone else can point to
> one.
> >
> > On Sat, Dec 11, 2021 at 11:43 PM Alessandro Solimando <
> > alessandro.solima...@gmail.com> wrote:
> >
> > > Do you have any code pointer for achieving that, Jacques?
> > >
> > > My main concern is how to estimate the new cost. Do you leverage the
> > > estimation of predicate selectivity over the partitioning expression
> > maybe?
> > >
> > > Il Dom 12 Dic 2021, 05:48 Jacques Nadeau  ha
> > scritto:
> > >
> > > > What we have done in the past is push filters into a scan and alter
> the
> > > > costing (and estimated row count). In cases where the filter or
> > portions
> > > of
> > > > the filter can be applied against partitioning columns, you prune
> > > > partitions and use a new row count estimate/cost estimate based on
> the
> > > > reduced partition set.
> > > >
> > > >
> > > > On Fri, Dec 10, 2021 at 10:25 AM Maxim Gramin <
> mgra...@querifylabs.com
> > >
> > > > wrote:
> > > >
> > > > > I assume that some of the filter conditions (which are involved in
> > the
> > > > > choice of partitions ) may by pushdown'ed to TableScan
> > > > >
> > > > > On Fri, Dec 10, 2021 at 7:29 PM Константин Новиков
> > > > >  wrote:
> > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Given some partitioned storage, we can omit the scan of some
> > > partitions
> > > > > > when a filter is present. How can the lower cost of the scan be
> > > > > > represented? As far as I can tell the current approach only
> allows
> > > > > > providing a single cost for the TableScan and Filter can only add
> > to
> > > > > > that. Should my implementation provide a rule that combines
> > > > > > Filter+TableScan?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Druid test hangs?

[Stamatis Zampetakis]

If there is a systematic way to do it I would be interested to know.

In the past, when I encountered similar hangs in CI what I ended-up doing
is adding debugging commits in the PR with a thread printing stack traces
of other threads at some intervals.

Best,
Stamatis

On Sun, Dec 12, 2021 at 7:00 PM Jacques Nadeau  wrote:

> It could be infra but I'm wondering if it is some kind of concurrency bug.
>
> Anyone know if there is a straightforward way to add a secondary process in
> a github workflow that takes a jstack after an hour or something (if the
> tests run that long). Trying to jump on an instance when this happens and
> do this manually sounds like an effort in frustration.
>
> I guess another option would be to modify the druid job to provide info on
> tests that are running so that we can see if it always locks on the same
> test.
>
> On Sat, Dec 11, 2021 at 11:39 PM Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
>
> > I started noticing that intermittently around a month ago, I had a quick
> > look back then but I could not pinpoint the root cause.
> >
> > I don't think it is expected, and I guess it comes from test infra setup
> > rather than the Calcite code itself.
> >
> > Il Dom 12 Dic 2021, 05:43 Jacques Nadeau  ha
> scritto:
> >
> > > I see a couple of recent builds with Druid tests hanging. Is that a
> > normal
> > > thing or something that has started recently.
> > >
> > > Examples:
> > >
> https://github.com/apache/calcite/runs/4487013505?check_suite_focus=true
> > >
> >
> https://github.com/jacques-n/calcite/runs/4494836558?check_suite_focus=true
> > >
> >
>