Re: [DISCUSS] Towards Calcite 1.31.0

[Andrei Sereda]

Hi Julian,

I think it is a great idea.

Please take a look at proposed release notes:
https://github.com/apache/calcite/pull/2858

Thanks,
Andrei.


On Wed, Jul 20, 2022 at 4:31 PM Julian Hyde  wrote:

> Andrei,
>
> One thing that has worked well is for the RM to create a PR with the
> release notes. People can then review the release notes, comment on that
> PR, and add their own release notes. As RM you would not submit the PR, but
> instead manually rebase onto the final RC.
>
> Julian
>
>
> > On Jul 19, 2022, at 6:44 PM, Andrei Sereda  wrote:
> >
> > Hello,
> >
> > Anything specific you'd like to be added to the release notes of 1.31 ?
> > If so, please provide a short summary as well as JIRA id.
> >
> >
> > Andrei.
> >
> > On Tue, Jul 12, 2022 at 10:07 PM Andrei Sereda 
> wrote:
> >
> >> Hi Julian,
> >>
> >>
> >>> Andrei, Are there any remaining blockers for 1.31?
> >> No blockers for 1.31. Any pending issues can be moved towards 1.32.
> >>
> >>> Can we proceed with an RC0?
> >>
> >> Yes, I'll start the release very soon.
> >>
> >> Thanks for fixing the regression.
> >>
> >> Andrei.
> >>
> >>
> >> On Tue, Jul 12, 2022 at 4:35 PM Julian Hyde 
> >> wrote:
> >>
> >>> I merged https://issues.apache.org/jira/browse/CALCITE-5194 <
> >>> https://issues.apache.org/jira/browse/CALCITE-5194>, the fix for the
> >>> regression due to https://issues.apache.org/jira/browse/CALCITE-35 <
> >>> https://issues.apache.org/jira/browse/CALCITE-35>, yesterday.
> >>>
> >>> Andrei, Are there any remaining blockers for 1.31? Can we proceed with
> an
> >>> RC0?
> >>>
> >>> Julian
> >>>
> >>>
>  On Jun 21, 2022, at 11:29 PM, Yanjing Wang  >
> >>> wrote:
> 
>  Thanks Andrei, now there is nobody to review these PRs, It's ok to
> >>> change
>  to 1.32，I think pr-2838 
> >>> for
>  CALCITE-5045 
> >>> should be
>  merged before 1.31, because it's simple and Yingyu Wang  encountered
> >>> this
>  problem also.
> 
>  Andrei Sereda  于2022年6月22日周三 10:06写道：
> 
> > Julian, how much time do you think is necessary to fix CALCITE-35
> > regression ?
> >
> > Yanjing, I see some discussions/attempts in JIRA/github to review the
> >>> PRs.
> > Do you think it can still be reviewed and merged before 1.31 ?
> >
> > Viliam (and Julian), thanks for fixing (and merging) CALCITE-5157
> >
> > Dmitry (and Stamatis / Ruben), thanks for fixing (and merging)
> > CALCITE-5134.
> > As I understand CALCITE-5127 is delayed until 1.32 ?
> >
> >
> >
> > On Tue, Jun 21, 2022 at 2:57 AM Julian Hyde 
> > wrote:
> >
> >> I agree that it’s a regression, and caused by my CALCITE-35 change.
> >>> I’ve
> >> assigned the bug to myself. I’m not sure I have time to fix it this
> >>> week,
> >> so we can back out my change if necessary to get the release out on
> >> schedule. If someone else can fix it I would be grateful.
> >>
> >>
> >>> On Jun 19, 2022, at 2:26 AM, Vova Vysotskyi 
> >> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I have found the following regression while verifying the new
> release
> > to
> >> work with Apache Drill:
> > https://issues.apache.org/jira/browse/CALCITE-5194
> >> .
> >>> This issue appeared after changes for
> >> https://issues.apache.org/jira/browse/CALCITE-35.
> >>> Since it is a regression, I suggest treating it as a blocker for
> the
> >> upcoming release.
> >>>
> >>> Kind regards,
> >>> Volodymyr Vysotskyi
> >>>
> >>> On 2022/06/16 06:38:45 Viliam Durina wrote:
>  I'll try to work on comments in CALCITE-5157
>   today
> 
>  On Wed, 15 Jun 2022 at 07:30, Yanjing Wang <
> >>> zhuangzixiao...@gmail.com
> >>
>  wrote:
> 
> > Hi Andrei,
> > The followings need review process to merge.
> > https://issues.apache.org/jira/browse/CALCITE-4512
> > https://issues.apache.org/jira/browse/CALCITE-5045
> > https://issues.apache.org/jira/browse/CALCITE-5043
> > https://issues.apache.org/jira/browse/CALCITE-4987
> > It would be great if they were fixed in this release.
> >
> > Dmitry Sysolyatin  于2022年6月14日周二
> 17:18写道：
> >
> >> Hi!
> >> It would be good to merge:
> >> https://issues.apache.org/jira/browse/CALCITE-5134
> >> https://issues.apache.org/jira/browse/CALCITE-5127
> >>
> >> Both of those issues are related to correct execution queries
> with
> >> subqueries.
> >>
> >> On Tue, Jun 14, 2022 at 6:59 AM Andrei Sereda <
> aser...@gmail.com>
> >> wrote:
> >>
> >>> Thanks to all who reviewed the PRs for 1.31 release.
> >>>
> >>> There are about 16 issues remaining 

Re: Requesting Information Regarding Data Federation

[Charles Givre]

Hi Pranav, 
You might want to take a look at Apache Drill, as it uses Calcite as a query 
planner and can executed federated queries against a pretty wide array of data 
sets.
Best, 
-- C

> On Jul 20, 2022, at 6:56 PM, Pranav Deshpande  
> wrote:
> 
> Dear Apache Calcite Team,
> I am trying to learn Calcite and wish to build a poc for data federation.
> 
> In the video here, https://www.youtube.com/watch?v=4JAOkLKrcYE, somehow the
> presenter and his team managed to squash parts of the Relational Nodes into
> "Spark Tables" and then Spark handled the execution of those.
> 
> How do I exactly go about doing this?
> 
> As per this discussion I understand that one has to create a RelOptRule to
> do the same.
> 
> Also, one has to somehow define the cost (I don't know how to do this).
> 
> Is there a simple tutorial which demonstrates the basics of this? Like some
> kind of simple implementation with ListTable etc.
> 
> Thanks & Regards,
> Pranav

Requesting Information Regarding Data Federation

[Pranav Deshpande]

Dear Apache Calcite Team,
I am trying to learn Calcite and wish to build a poc for data federation.

In the video here, https://www.youtube.com/watch?v=4JAOkLKrcYE, somehow the
presenter and his team managed to squash parts of the Relational Nodes into
"Spark Tables" and then Spark handled the execution of those.

How do I exactly go about doing this?

As per this discussion I understand that one has to create a RelOptRule to
do the same.

Also, one has to somehow define the cost (I don't know how to do this).

Is there a simple tutorial which demonstrates the basics of this? Like some
kind of simple implementation with ListTable etc.

Thanks & Regards,
Pranav

Re: "Proxying" schema that intercepts calls to members and returns stub values?

[Julian Hyde]

That is probably very straightforward to write.

The nearest thing that currently exists is CloneSchema, which I would describe 
as a ‘caching proxy’. You could copy it and make each table return a single row 
of all ones. Or I would be inclined to return the ‘zero’ value for each 
datatype (0 for ints, false for BOOLEAN, 0.0 for floats, empty string, DATE 
‘1970-1-1’, etc.).

Feel free to create a JIRA case sketching out the requirement, even if you 
don’t currently have time to implement it.

Julian


> On Jul 18, 2022, at 3:09 PM, Gavin Ray  wrote:
> 
> How difficult would it be to create a proxying schema that returns the
> integer value 1 for every column requested
> 
> This would be useful for a few things probably, but one neat use it would
> unlock is
> the ability to use Calcite for translation between SQL dialects without
> having a schema beforehand.
> 
> You could even host a small public webservice people could use, how nifty
> would that be?

Re: [DISCUSS] Towards Calcite 1.31.0

[Julian Hyde]

Andrei,

One thing that has worked well is for the RM to create a PR with the release 
notes. People can then review the release notes, comment on that PR, and add 
their own release notes. As RM you would not submit the PR, but instead 
manually rebase onto the final RC.

Julian


> On Jul 19, 2022, at 6:44 PM, Andrei Sereda  wrote:
> 
> Hello,
> 
> Anything specific you'd like to be added to the release notes of 1.31 ?
> If so, please provide a short summary as well as JIRA id.
> 
> 
> Andrei.
> 
> On Tue, Jul 12, 2022 at 10:07 PM Andrei Sereda  wrote:
> 
>> Hi Julian,
>> 
>> 
>>> Andrei, Are there any remaining blockers for 1.31?
>> No blockers for 1.31. Any pending issues can be moved towards 1.32.
>> 
>>> Can we proceed with an RC0?
>> 
>> Yes, I'll start the release very soon.
>> 
>> Thanks for fixing the regression.
>> 
>> Andrei.
>> 
>> 
>> On Tue, Jul 12, 2022 at 4:35 PM Julian Hyde 
>> wrote:
>> 
>>> I merged https://issues.apache.org/jira/browse/CALCITE-5194 <
>>> https://issues.apache.org/jira/browse/CALCITE-5194>, the fix for the
>>> regression due to https://issues.apache.org/jira/browse/CALCITE-35 <
>>> https://issues.apache.org/jira/browse/CALCITE-35>, yesterday.
>>> 
>>> Andrei, Are there any remaining blockers for 1.31? Can we proceed with an
>>> RC0?
>>> 
>>> Julian
>>> 
>>> 
 On Jun 21, 2022, at 11:29 PM, Yanjing Wang 
>>> wrote:
 
 Thanks Andrei, now there is nobody to review these PRs, It's ok to
>>> change
 to 1.32，I think pr-2838 
>>> for
 CALCITE-5045 
>>> should be
 merged before 1.31, because it's simple and Yingyu Wang  encountered
>>> this
 problem also.
 
 Andrei Sereda  于2022年6月22日周三 10:06写道：
 
> Julian, how much time do you think is necessary to fix CALCITE-35
> regression ?
> 
> Yanjing, I see some discussions/attempts in JIRA/github to review the
>>> PRs.
> Do you think it can still be reviewed and merged before 1.31 ?
> 
> Viliam (and Julian), thanks for fixing (and merging) CALCITE-5157
> 
> Dmitry (and Stamatis / Ruben), thanks for fixing (and merging)
> CALCITE-5134.
> As I understand CALCITE-5127 is delayed until 1.32 ?
> 
> 
> 
> On Tue, Jun 21, 2022 at 2:57 AM Julian Hyde 
> wrote:
> 
>> I agree that it’s a regression, and caused by my CALCITE-35 change.
>>> I’ve
>> assigned the bug to myself. I’m not sure I have time to fix it this
>>> week,
>> so we can back out my change if necessary to get the release out on
>> schedule. If someone else can fix it I would be grateful.
>> 
>> 
>>> On Jun 19, 2022, at 2:26 AM, Vova Vysotskyi 
>> wrote:
>>> 
>>> Hello,
>>> 
>>> I have found the following regression while verifying the new release
> to
>> work with Apache Drill:
> https://issues.apache.org/jira/browse/CALCITE-5194
>> .
>>> This issue appeared after changes for
>> https://issues.apache.org/jira/browse/CALCITE-35.
>>> Since it is a regression, I suggest treating it as a blocker for the
>> upcoming release.
>>> 
>>> Kind regards,
>>> Volodymyr Vysotskyi
>>> 
>>> On 2022/06/16 06:38:45 Viliam Durina wrote:
 I'll try to work on comments in CALCITE-5157
  today
 
 On Wed, 15 Jun 2022 at 07:30, Yanjing Wang <
>>> zhuangzixiao...@gmail.com
>> 
 wrote:
 
> Hi Andrei,
> The followings need review process to merge.
> https://issues.apache.org/jira/browse/CALCITE-4512
> https://issues.apache.org/jira/browse/CALCITE-5045
> https://issues.apache.org/jira/browse/CALCITE-5043
> https://issues.apache.org/jira/browse/CALCITE-4987
> It would be great if they were fixed in this release.
> 
> Dmitry Sysolyatin  于2022年6月14日周二 17:18写道：
> 
>> Hi!
>> It would be good to merge:
>> https://issues.apache.org/jira/browse/CALCITE-5134
>> https://issues.apache.org/jira/browse/CALCITE-5127
>> 
>> Both of those issues are related to correct execution queries with
>> subqueries.
>> 
>> On Tue, Jun 14, 2022 at 6:59 AM Andrei Sereda 
>> wrote:
>> 
>>> Thanks to all who reviewed the PRs for 1.31 release.
>>> 
>>> There are about 16 issues remaining tagged for 1.31 (list
>>> <
>>> 
>> 
> 
>> 
> 
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20fixVersion%20%3D%201.31.0%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22In%20Review%22%2C%20%22In%20Implementation%22)%20ORDER%20BY%20priority%20DESC
 
>>> ).
>>> 
>>> Please let me know which ones are important for this release (and
>> I'll
>>> 

Understanding the MULTISET constructor

[Cyril de Catheu]

Hello Calcite community,

I'm trying to hardcode a MULTISET manually in a query.
I could not find a way to do this in the doc about value constructors
.
But I saw the row/map/array constructors (eg ARRAY/[val1, val2, ...]) and I
found that MULTISET[val1, ...] seems to work.

Q1 - Is this the correct way to create a MULTISET?

When creating a multiset this way, I have a strange behavior with Strings.
Q2 - It seems the strings are padded to the longest string in the MULTISET
? Is this the expected behavior?

Eg MULTISET["a", "abcd" ] becomes MULTISET["a   ", "abcd" ]. (notice the
spaces after a)

To reproduce in sqline: (the connection used is not important: this is just
a copy/paste of the tutorial)

cd calcite/example/csv
./sqlline
!connect jdbc:calcite:model=src/test/resources/model.json admin admin

select
MULTISET['dev', 'a'] MULTISET INTERSECT MULTISET['lol','dev'],
MULTISET['dev', 'ab'] MULTISET INTERSECT MULTISET['lol','dev'],
MULTISET['dev', 'abc'] MULTISET INTERSECT MULTISET['lol','dev'],
MULTISET['dev', 'abcd'] MULTISET INTERSECT MULTISET['lol','dev'],
MULTISET['dev', 'abcde'] MULTISET INTERSECT MULTISET['lol','dev']
;

++++++
| EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 | EXPR$4 |
++++++
| [dev]  | [dev]  | [dev]  | [] | [] |

--> intersection is broken due to the padding

Looking at debug logs I do see the padding:
org.apache.calcite.sql.parser - Reduced MULTISET['dev', 'abcde'] MULTISET
INTERSECT ALL MULTISET['lol', 'dev'] *#dev is not padded yet*
org.apache.calcite.sql.parser - Reduced (MULTISET['dev', 'abcde'] MULTISET
INTERSECT ALL MULTISET['lol', 'dev']) IS NOT EMPTY
org.apache.calcite.sql2rel - Plan after converting SqlNode to RelNode
LogicalProject(_ID=[$0])
  LogicalFilter(condition=[IS NOT EMPTY(MULTISET INTERSECT ALL($SLICE($3),
$SLICE($4)))])
LogicalJoin(condition=[true], joinType=[inner])
  LogicalJoin(condition=[true], joinType=[inner])
LogicalTableScan(table=[[objects, objects]])
Collect(field=[EXPR$0])
  LogicalValues(tuples=[[{ 'dev  ' }, { 'abcde' }]])  *#dev is
padded with space here*
  Collect(field=[EXPR$0])
LogicalValues(tuples=[[{ 'lol' }, { 'dev' }]])

Am I missing something?

Thanks for reading.
Cyril de Catheu

-- 

[image: StarTree] 
Cyril de Catheu
Software Engineer
+33 684-829-908 <+33+684-829-908>
Follow us: [image: RSS] [image: LinkedIn]
[image: Twitter]
[image: Slack]
[image:
YouTube] 

[image: Try StarTree Cloud Today]