CALCITE-3147 | Not Supported

[Ravi Kapoor]

Hi Team,
As per https://issues.apache.org/jira/browse/CALCITE-3147
the triple quoted string for BigQuerySqlDialect should also be
supported in case of query falls in multiple lines.
However, I cannot see it working. Neither do we have a test case
supporting the use case.
Can you please provide some insights on that?

I have a use case where RexLiteral value falls under multiple new
lines in BQ dialect and the
unparsed string generated as shown:

'SELECT wuaht_.report_access_brch_cd as branch_cd,
wuaht_.report_access_brch_nm AS branch_name
 FROM   BRCH_INCTV_PROD_DATA.wire_user_access_hier_1 as wuaht_
 WHERE  wuaht_.system_access_swb_prsn_id = ?
 GROUP BY 1,2;';  -> RexLiteral
which fails in validation.

I can set it as .withLiteralQuoteString(" ''' ") to the context
But that would make all literals wrapped in triple quotes.
Can we wrap it in triple quotes only when drills down to multiple lines?

Thanks,
Ravi

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Julian Hyde]

Merging the repositories is unnecessary, and a massive waste of time.

Any supposed “time savings” due to the the merged repositories will be eaten up 
by dozens of small issues that will crop up long after the merge.

I was release manager for Calcite and Avatica this time around and it took me 
about a week, because the process had changed since last time I was release 
manager. And now we’re going to change the process again? Absolutely fucking 
insane.

Also, we will lose the history of the Avatica source code. All of the commit 
hashes for bugs fixed long ago will change.

At the time that we split the repositories, I was probably against the idea, 
because it was work. Now that we’ve split, I am strongly against recombining 
the repositories.

Honestly, people, we have better things to do.

Julian


> On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis  wrote:
> 
> The two projects have more or less the same release cadence, they're
> maintained by the same community, they're written in the same language, and
> they use the same build system.
> 
> Although I understand the reasons the project was split in the first place,
> at the moment I tend to agree that having them in the same repo would be
> more practical.
> 
> Truth is the merge may require quite a bit of work so we should take this
> into consideration.
> 
> If people prefer keeping the projects as they are I am perfectly fine with
> that as well. I am leaving the decision to those who feel strongly about
> this change.
> 
> Best,
> Stamatis
> 
> 
> 
> On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov  wrote:
> 
>> +1 for a single repo.
>> 
>> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
>> sitnikov.vladi...@gmail.com
>>> :
>> 
>>> Michael> is not proposing to change the
>>> Michael>structure of modules within both projects, merely to have the
>> code
>>> for
>>> Michael>both in a single repository.
>>> 
>>> I propose to integrate them into a single build, and keep the set of the
>>> published jars.
>>> However, the modules and dependency structure could be kept as is.
>>> 
>>> We might want to rename folders like
>>> calcite-core
>>> calcite-linq4j
>>> ..
>>> avatica-core
>>> avatica-server
>>> 
>>> However, I am not sure it is that important to discuss folder names now.
>>> The idea is that as you "open Calcite in IDE, you see both Avatica and
>>> Calcite modules"
>>> 
>>> Michael>Is there any reason we couldn't have a separate release schedule
>> if
>>> Michael>both projects are in the same repository?
>>> 
>>> A different schedule means Calcite must support at least two different
>>> Avatica versions.
>>> In other words, if we allow clients to update Avatica at their will, then
>>> we should allow them building Calcite with different Avatica versions,
>>> which implies Calcite test code should succeed for multiple different
>>> Avatica vesions.
>>> 
>>> That makes it harder to write tests: we have to execute tests with two
>>> different Avatica releases (or even more than two).
>>> 
>>> There are at least two sources for complexity:
>>> 
>>> a) We have to write tests that tolerate multiple versions. For instance,
>>> "if (avatica.18+) {...}" and so on.
>>> That is not really trivial, especially taking into account some of the
>>> tests are created with non-yet-popular
>>> technologies like Quidem where you can't really google solutions. So the
>>> "trivial" task of "making a test to expect two possible outcomes"
>>> becomes less trivial as you try to pass the version from GitHub Action to
>>> Gradle to JUnit to Quidem to no-one-knows-which class.
>>> If we support one Avatica version only, that is not needed. We just patch
>>> the test in Avatica and Calcite and that is it.
>>> Single repo avoids "Gradle vs Quidem" dance.
>>> 
>>> b) If we claim that we support 5 different Guava versions, 3 different
>> JDK
>>> versions, 2 different Avatica versions,
>>> then we have to execute 5*3*2 = 30 combinations of the tests.
>>> That is not really a full matrix, however, things get way easier if we
>>> support one Avatica version only.
>>> The amount of tests we need to do during a proper release is much less,
>> and
>>> it is easier to commit
>>> changes that touch Avatica and Calcite at the same time.
>>> 
>>> 
>>> Vladimir
>>> 
>> 

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Stamatis Zampetakis]

The two projects have more or less the same release cadence, they're
maintained by the same community, they're written in the same language, and
they use the same build system.

Although I understand the reasons the project was split in the first place,
at the moment I tend to agree that having them in the same repo would be
more practical.

Truth is the merge may require quite a bit of work so we should take this
into consideration.

If people prefer keeping the projects as they are I am perfectly fine with
that as well. I am leaving the decision to those who feel strongly about
this change.

Best,
Stamatis



On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov  wrote:

> +1 for a single repo.
>
> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com
> >:
>
> > Michael> is not proposing to change the
> > Michael>structure of modules within both projects, merely to have the
> code
> > for
> > Michael>both in a single repository.
> >
> > I propose to integrate them into a single build, and keep the set of the
> > published jars.
> > However, the modules and dependency structure could be kept as is.
> >
> > We might want to rename folders like
> > calcite-core
> > calcite-linq4j
> > ..
> > avatica-core
> > avatica-server
> >
> > However, I am not sure it is that important to discuss folder names now.
> > The idea is that as you "open Calcite in IDE, you see both Avatica and
> > Calcite modules"
> >
> > Michael>Is there any reason we couldn't have a separate release schedule
> if
> > Michael>both projects are in the same repository?
> >
> > A different schedule means Calcite must support at least two different
> > Avatica versions.
> > In other words, if we allow clients to update Avatica at their will, then
> > we should allow them building Calcite with different Avatica versions,
> > which implies Calcite test code should succeed for multiple different
> > Avatica vesions.
> >
> > That makes it harder to write tests: we have to execute tests with two
> > different Avatica releases (or even more than two).
> >
> > There are at least two sources for complexity:
> >
> > a) We have to write tests that tolerate multiple versions. For instance,
> > "if (avatica.18+) {...}" and so on.
> > That is not really trivial, especially taking into account some of the
> > tests are created with non-yet-popular
> > technologies like Quidem where you can't really google solutions. So the
> > "trivial" task of "making a test to expect two possible outcomes"
> > becomes less trivial as you try to pass the version from GitHub Action to
> > Gradle to JUnit to Quidem to no-one-knows-which class.
> > If we support one Avatica version only, that is not needed. We just patch
> > the test in Avatica and Calcite and that is it.
> > Single repo avoids "Gradle vs Quidem" dance.
> >
> > b) If we claim that we support 5 different Guava versions, 3 different
> JDK
> > versions, 2 different Avatica versions,
> > then we have to execute 5*3*2 = 30 combinations of the tests.
> > That is not really a full matrix, however, things get way easier if we
> > support one Avatica version only.
> > The amount of tests we need to do during a proper release is much less,
> and
> > it is easier to commit
> > changes that touch Avatica and Calcite at the same time.
> >
> >
> > Vladimir
> >
>

Re: How to get the real column name and data type from a SQL view ?

[Alessandro Solimando]

Hi Justin,
the code is indeed recursive, I have never played with too complex queries
so I can't say for sure but I bet it can handle complex cases like the one
you described.

`getExpressionLineage` too could come handy for such tasks.

Armstrong, if possible it would definitely help sharing the snippet of code
you tried with actual and expected output.

Best regards,
Alessandro

On Tue, 9 Nov 2021 at 18:58, Justin Swanhart  wrote:

> I have not looked at the code, so excuse me if this is a terrible question,
> but is this recursive?
>
> Ie:
>
> create view a_view as select a_column from a_table where b_column = 1;
>
> select * from
>  ( -- this could be a view or just a dependent subquery like this
>select cast(sq1_col ...) as a_column from (
>  -- same here, but it is explicitly a view
>  select cast(a_column ...) as sq1_col from a_view
>) sq2
> ) sq1
>
> The metadata for a_column might be completely different in each of the
> views until you get to a_view.  I added a casts here to make that clear.
>
>
>
>
> On Tue, Nov 9, 2021 at 5:35 AM JiaTao Tao  wrote:
>
> > May be you can try this:
> > org.apache.calcite.rel.metadata.RelMetadataQuery#getColumnOrigins
> > But i'm not tested yet.
> > Pls give feedback whether it works or not.
> >
> > Regards!
> >
> > Aron Tao
> >
> >
> > Armstrong  于2021年11月3日周三 下午10:09写道：
> >
> > > Hi guys,
> > >  Thanks for your time to read this help message. I'm new to Calcite
> > and
> > > reading some codes and examples from calcite Github repo. I'm wandering
> > how
> > > to get the column real name and type from view.
> > >  For example, I have some SQL as below.  A kafka topic table left
> > join
> > > MySQL table, and use a View join_result_view represent the join
> result. I
> > > want to get the real column name and data type from this view.
> > >  How to get  result like this ? "user_id", bigint ;  "log_time",
> > bigint
> > >  I already try use SqlParse parse these sqls, and get the SqlNode,
> > but
> > > I can't get the real column name and type. Maybe I missed out the
> correct
> > > method. Hopefully some one can identify the problem, thanks a lot.
> > >
> > > CREATE TABLE user_action_source (
> > > `user_id` STRING,
> > > `item_id` BIGINT,
> > > `bhv_type` STRING,
> > > `bhv_time` INT,
> > > `play_duration` INT,
> > > `ts` BIGINT,
> > > proc_time as PROCTIME()
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_action_log',
> > > 'parallelism' = '5',
> > > 'properties.cluster' = 'asv_cluster',
> > > 'properties.group.id' = 'demo-job',
> > > 'scan.startup.mode' = 'latest-offset',
> > > 'format' = 'json',
> > > 'json.fail-on-missing-field' = 'false',
> > > 'json.ignore-parse-errors' = 'true',
> > > 'scan.manually-commit-offsets-interval' = '5000'
> > > );
> > >
> > > CREATE TABLE item_source (
> > > `item_id` BIGINT,
> > > `video_name` STRING,
> > > `video_count` INT,
> > > `video_duration` INT,
> > > `authors` STRING
> > > ) WITH (
> > > 'connector' = 'jdbc',
> > > 'url' = 'jdbc:mysql://localhost:3306/mydatabase',
> > > 'table-name' = 'item_info'
> > > );
> > >
> > > create view join_result_view as
> > > select a.user_id, a.item_id, a.bhv_type, a.ts as log_time,
> b.video_name,
> > > b.video_count, b.video_duration, b.authors
> > > from user_action_source  as a left join item_source  FOR SYSTEM_TIME AS
> > OF
> > > a.proc_time as b
> > > on a.item_id = b.item_id;
> > >
> >
>

[jira] [Created] (CALCITE-4882) Explore a LambdaMetadataFactory alternative to Janino for metadata retrieval

[Jacques Nadeau (Jira)]

Jacques Nadeau created CALCITE-4882:
---

 Summary: Explore a LambdaMetadataFactory alternative to Janino for 
metadata retrieval
 Key: CALCITE-4882
 URL: https://issues.apache.org/jira/browse/CALCITE-4882
 Project: Calcite
  Issue Type: Improvement
Reporter: Jacques Nadeau






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Ozerov]

+1 for a single repo.

Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov :

> Michael> is not proposing to change the
> Michael>structure of modules within both projects, merely to have the code
> for
> Michael>both in a single repository.
>
> I propose to integrate them into a single build, and keep the set of the
> published jars.
> However, the modules and dependency structure could be kept as is.
>
> We might want to rename folders like
> calcite-core
> calcite-linq4j
> ..
> avatica-core
> avatica-server
>
> However, I am not sure it is that important to discuss folder names now.
> The idea is that as you "open Calcite in IDE, you see both Avatica and
> Calcite modules"
>
> Michael>Is there any reason we couldn't have a separate release schedule if
> Michael>both projects are in the same repository?
>
> A different schedule means Calcite must support at least two different
> Avatica versions.
> In other words, if we allow clients to update Avatica at their will, then
> we should allow them building Calcite with different Avatica versions,
> which implies Calcite test code should succeed for multiple different
> Avatica vesions.
>
> That makes it harder to write tests: we have to execute tests with two
> different Avatica releases (or even more than two).
>
> There are at least two sources for complexity:
>
> a) We have to write tests that tolerate multiple versions. For instance,
> "if (avatica.18+) {...}" and so on.
> That is not really trivial, especially taking into account some of the
> tests are created with non-yet-popular
> technologies like Quidem where you can't really google solutions. So the
> "trivial" task of "making a test to expect two possible outcomes"
> becomes less trivial as you try to pass the version from GitHub Action to
> Gradle to JUnit to Quidem to no-one-knows-which class.
> If we support one Avatica version only, that is not needed. We just patch
> the test in Avatica and Calcite and that is it.
> Single repo avoids "Gradle vs Quidem" dance.
>
> b) If we claim that we support 5 different Guava versions, 3 different JDK
> versions, 2 different Avatica versions,
> then we have to execute 5*3*2 = 30 combinations of the tests.
> That is not really a full matrix, however, things get way easier if we
> support one Avatica version only.
> The amount of tests we need to do during a proper release is much less, and
> it is easier to commit
> changes that touch Avatica and Calcite at the same time.
>
>
> Vladimir
>

Re: How to get the real column name and data type from a SQL view ?

[Justin Swanhart]

I have not looked at the code, so excuse me if this is a terrible question,
but is this recursive?

Ie:

create view a_view as select a_column from a_table where b_column = 1;

select * from
 ( -- this could be a view or just a dependent subquery like this
   select cast(sq1_col ...) as a_column from (
 -- same here, but it is explicitly a view
 select cast(a_column ...) as sq1_col from a_view
   ) sq2
) sq1

The metadata for a_column might be completely different in each of the
views until you get to a_view.  I added a casts here to make that clear.




On Tue, Nov 9, 2021 at 5:35 AM JiaTao Tao  wrote:

> May be you can try this:
> org.apache.calcite.rel.metadata.RelMetadataQuery#getColumnOrigins
> But i'm not tested yet.
> Pls give feedback whether it works or not.
>
> Regards!
>
> Aron Tao
>
>
> Armstrong  于2021年11月3日周三 下午10:09写道：
>
> > Hi guys,
> >  Thanks for your time to read this help message. I'm new to Calcite
> and
> > reading some codes and examples from calcite Github repo. I'm wandering
> how
> > to get the column real name and type from view.
> >  For example, I have some SQL as below.  A kafka topic table left
> join
> > MySQL table, and use a View join_result_view represent the join result. I
> > want to get the real column name and data type from this view.
> >  How to get  result like this ? "user_id", bigint ;  "log_time",
> bigint
> >  I already try use SqlParse parse these sqls, and get the SqlNode,
> but
> > I can't get the real column name and type. Maybe I missed out the correct
> > method. Hopefully some one can identify the problem, thanks a lot.
> >
> > CREATE TABLE user_action_source (
> > `user_id` STRING,
> > `item_id` BIGINT,
> > `bhv_type` STRING,
> > `bhv_time` INT,
> > `play_duration` INT,
> > `ts` BIGINT,
> > proc_time as PROCTIME()
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_action_log',
> > 'parallelism' = '5',
> > 'properties.cluster' = 'asv_cluster',
> > 'properties.group.id' = 'demo-job',
> > 'scan.startup.mode' = 'latest-offset',
> > 'format' = 'json',
> > 'json.fail-on-missing-field' = 'false',
> > 'json.ignore-parse-errors' = 'true',
> > 'scan.manually-commit-offsets-interval' = '5000'
> > );
> >
> > CREATE TABLE item_source (
> > `item_id` BIGINT,
> > `video_name` STRING,
> > `video_count` INT,
> > `video_duration` INT,
> > `authors` STRING
> > ) WITH (
> > 'connector' = 'jdbc',
> > 'url' = 'jdbc:mysql://localhost:3306/mydatabase',
> > 'table-name' = 'item_info'
> > );
> >
> > create view join_result_view as
> > select a.user_id, a.item_id, a.bhv_type, a.ts as log_time, b.video_name,
> > b.video_count, b.video_duration, b.authors
> > from user_action_source  as a left join item_source  FOR SYSTEM_TIME AS
> OF
> > a.proc_time as b
> > on a.item_id = b.item_id;
> >
>

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Sitnikov]

Michael> is not proposing to change the
Michael>structure of modules within both projects, merely to have the code
for
Michael>both in a single repository.

I propose to integrate them into a single build, and keep the set of the
published jars.
However, the modules and dependency structure could be kept as is.

We might want to rename folders like
calcite-core
calcite-linq4j
..
avatica-core
avatica-server

However, I am not sure it is that important to discuss folder names now.
The idea is that as you "open Calcite in IDE, you see both Avatica and
Calcite modules"

Michael>Is there any reason we couldn't have a separate release schedule if
Michael>both projects are in the same repository?

A different schedule means Calcite must support at least two different
Avatica versions.
In other words, if we allow clients to update Avatica at their will, then
we should allow them building Calcite with different Avatica versions,
which implies Calcite test code should succeed for multiple different
Avatica vesions.

That makes it harder to write tests: we have to execute tests with two
different Avatica releases (or even more than two).

There are at least two sources for complexity:

a) We have to write tests that tolerate multiple versions. For instance,
"if (avatica.18+) {...}" and so on.
That is not really trivial, especially taking into account some of the
tests are created with non-yet-popular
technologies like Quidem where you can't really google solutions. So the
"trivial" task of "making a test to expect two possible outcomes"
becomes less trivial as you try to pass the version from GitHub Action to
Gradle to JUnit to Quidem to no-one-knows-which class.
If we support one Avatica version only, that is not needed. We just patch
the test in Avatica and Calcite and that is it.
Single repo avoids "Gradle vs Quidem" dance.

b) If we claim that we support 5 different Guava versions, 3 different JDK
versions, 2 different Avatica versions,
then we have to execute 5*3*2 = 30 combinations of the tests.
That is not really a full matrix, however, things get way easier if we
support one Avatica version only.
The amount of tests we need to do during a proper release is much less, and
it is easier to commit
changes that touch Avatica and Calcite at the same time.


Vladimir

Re: Federated Query

[Yogendra Sharma]

I have not but i will explore it right away.

Get Outlook for Android

From: Charles Givre 
Sent: Tuesday, November 9, 2021 6:12:01 PM
To: dev@calcite.apache.org 
Subject: Re: Federated Query

Yogendra,
Out of curiosity, have you tried Apache Drill?  Drill uses Calcite for query 
planning, but does exactly what you are describing.
Best,
-- C

> On Nov 9, 2021, at 5:31 AM, JiaTao Tao  wrote:
>
> Yes, you can.
> I've done this, twice.
> Once you get the query plan, you can translate the calcite's plan to
> other engines.
> The main work is to define a SQL standard and align type system.
>
> Regards!
>
> Aron Tao
>
>
> Yogendra Sharma  于2021年11月2日周二 上午3:38写道：
>
>> Hi Team,
>>
>> I am exploring Apache Calcite to run federated queries on three different
>> databases. I could not find a working example anywhere on internet.
>>
>> Can you guys confirm if it is possible and if yes, is there an example
>> that you can point me to?
>>
>> Thanks,
>> Yogendra
>>
>>

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Michael Mior]

> There are many reasons to divide code into modules.

Unless I'm misunderstanding, Vladimir is not proposing to change the
structure of modules within both projects, merely to have the code for
both in a single repository.

> to allow separate release schedules

Is there any reason we couldn't have a separate release schedule if
both projects are in the same repository? However, it does seem true
that if we keep a separate release schedule, the benefits of combining
the repositories are less obvious.

I don't really have a strong opinion one way or the other, but I think
the decision of merging repositories and restructuring code or combing
release schedules are two separate decisions.

--
Michael Mior
mm...@apache.org

Le lun. 8 nov. 2021 à 19:43, Julian Hyde  a écrit :
>
> There are many reasons to divide code into modules. To allow separate 
> communities, to allow separate release schedules, to reduce coupling, to make 
> it easier to contribute (because contributors don’t need to understand a 
> large code base), to increase adoption (because people perceive that they can 
> use component A without using component B).
>
> The last of these was particularly on my mind when we split Avatica from 
> Calcite. I was pleased to see, for example, Apache Phoenix using Avatica 
> successfully (including building an ODBC driver) even though their (separate) 
> attempt to adopt Calcite failed.
>
> Splitting code into modules makes it easier to continue to splitting. If 
> Avatica had remained part of Calcite, both written in Java and using the same 
> build system and release process, it’s less likely that Avatica-go would have 
> happened.
>
> I think the split between Avatica and Calcite repositories is working great.
>
> Julian
>
>
> > On Nov 8, 2021, at 3:56 PM, Josh Elser  wrote:
> >
> > These repositories are separate because they have separate release 
> > schedules. Those who were working on calcite.git typically do not want to 
> > be bothered with changes to calcite-avatica.git, and vice versa. Further, 
> > there are downstream users of Avatica directly (without Calcite) who would 
> > be burdened by waiting for a new Calcite release as opposed to a (much more 
> > simple) Avatica release.
> >
> > > Recently there was a PR that improves error messages in Avatica:
> >
> > If I am interpreting your analysis correctly, you are arguing for wanting a 
> > shorter cycle to "implementing" all parts of a change (both in Avatica and 
> > Calcite). The net amount of code you have to write doesn't change with how 
> > it works now compared to how you are suggesting it should work.
> >
> > If there are breaking changes in Avatica, they would need to be accounted 
> > for when Calcite is bumped to the next version of Avatica.
> >
> > In terms of wire compatibility, there has been no breaking wire compat 
> > change that wasn't due to a problem with the protocol itself (where we had 
> > to break it). As far as API compatibility goes, I do not believe we have 
> > ever _had_ to break API compatibility. The untracked runtime changes (like 
> > the one you cite) are a bigger smell to me in Calcite but that's a 
> > different conversation to be had.
> >
> > > * Avatica has fewer commits than Calcite, so having a separate
> > > calcite-avatica repository does not help for segregating PR/issue/commit
> > > queue
> >
> > I disagree with your assessment. Having a separate repository makes it 
> > extremely easy for me to watch Avatica commits/pull-requests which I am 
> > capable of reviewing vs. Calcite pull-requests which I am not comfortable 
> > reviewing.
> >
> > > * calcite-avatica-go seems to reside in its own repository, so I do not 
> > > see
> > > why do we split Java implementations across calcite and calcite-avatica
> > > repository
> >
> > I have no objections to combining these two repositories together.
> >
> > It seems to me that combining these repositories is one possible solution 
> > to address the friction of making changes across these two repositories, 
> > rather than starting from the root problem "How can we make co-dependent 
> > changes easier?"
> >
> > I would challenge us instead of complicating releases (which are already 
> > complicated): what should the API/compatibility surface of Avatica be, and 
> > what how can be make the gaps you've experienced better? Such a fix will go 
> > a long way for downstream users of Avatica, too.
> >
> > - Josh
> >
> > On 11/8/21 2:39 PM, Vladimir Sitnikov wrote:
> >> Hi,
> >> Currently, we have calcite-avatica and calcite in different repositories.
> >> Frankly speaking, I do not know what it brings, however, it does create
> >> points of friction:
> >> 1) If a feature touches Avatica and Calcite, then PRs are hard to create
> >> and maintain. We just can't create a single PR across both repositories
> >> 2) If we support a single Avatica version only in Calcite, then the point
> >> of having different repositories is even mooter.
> >> 3) CI 

Re: DISCUSS: merge calcite-avatica and calcite repositories

[jincheng sun]

Second Vladimir's proposal // Generally speaking，we should decouple the
dependence of the two projects on the master branch, Of course, if you
merge into one repos, the issue of interdependence does not exist.

Best,
Jincheng Sun


Yogendra Sharma  于2021年11月9日周二 下午3:28写道：

> I agree with Vladimir.
>
> Although i am very new here but in my mind having one repository will
> reduce lot of efforts while not necessarily it will reduce flexibility and
> modularity of the the project.
>
> My initial impression is that we can achieve the same modularity
> constraints even after merging the code into one repo.
>
> Get Outlook for Android
>
> 
> From: Vladimir Sitnikov 
> Sent: Tuesday, November 9, 2021, 12:38
> To: Apache Calcite dev list
> Subject: Re: DISCUSS: merge calcite-avatica and calcite repositories
>
> I know the below is too wordy, however, English is not my native language,
> so I tend to overexplain and duplicate thoughts.
>
> Julian>To allow separate communities
>
> This is something I do not understand.
> Let me be explicit: I am OK to maintain bits of avatica code when Calcite
> fixes overlap with Avatica for some reason (e.g. throwing exceptions).
> I am not OK with duplicating the same work just because someone else wants
> to have a separate repository.
>
> I was maintaining and releasing multiple (four!) repositories for pgjdbc
> when it was using Maven
> (Maven was not able to release jre6, jre7, and jre8 artifacts from within a
> single repository), and it was a big relief when the code was recollected
> into a single repository again.
>
> That is why I am advocating for merging avatica code back.
>
> Julian>Splitting code into modules makes it easier to continue to splitting
>
> I agree it is worth having calcite-core and avatica as separate jar
> artifacts.
> It is even worth splitting "enumerable" out of calcite-core into its own
> module.
> However, I do not see how splitting the repositories helps. I do not expect
> somebody to use calcite-avatica as a git submodule or something like that.
>
> Julian>to allow separate release schedules
>
> Here are the recent releases.
> There's often a pattern that "Calcite release follows Avatica release".
> It is even mentioned in the mails where someone suggests releasing Avatica,
> then releasing Calcite".
>
> A notable exception was rel/avatica-1.16.0 which was not connected to the
> release of Calcite.
> The rest Avatica releases seem to be quite close to Calcite releases, so I
> do not see what do we gain from "separate release schedules".
> It looks like co-locating the release would be easier for both maintenance
> overheads, testing, and consumption.
>
>
> 2016-03-16 rel/calcite-avatica-1.7.0
>
> 2016-03-17 calcite-1.7.0
>
> 2016-03-15 rel/calcite-avatica-1.7.1
>
> 2016-06-01 rel/calcite-avatica-1.8.0
>
> 2016-06-08 calcite-1.8.0
>
>
> 2016-09-17 calcite-1.9.0
>
>
> 2016-10-07 calcite-1.10.0
>
> 2016-10-27 rel/calcite-avatica-1.9.0
>
>
> 2017-01-03 calcite-1.11.0
>
> 2017-03-20 calcite-1.12.0
>
> 2017-05-23 rel/avatica-1.10.0
>
> 2017-06-22 calcite-1.13.0
>
>
> 2017-09-27 calcite-1.14.0
>
> 2017-12-05 calcite-1.15.0
>
> 2018-03-06 rel/avatica-1.11.0
>
> 2018-03-14 calcite-1.16.0
>
> 2018-06-21 rel/avatica-1.12.0
>
> 2018-07-16 calcite-1.17.0
>
> 2018-11-29 rel/avatica-1.13.0
>
> 2018-12-18 calcite-1.18.0
>
> 2019-03-20 calcite-1.19.0
>
> 2019-04-25 rel/avatica-1.14.0
>
> 2019-05-09 rel/avatica-1.15.0
>
> 2019-06-19 calcite-1.20.0
>
>
> 2019-09-06 calcite-1.21.0
>
> 2019-12-19 rel/avatica-1.16.0
>
>
> 2020-05-23 calcite-1.23.0
>
> 2020-06-22 rel/avatica-1.17.0
>
> 2020-07-23 calcite-1.24.0
> 2020-08-22 calcite-1.25.0
>
> 2020-10-06 calcite-1.26.0
>
>
> 2021-05-18 rel/avatica-1.18.0
>
> 2021-06-03 calcite-1.27.0
>
> 2021-10-04 rel/avatica-1.19.0
>
> 2021-10-19 calcite-1.28.0
>
> Julian>The last of these was particularly on my mind when we split Avatica
> from
> Calcite.
> Julian>I was pleased to see, for example, Apache Phoenix using Avatica
> successfully
> (including building an ODBC driver)
>
> Does it really help Phonenix that we split repositories?
> I am sure they should not really depend on the repositories.
> How would their life be harder if we release the same set of artifacts and
> jars from a single source tree?
>
> Julian>If Avatica had remained part of Calcite, both written in Java and
> using the same build system and release process,
> Julian>it’s less likely that Avatica-go would have happened
>
> Does Avatica-go use calcite-avatica repository?
> So far it looks like Avatica-go uses
> https://github.com/apache/calcite-avatica-go, and I do not suggest merging
> it into calcite repository yet.
> I do not see how merging calcite-avatica and calcite repositories would
> block the evolution or emergence of avatica-go, avatica-rust, etc.
>
> Julian>release process
>
> I identified https://issues.apache.org/jira/browse/CALCITE-4881 "Calcite
> release tags should have rel/... prefix 

Re: Federated Query

[Charles Givre]

Yogendra, 
Out of curiosity, have you tried Apache Drill?  Drill uses Calcite for query 
planning, but does exactly what you are describing.  
Best,
-- C

> On Nov 9, 2021, at 5:31 AM, JiaTao Tao  wrote:
> 
> Yes, you can.
> I've done this, twice.
> Once you get the query plan, you can translate the calcite's plan to
> other engines.
> The main work is to define a SQL standard and align type system.
> 
> Regards!
> 
> Aron Tao
> 
> 
> Yogendra Sharma  于2021年11月2日周二 上午3:38写道：
> 
>> Hi Team,
>> 
>> I am exploring Apache Calcite to run federated queries on three different
>> databases. I could not find a working example anywhere on internet.
>> 
>> Can you guys confirm if it is possible and if yes, is there an example
>> that you can point me to?
>> 
>> Thanks,
>> Yogendra
>> 
>> 

Re: Extending Grammar Mapping

[JiaTao Tao]

It seems you needn't create a new "SqlCall"
What's the problem you're having?

Regards!

Aron Tao


Jeremy Dyer  于2021年11月4日周四 上午5:26写道：

> Hello,
>
> I am working on a custom parser grammar that extends
> "PostgresqlSqlDialect". I have a simple case where I want to allow my
> grammar to accept the token "SUBSTR" and have it behave exactly like
> "SUBSTRING" would. Is there a simple way to map this behavior without
> having to create a new "SqlCall" extended class and correlating ftl files?
>
> - Jeremy Dyer
>

Re: How to get the real column name and data type from a SQL view ?

[JiaTao Tao]

May be you can try this:
org.apache.calcite.rel.metadata.RelMetadataQuery#getColumnOrigins
But i'm not tested yet.
Pls give feedback whether it works or not.

Regards!

Aron Tao


Armstrong  于2021年11月3日周三 下午10:09写道：

> Hi guys,
>  Thanks for your time to read this help message. I'm new to Calcite and
> reading some codes and examples from calcite Github repo. I'm wandering how
> to get the column real name and type from view.
>  For example, I have some SQL as below.  A kafka topic table left join
> MySQL table, and use a View join_result_view represent the join result. I
> want to get the real column name and data type from this view.
>  How to get  result like this ? "user_id", bigint ;  "log_time", bigint
>  I already try use SqlParse parse these sqls, and get the SqlNode, but
> I can't get the real column name and type. Maybe I missed out the correct
> method. Hopefully some one can identify the problem, thanks a lot.
>
> CREATE TABLE user_action_source (
> `user_id` STRING,
> `item_id` BIGINT,
> `bhv_type` STRING,
> `bhv_time` INT,
> `play_duration` INT,
> `ts` BIGINT,
> proc_time as PROCTIME()
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_action_log',
> 'parallelism' = '5',
> 'properties.cluster' = 'asv_cluster',
> 'properties.group.id' = 'demo-job',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'json',
> 'json.fail-on-missing-field' = 'false',
> 'json.ignore-parse-errors' = 'true',
> 'scan.manually-commit-offsets-interval' = '5000'
> );
>
> CREATE TABLE item_source (
> `item_id` BIGINT,
> `video_name` STRING,
> `video_count` INT,
> `video_duration` INT,
> `authors` STRING
> ) WITH (
> 'connector' = 'jdbc',
> 'url' = 'jdbc:mysql://localhost:3306/mydatabase',
> 'table-name' = 'item_info'
> );
>
> create view join_result_view as
> select a.user_id, a.item_id, a.bhv_type, a.ts as log_time, b.video_name,
> b.video_count, b.video_duration, b.authors
> from user_action_source  as a left join item_source  FOR SYSTEM_TIME AS OF
> a.proc_time as b
> on a.item_id = b.item_id;
>

Re: Federated Query

[JiaTao Tao]

Yes, you can.
I've done this, twice.
Once you get the query plan, you can translate the calcite's plan to
other engines.
The main work is to define a SQL standard and align type system.

Regards!

Aron Tao


Yogendra Sharma  于2021年11月2日周二 上午3:38写道：

> Hi Team,
>
> I am exploring Apache Calcite to run federated queries on three different
> databases. I could not find a working example anywhere on internet.
>
>  Can you guys confirm if it is possible and if yes, is there an example
> that you can point me to?
>
> Thanks,
> Yogendra
>
>