Progress on adding TUMBLE as table value function (CALCITE-3272)

[Rui Wang]

Hi community,

I have created a new PR ( https://github.com/apache/calcite/pull/1587) to
demonstrate the progress of TUMBLE table value function (CALCITE-3272).
Julian suggested me to have a working version that adds a stream.iq and
have an enumerable implementation. Those are in the PR.

High level speaking, the PR is adding a support of the following:

SELECT *
FROM TABLE(Tumble(
  TABLE ORDERS ,
  'ROWTIME' ,
 INTERVAL '1' MINUTES))


One missing feature so far is adding support of DESCRIPTOR, which is
intentionally cut off from the PR because that will make the PR more
complicated. Thus DESCRIPTOR is left as future work.

The PR solves not only CALCITE-3272, but also it's blockers:
https://jira.apache.org/jira/browse/CALCITE-3340
https://jira.apache.org/jira/browse/CALCITE-3501
https://jira.apache.org/jira/browse/CALCITE-3499
https://jira.apache.org/jira/browse/CALCITE-3418


I will probably need some guidance on how to proceed to get the PR merged.
Please let me know if you have any thoughts.

-Rui

Re: [DISCUSS] Towards Avatica 1.16.0

[Francis Chuang]

Good idea. I think other outdated dependencies (if any) can also be 
updated in this change as well.


On 15/11/2019 3:04 pm, Julian Hyde wrote:

Can we also upgrade Jetty? I tried to upgrade Calcite [1] but it gets runtime 
errors because Avatica is on an earlier version.

Julian

[1] https://github.com/apache/calcite/pull/1580 



On Nov 11, 2019, at 12:01 PM, Vladimir Sitnikov  
wrote:

The list you provide is outdated.


these plugins weren't added to the Gradle build at
some point.


Please feel free to execute "./gradlew tasks" and check what is available

To my best knowledge, the build script is good enough.

The missing items are

* "Unused dependency"
* "Used but undeclared dependency"


That, however, is not significant for Avatica since the number of
dependencies is low.
"unused dependency" and "used but undeclared" plugin for Gradle does not
exist yet.
We need "java-library" support, and that is a missing feature (
https://github.com/wfhartford/gradle-dependency-analyze/issues/39)

Vladimir

Re: [DISCUSS] Towards Avatica 1.16.0

[Julian Hyde]

Can we also upgrade Jetty? I tried to upgrade Calcite [1] but it gets runtime 
errors because Avatica is on an earlier version.

Julian

[1] https://github.com/apache/calcite/pull/1580 
 

> On Nov 11, 2019, at 12:01 PM, Vladimir Sitnikov  
> wrote:
> 
> The list you provide is outdated.
> 
>> these plugins weren't added to the Gradle build at
>> some point.
> 
> Please feel free to execute "./gradlew tasks" and check what is available
> 
> To my best knowledge, the build script is good enough.
> 
> The missing items are
>> * "Unused dependency"
>> * "Used but undeclared dependency"
> 
> That, however, is not significant for Avatica since the number of
> dependencies is low.
> "unused dependency" and "used but undeclared" plugin for Gradle does not
> exist yet.
> We need "java-library" support, and that is a missing feature (
> https://github.com/wfhartford/gradle-dependency-analyze/issues/39)
> 
> Vladimir

Calcite-Master - Build # 1436 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1436)

Status: Failure

Check console output at https://builds.apache.org/job/Calcite-Master/1436/ to 
view the results.

[jira] [Created] (CALCITE-3506) Avatica: exception in the writetoProtoWithType function

[Enrique Saurez (Jira)]

Enrique Saurez created CALCITE-3506:
---

 Summary: Avatica: exception in the writetoProtoWithType function
 Key: CALCITE-3506
 URL: https://issues.apache.org/jira/browse/CALCITE-3506
 Project: Calcite
  Issue Type: Bug
  Components: avatica
Affects Versions: avatica-1.12.0
Reporter: Enrique Saurez


 am using Apache Calcite-Avatica version 1.12 (but the relevant code
 sections are not different from the master branch), and I am getting
 the following exception on the client side (but the actual error in on
 the server side):
 
||Exception||Heading 2||
|org.apache.calcite.avatica.AvaticaSqlException: Error -1 (0) :
 Remote driver error: ClassCastException: java.lang.Long cannot be cast
 to java.lang.Float
         at org.apache.calcite.avatica.Helper.createException(Helper.java:54)
         at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
         at 
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:557)
         at 
org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137)
         at 
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.getCustomerByName(Payment.java:400)
         at 
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.run(Payment.java:221)
         at 
com.oltpbenchmark.benchmarks.tpcc.TPCCWorker.executeWork(TPCCWorker.java:74)
         at com.oltpbenchmark.api.Worker.doWork(Worker.java:386)
         at com.oltpbenchmark.api.Worker.run(Worker.java:296)
         at java.lang.Thread.run(Thread.java:748)
 java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Float
         at 
org.apache.calcite.avatica.remote.TypedValue.writeToProtoWithType(TypedValue.java:594)
         at 
org.apache.calcite.avatica.remote.TypedValue.toProto(TypedValue.java:799)
         at org.apache.calcite.avatica.Meta$Frame.serializeScalar(Meta.java:985)
         at org.apache.calcite.avatica.Meta$Frame.parseColumn(Meta.java:971)
         at org.apache.calcite.avatica.Meta$Frame.toProto(Meta.java:936)
         at 
org.apache.calcite.avatica.remote.Service$ResultSetResponse.serialize(Service.java:841)
         at 
org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1158)
         at 
org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1113)
         at 
org.apache.calcite.avatica.remote.ProtobufTranslationImpl.serializeResponse(ProtobufTranslationImpl.java:348)
         at 
org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:57)
         at 
org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:31)
         at 
org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:95)
         at 
org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
         at 
org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
         at 
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.Server.handle(Server.java:499)
         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
         at 
[org.eclipse.jetty.io|http://org.eclipse.jetty.io/].AbstractConnection$2.run(AbstractConnection.java:544)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
         at java.lang.Thread.run(Thread.java:748)|Col A2|


 From the code, it seems like when the function "writeToProtoWithType"
 is called from "toProto", there is a boxing conversion from long
 (primitive) to Long (object), this is because
 for floats, "toProto" is using "((Float) o).longValue()" which returns
 a long. Then in "writeToProtoWithType" is being casted to float, which
 I think causes the CastClassException.
 
 If I add this code to the testFloat() function in the
 "core/src/test/java/org/apache/calcite/avatica/remote/TypedValueTest.java"
 file:
 
{code:java}
Common.TypedValue.Builder builder = Common.TypedValue.newBuilder();
 Common.Rep val = TypedValue.toProto(builder, Float.valueOf(3.14159f));
 Common.TypedValue typedVal = builder.build();{code}
 

it replicates the exception. I am not sure why are you using the longValue() on 
the float within the toProto function? Doesn't this cast lose information? 
Please let me know if you need any more information. I send an email to the dev 
mailing list, but somebody suggested to create the issue here.


 Thanks a lot for help!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Avatica: exception in the writetoProtoWithType function

[Michael Mior]

Unfortunately I'm not too familiar with Avatica, but I would suggest
that you file a JIRA issue so discussion can continue there. Thanks!
Regards

https://issues.apache.org/jira/projects/CALCITE/issues

--
Michael Mior
mm...@apache.org

Le jeu. 14 nov. 2019 à 13:20, Enrique Saurez  a écrit :
>
> Hi!
>
> I am using Apache Calcite-Avatica version 1.12 (but the relevant code
> sections are not different from the master branch), and I am getting
> the following exception on the client side (but the actual error in on
> the server side):
>
> org.apache.calcite.avatica.AvaticaSqlException: Error -1 (0) :
> Remote driver error: ClassCastException: java.lang.Long cannot be cast
> to java.lang.Float
> at org.apache.calcite.avatica.Helper.createException(Helper.java:54)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at 
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:557)
> at 
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137)
> at 
> com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.getCustomerByName(Payment.java:400)
> at 
> com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.run(Payment.java:221)
> at 
> com.oltpbenchmark.benchmarks.tpcc.TPCCWorker.executeWork(TPCCWorker.java:74)
> at com.oltpbenchmark.api.Worker.doWork(Worker.java:386)
> at com.oltpbenchmark.api.Worker.run(Worker.java:296)
> at java.lang.Thread.run(Thread.java:748)
> java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Float
> at 
> org.apache.calcite.avatica.remote.TypedValue.writeToProtoWithType(TypedValue.java:594)
> at 
> org.apache.calcite.avatica.remote.TypedValue.toProto(TypedValue.java:799)
> at 
> org.apache.calcite.avatica.Meta$Frame.serializeScalar(Meta.java:985)
> at org.apache.calcite.avatica.Meta$Frame.parseColumn(Meta.java:971)
> at org.apache.calcite.avatica.Meta$Frame.toProto(Meta.java:936)
> at 
> org.apache.calcite.avatica.remote.Service$ResultSetResponse.serialize(Service.java:841)
> at 
> org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1158)
> at 
> org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1113)
> at 
> org.apache.calcite.avatica.remote.ProtobufTranslationImpl.serializeResponse(ProtobufTranslationImpl.java:348)
> at 
> org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:57)
> at 
> org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:31)
> at 
> org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:95)
> at 
> org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
> at 
> org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
> at 
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
>
> From the code, it seems like when the function "writeToProtoWithType"
> is called from "toProto", there is a boxing conversion from long
> (primitive) to Long (object), this is because
> for floats, "toProto" is using "((Float) o).longValue()" which returns
> a long. Then in "writeToProtoWithType" is being casted to float, which
> I think causes the CastClassException.
>
> If I add this code to the testFloat() function in the
> "core/src/test/java/org/apache/calcite/avatica/remote/TypedValueTest.java"
> file:
>
> Common.TypedValue.Builder builder = Common.TypedValue.newBuilder();
> Common.Rep val = TypedValue.toProto(builder, Float.valueOf(3.14159f));
> Common.TypedValue typedVal = builder.build();
>
> it replicates the exception. I have a couple of questions:
>
> 1. Why are you using the longValue() on the float within the toProto
> function? Doesn't this cast lose information?
> 2. What kind of code would trigger this kind of behavior? It is hard
> for me to debug in detail, because it is only trigger when I running
> it in my remote server. If I have some idea of how this can be
> trigger, maybe I can find an example on my server code that is
> triggering it.
> 3. Is this a known problem? I couldn't find 

Re: CALCITE-2905: Maven -> Gradle: any thoughts

[Vladimir Sitnikov]

The Gradle build scripts are ready (
https://github.com/apache/calcite/pull/1571 )
There might be minor issues like "a resource file missing in one of the
jars",
however, I expect there will be no more major changes.


Unfortunately, there are changes that prevent building with both Maven and
Gradle from the same commit.
For instance, Maven uses
"core/src/main/resources/version/org-apache-calcite-jdbc.properties" for
figuring out driver version.
And Gradle build just generates CalciteDriverVersion.java
I don't want to over-engineer pom.xml files, so I did not try to make
pom.xml files buildable.
However, pom.xml files will be removed right after Gradle script is merged.

The other change is Druid migrated from JUnit4 to JUnit5 (to enable the use
of Assumptions).

I'm inclined to commit the change, so please feel free to review it.

In case you missed, you can start playing with the build with the commands
that are listed in the commit message:
https://github.com/apache/calcite/pull/1571/commits/4bbf1cdeac6e09959e027eaa482a67937228bf9d
"./gradlew tasks" could be a good start.

The file movements/edits are minimal, so I hope it won't affect pending PRs.


Note, as I said earlier, having both Avatica and Calcite in Gradle enables
us to load both projects to the IDE at the same time.
In the same way, one can test who Avatica fix impacts Calcite with a single
command line. No changes to pom.xml needed!!!

The procedure is to clone calcite-avatica and calcite side by side, and use
-PlocalAvatica property.

Vladimir

Re: [ANNOUNCE] Haisheng Yuan joins Calcite PMC

[Julian Hyde]

Welcome, Haisheng! As Stamatis said, you have been a wise voice in setting the 
direction of the project, and helpful in discussions.

> On Nov 11, 2019, at 9:01 PM, Matt Wang  wrote:
> 
> Congratulations!
> 
> 
> ---
> Best,
> Matt Wang
> 
> 
> On 11/12/2019 12:36，Haisheng Yuan wrote：
> Thanks, all!
> 
> It's my great honour to be a member of Apache Calcite project. Let's 
> collaborate to continue working on it and make it better.
> 
> Thanks again.
> - Haisheng
> 
> --
> 发件人：XING JIN
> 日　期：2019年11月12日 10:20:16
> 收件人：
> 主　题：Re: [ANNOUNCE] Haisheng Yuan joins Calcite PMC
> 
> Congratulations Haisheng ~
> You well deserved !
> 
> Kevin Risden  于2019年11月12日周二 上午3:13写道：
> 
> Congrats and welcome!
> 
> Kevin Risden
> 
> 
> On Mon, Nov 11, 2019 at 2:10 PM Chunhui Shi  wrote:
> 
> Congratulations!
> 
> On Mon, Nov 11, 2019 at 10:09 AM Jinfeng Ni  wrote:
> 
> Congratulations!
> 
> 
> On Tue, Nov 12, 2019 at 1:23 AM Rui Wang  wrote:
> 
> Congrats HaiSheng!
> 
> 
> -Rui
> 
> On Mon, Nov 11, 2019 at 8:05 AM Stamatis Zampetakis <
> zabe...@gmail.com
> 
> wrote:
> 
> Congrats Haisheng!
> 
> Reviews, code contributions, design discussions, helping users, and
> many
> more things for improving the project.
> 
> Personally, I also learn a lot from our interactions.
> 
> All these are much appreciated; keep it up!!
> 
> Best,
> Stamatis
> 
> On Mon, Nov 11, 2019, 4:17 PM Michael Mior 
> wrote:
> 
> Welcome and congratulations HaiSheng!
> --
> Michael Mior
> mm...@apache.org
> 
> Le dim. 10 nov. 2019 à 22:45, Francis Chuang
>  a écrit :
> 
> I'm pleased to announce that Haisheng has accepted an
> invitation
> to
> join the Calcite PMC. Haisheng has been a consistent and
> helpful
> figure in the Calcite community for which we are very grateful.
> We
> look forward to the continued contributions and support.
> 
> Please join me in congratulating Haisheng!
> 
> - Francis (on behalf of the Calcite PMC)
> 
> 
> 
> 
> 
> 

Re: CsvAdapter (Json content) from String / InputStream

[Andrei Sereda]

> I see that this feature request relates to Source.java and Sources.java,
which are in org.apache.calcite.util in core.
I'm not planning to change Source.java it already exposes Reader /
InputStream

> If you add some capability, it is fine to add It to the CSV adapter
example, but it is much more important that that capability exists in the
file adapter.
I will add to both. The general idea behind this change is that currently
CSV / File Adapter(s) require inputs to be File(s) or URL(s) which forces
users to create temporal resources manually (when their content is already
in-memory). If input to CSV / File adapter(s) is generic Readable [1] /
Reader [2] or InputStream it gives more flexibility to users.

[1] https://docs.oracle.com/javase/7/docs/api/java/lang/Readable.html
[2] https://docs.oracle.com/javase/7/docs/api/java/io/Reader.html


On Thu, Nov 14, 2019 at 2:45 PM Julian Hyde  wrote:

> I see that this feature request relates to Source.java and Sources.java,
> which are in org.apache.calcite.util in core.
>
> If you add some capability, it is fine to add It to the CSV adapter
> example, but it is much more important that that capability exists in the
> file adapter.
>
> Julian
>
>
> > On Nov 14, 2019, at 11:36 AM, Andrei Sereda  wrote:
> >
> > I think the change is straightforward (will not add complexity).
> >
> > On Thu, Nov 14, 2019 at 1:24 PM Julian Hyde  wrote:
> >
> >> Remember that CsvAdapter is in the “example” module. Keep it simple.
> >>
> >> The file adapter can also parse CSV files.
> >>
> >> Julian
> >>
> >>
> >>
> >>> On Nov 14, 2019, at 9:40 AM, Andrei Sereda  wrote:
> >>>
> >>> Hello,
> >>>
> >>> Source object already exposes Reader / InputStream API. Probably
> >>> JsonEnumerator can be changed to use those methods.
> >>>
> >>> Do you mind creating a JIRA ticket ? I'll take a look.
> >>>
> >>> Thanks,
> >>> Andrei.
> >>>
> >>> On Thu, Nov 14, 2019 at 7:45 AM Yanna elina <
> yannaelinasul...@gmail.com>
> >>> wrote:
> >>>
>  Hi guys ,
>  I saw in the code that this nice adapter makes it possible to make SQL
>  queries on the data JSON
> 
> 
> >>
> https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv
> 
>  But  it's limited on File / URL.
>  JsonTable constructor accepte only a Source object and this Source
> >> object
>  can be construct only accross  a File / URL.
> 
>  it could be nice to have the possibility to make this source from
>  ImputStream too .
> 
>  Creating a temp-file from an InputStream or String can be excesive.
> 
>  Thanks
> 
> >>
> >>
>
>

Re: CsvAdapter (Json content) from String / InputStream

[Julian Hyde]

I see that this feature request relates to Source.java and Sources.java, which 
are in org.apache.calcite.util in core.

If you add some capability, it is fine to add It to the CSV adapter example, 
but it is much more important that that capability exists in the file adapter.

Julian


> On Nov 14, 2019, at 11:36 AM, Andrei Sereda  wrote:
> 
> I think the change is straightforward (will not add complexity).
> 
> On Thu, Nov 14, 2019 at 1:24 PM Julian Hyde  wrote:
> 
>> Remember that CsvAdapter is in the “example” module. Keep it simple.
>> 
>> The file adapter can also parse CSV files.
>> 
>> Julian
>> 
>> 
>> 
>>> On Nov 14, 2019, at 9:40 AM, Andrei Sereda  wrote:
>>> 
>>> Hello,
>>> 
>>> Source object already exposes Reader / InputStream API. Probably
>>> JsonEnumerator can be changed to use those methods.
>>> 
>>> Do you mind creating a JIRA ticket ? I'll take a look.
>>> 
>>> Thanks,
>>> Andrei.
>>> 
>>> On Thu, Nov 14, 2019 at 7:45 AM Yanna elina 
>>> wrote:
>>> 
 Hi guys ,
 I saw in the code that this nice adapter makes it possible to make SQL
 queries on the data JSON
 
 
>> https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv
 
 But  it's limited on File / URL.
 JsonTable constructor accepte only a Source object and this Source
>> object
 can be construct only accross  a File / URL.
 
 it could be nice to have the possibility to make this source from
 ImputStream too .
 
 Creating a temp-file from an InputStream or String can be excesive.
 
 Thanks
 
>> 
>> 

Re: CsvAdapter (Json content) from String / InputStream

[Andrei Sereda]

I think the change is straightforward (will not add complexity).

On Thu, Nov 14, 2019 at 1:24 PM Julian Hyde  wrote:

> Remember that CsvAdapter is in the “example” module. Keep it simple.
>
> The file adapter can also parse CSV files.
>
> Julian
>
>
>
> > On Nov 14, 2019, at 9:40 AM, Andrei Sereda  wrote:
> >
> > Hello,
> >
> > Source object already exposes Reader / InputStream API. Probably
> > JsonEnumerator can be changed to use those methods.
> >
> > Do you mind creating a JIRA ticket ? I'll take a look.
> >
> > Thanks,
> > Andrei.
> >
> > On Thu, Nov 14, 2019 at 7:45 AM Yanna elina 
> > wrote:
> >
> >> Hi guys ,
> >> I saw in the code that this nice adapter makes it possible to make SQL
> >> queries on the data JSON
> >>
> >>
> https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv
> >>
> >> But  it's limited on File / URL.
> >> JsonTable constructor accepte only a Source object and this Source
> object
> >> can be construct only accross  a File / URL.
> >>
> >> it could be nice to have the possibility to make this source from
> >> ImputStream too .
> >>
> >> Creating a temp-file from an InputStream or String can be excesive.
> >>
> >> Thanks
> >>
>
>

Re: CsvAdapter (Json content) from String / InputStream

[Julian Hyde]

Remember that CsvAdapter is in the “example” module. Keep it simple.

The file adapter can also parse CSV files.

Julian



> On Nov 14, 2019, at 9:40 AM, Andrei Sereda  wrote:
> 
> Hello,
> 
> Source object already exposes Reader / InputStream API. Probably
> JsonEnumerator can be changed to use those methods.
> 
> Do you mind creating a JIRA ticket ? I'll take a look.
> 
> Thanks,
> Andrei.
> 
> On Thu, Nov 14, 2019 at 7:45 AM Yanna elina 
> wrote:
> 
>> Hi guys ,
>> I saw in the code that this nice adapter makes it possible to make SQL
>> queries on the data JSON
>> 
>> https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv
>> 
>> But  it's limited on File / URL.
>> JsonTable constructor accepte only a Source object and this Source object
>> can be construct only accross  a File / URL.
>> 
>> it could be nice to have the possibility to make this source from
>> ImputStream too .
>> 
>> Creating a temp-file from an InputStream or String can be excesive.
>> 
>> Thanks
>> 

Avatica: exception in the writetoProtoWithType function

[Enrique Saurez]

Hi!

I am using Apache Calcite-Avatica version 1.12 (but the relevant code
sections are not different from the master branch), and I am getting
the following exception on the client side (but the actual error in on
the server side):

org.apache.calcite.avatica.AvaticaSqlException: Error -1 (0) :
Remote driver error: ClassCastException: java.lang.Long cannot be cast
to java.lang.Float
at org.apache.calcite.avatica.Helper.createException(Helper.java:54)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at 
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:557)
at 
org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137)
at 
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.getCustomerByName(Payment.java:400)
at 
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment.run(Payment.java:221)
at 
com.oltpbenchmark.benchmarks.tpcc.TPCCWorker.executeWork(TPCCWorker.java:74)
at com.oltpbenchmark.api.Worker.doWork(Worker.java:386)
at com.oltpbenchmark.api.Worker.run(Worker.java:296)
at java.lang.Thread.run(Thread.java:748)
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Float
at 
org.apache.calcite.avatica.remote.TypedValue.writeToProtoWithType(TypedValue.java:594)
at 
org.apache.calcite.avatica.remote.TypedValue.toProto(TypedValue.java:799)
at org.apache.calcite.avatica.Meta$Frame.serializeScalar(Meta.java:985)
at org.apache.calcite.avatica.Meta$Frame.parseColumn(Meta.java:971)
at org.apache.calcite.avatica.Meta$Frame.toProto(Meta.java:936)
at 
org.apache.calcite.avatica.remote.Service$ResultSetResponse.serialize(Service.java:841)
at 
org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1158)
at 
org.apache.calcite.avatica.remote.Service$ExecuteResponse.serialize(Service.java:1113)
at 
org.apache.calcite.avatica.remote.ProtobufTranslationImpl.serializeResponse(ProtobufTranslationImpl.java:348)
at 
org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:57)
at 
org.apache.calcite.avatica.remote.ProtobufHandler.encode(ProtobufHandler.java:31)
at 
org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:95)
at 
org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
at 
org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
at 
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)

>From the code, it seems like when the function "writeToProtoWithType"
is called from "toProto", there is a boxing conversion from long
(primitive) to Long (object), this is because
for floats, "toProto" is using "((Float) o).longValue()" which returns
a long. Then in "writeToProtoWithType" is being casted to float, which
I think causes the CastClassException.

If I add this code to the testFloat() function in the
"core/src/test/java/org/apache/calcite/avatica/remote/TypedValueTest.java"
file:

Common.TypedValue.Builder builder = Common.TypedValue.newBuilder();
Common.Rep val = TypedValue.toProto(builder, Float.valueOf(3.14159f));
Common.TypedValue typedVal = builder.build();

it replicates the exception. I have a couple of questions:

1. Why are you using the longValue() on the float within the toProto
function? Doesn't this cast lose information?
2. What kind of code would trigger this kind of behavior? It is hard
for me to debug in detail, because it is only trigger when I running
it in my remote server. If I have some idea of how this can be
trigger, maybe I can find an example on my server code that is
triggering it.
3. Is this a known problem? I couldn't find it in the mailing list,
but I am new to the community.

Thanks a lot for help!

Best regards,
Enrique Saurez

Re: CsvAdapter (Json content) from String / InputStream

[Andrei Sereda]

Hello,

Source object already exposes Reader / InputStream API. Probably
JsonEnumerator can be changed to use those methods.

Do you mind creating a JIRA ticket ? I'll take a look.

Thanks,
Andrei.

On Thu, Nov 14, 2019 at 7:45 AM Yanna elina 
wrote:

> Hi guys ,
> I saw in the code that this nice adapter makes it possible to make SQL
> queries on the data JSON
>
> https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv
>
> But  it's limited on File / URL.
> JsonTable constructor accepte only a Source object and this Source object
> can be construct only accross  a File / URL.
>
> it could be nice to have the possibility to make this source from
> ImputStream too .
>
> Creating a temp-file from an InputStream or String can be excesive.
>
> Thanks
>

[jira] [Created] (CALCITE-3505) Infinite matching of FilterProjectTransposeRule causes stackoverflow

[Jin Xing (Jira)]

Jin Xing created CALCITE-3505:
-

 Summary: Infinite matching of FilterProjectTransposeRule causes 
stackoverflow
 Key: CALCITE-3505
 URL: https://issues.apache.org/jira/browse/CALCITE-3505
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: Jin Xing






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: CALCITE-2905: Maven -> Gradle: any thoughts

[Vladimir Sitnikov]

>Currently we have two ways of running/skipping a slow test

You wanted to say "at least two", did you?
There are properties like calcite.test.druid which target a single test
class.

>Does this make sense considering the ongoing work of moving to Gradle and
JUnit5?

The current Gradle-JUnit5 build uses **both** skipping options at the same
time, because I want minimal changes to the source code.
I'm not sure what is the better way, but I do like how JUnit5 enables to
have short tags and tag expressions to filter tests.

We could even tag tests with "@ExpectedDuration" and provide an option like
"run all tests that typically fit in 5sec"

>to unify the two by annotating all slow tests with the appropriate
annotation

I would suggest doing that after Gradle migration. WDYT?

Vladimir

Re: CALCITE-2905: Maven -> Gradle: any thoughts

[Stamatis Zampetakis]

Hi Vladimir,

Thanks for working on this.

I have a quick question regarding fast/slow tests.

Currently we have two ways of running/skipping a slow test:
* using the SlowTests annotation and using JUnit categories;
* using the system property calcite.test.slow.

I was planning (WIP in [1]) to unify the two by annotating all slow tests
with the appropriate annotation.
Does this make sense considering the ongoing work of moving to Gradle and
JUnit5?
Are you planning to keep/use these kind of annotations?

Best,
Stamatis

[1] https://github.com/apache/calcite/pull/1375




On Mon, Nov 11, 2019 at 11:30 PM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:

> As you might know, Avatica is already migrated to Gradle.
>
> On top of that, Calcite is almost there as well:
> https://github.com/apache/calcite/pull/1571
>
> The missing bit is probably the notion of tests "fast/slow/etc/etc".
> Unfortunately, there are tests that assert on folder layout like
> "target/test-classes".
> I'm going to ignore those tests for now, and fix them as pom.xml files are
> removed.
>
> I'm going to configure Gradle to use JUnit5 engine to run the tests.
>
> In the meantime, everybody's welcome to preview Gradle build.
>
> Vladimir
>

CsvAdapter (Json content) from String / InputStream

[Yanna elina]

Hi guys ,
I saw in the code that this nice adapter makes it possible to make SQL
queries on the data JSON
https://github.com/apache/calcite/tree/ab71c4cae5a5c3c7d979337a2d38ddaf271aa206/example/csv/src/main/java/org/apache/calcite/adapter/csv

But  it's limited on File / URL.
JsonTable constructor accepte only a Source object and this Source object
can be construct only accross  a File / URL.

it could be nice to have the possibility to make this source from
ImputStream too .

Creating a temp-file from an InputStream or String can be excesive.

Thanks

Re: [DISCUSS] Support Sql Hint for Calcite

[Vladimir Ozerov]

Hi Danny,

Thank you very much for making this happen. Query hints are a very valuable
addition to the product.

Regards,
Vladimir.

вс, 10 нояб. 2019 г. в 05:58, Danny Chan :

> Hi, fellows, I’m planning to merge the hints PR in the following week, I’m
> very appreciated if you have other more review comment address. [1]
>
> Or if you have other thoughts, please address it here :)
>
> [1] https://github.com/apache/calcite/pull/1354
>
> Best,
> Danny Chan
> 在 2019年10月30日 +0800 AM1:42，Julian Hyde ，写道：
> > Sure, we can make sure something gets into 1.22. There is consensus
> about the parser extensions, whereas the extensions to RelNode and the
> planner engine are a little more experimental. So let’s go forward with
> that, stating which parts we think are likely to change.
> >
> > Julian
> >
> >
> > > On Oct 29, 2019, at 2:09 AM, Seliverstov Igor 
> wrote:
> > >
> > > Colleagues,
> > >
> > > Not only Hazelcast and Apache Flink are interested in SQL hints.
> Apache Ignite community is working on Calcite integration too, it’s
> important for us to have appropriate API at current development stage. This
> case we’ll be able to adapt our solution for SQL hints usage, probably
> determining additional approach weaknesses or inconveniences.
> > >
> > > Regards,
> > > Igor
> > >
> > > > 29 окт. 2019 г., в 11:51, Danny Chan 
> написал(а):
> > > >
> > > > Julian, can we make some effort to push this feature into release
> 1.22, there are users like Vladimir Ozerov from Hazelcast that are
> interesting on this feature, also the Apache Flink.
> > > >
> > > > I agree that this internal design is not that perfect, at this
> moment, we may hardly to conclude a perfect solution, but at least, the
> syntax would remain unchanged in the future.
> > > >
> > > > So can we mark this feature as experimental and we can promote the
> internal design when accept more feedbacks from the Calcite uses (from
> Apache Flink or from users like Vladimir).
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年10月18日 +0800 AM4:55，Julian Hyde ，写道：
> > > > > I wonder whether it is possible to add some kind of “action
> handler” to the planner engine, called, for example, when a rule has fired
> and is registering the RelNode created by the rule. People can write their
> own action handlers to copy hints around. Since the action handlers are the
> user’s code, they can iterate faster to find a hint-propagation strategy
> that works in practice.
> > > > >
> > > > > Another idea is to use VolcanoPlanner.Provenance[1]. A RelNode can
> find its ancestor RelNodes, and the rules that fired to create it. So it
> can grab hints from those ancestors. It does not need to copy those hints
> onto itself.
> > > > >
> > > > > Julian
> > > > >
> > > > > [1]
> https://calcite.apache.org/apidocs/org/apache/calcite/plan/volcano/VolcanoPlanner.Provenance.html
> <
> https://calcite.apache.org/apidocs/org/apache/calcite/plan/volcano/VolcanoPlanner.Provenance.html
> >
> > > > >
> > > > > > On Oct 16, 2019, at 8:38 PM, Haisheng Yuan <
> h.y...@alibaba-inc.com> wrote:
> > > > > >
> > > > > > Julian,
> > > > > > Your concern is very valid, and that is also our main concern.
> > > > > > I was thinking whether we can put hint into the MEMO group, so
> that both logical and physical expression in the same group can share the
> same hint, without copying the hint explicitly. But for newly generated
> expression that doesn't belong to the original group, we still need to copy
> hints. What's worse, in HepPlanner, there is no such concept, we may still
> need to copy hints explicity in planner rules, if we want to keep the hint,
> which is burdensome.
> > > > > >
> > > > > > - Haisheng
> > > > > >
> > > > > >
> --
> > > > > > 发件人：Danny Chan
> > > > > > 日 期：2019年10月16日 14:54:46
> > > > > > 收件人：
> > > > > > 主 题：Re: [DISCUSS] Support Sql Hint for Calcite
> > > > > >
> > > > > > Thanks for the clarification.
> > > > > >
> > > > > > I understand you worried. Yes, the effort/memory would be wasted
> or meaningless if hints are not used. This is just what a hint does, it is
> a “hint” and non-mandatory, but we should give the chance to let user see
> them, it is the use that decide if to use the hints and how to use them.
> For big queries I have no confidence to cover the corner cases. So can we
> mark this feature as experimental and used for simple queries(no
> decorrelation) first ?
> > > > > >
> > > > > > For “reversible”, during the implementation, I try to make the
> modifications non-invasive with the current codes. That is why I made all
> the interfaces about the hint into one class named RelWithHInt. Different
> with trait, I didn’t force users to pass in the hints in the RelNode
> constructor. I think if is not a bigwork if we want to remove the API.
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > > > 在 2019年10月16日 +0800 AM11:14，Julian Hyde ，写道：
> > > > > > > By “skeptical” I mean that I 

[jira] [Created] (CALCITE-3504) allow empty ARRAY[] literals

[Pressenna (Jira)]

Pressenna created CALCITE-3504:
--

 Summary: allow empty ARRAY[] literals
 Key: CALCITE-3504
 URL: https://issues.apache.org/jira/browse/CALCITE-3504
 Project: Calcite
  Issue Type: Wish
Affects Versions: 1.21.0
Reporter: Pressenna


Currently an ARRAY expression requires at least one element.
Please allow empty ARRAY expressions.

{code:sql}
SELECT ARRAY[] FROM foo;

-- or more concise for updates:

UPDATE food set array_column = ARRAY[];
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Proposal to add API to force rules matching specific rels

[Vladimir Ozerov]

Hi Xing,

Thanks for your suggestion. Yes, this may help, and if I get your idea
right, I already had it in my reproducer:
1) Create the converted physical input:
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
2) Use it in case no physical children were found:
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L79

This idea is borrowed from Apache Drill physical rules. But the problem is
that this approach leads to a suboptimal plan - parent node doesn't know
the future distribution of a child node. And as a result, it doesn't know
it's own distribution. So the final plan is constructed in that way:
1.1) Root enforced "SINGLETON" on its child:
-> PhysicalRoot[SINGLETON]
 -> Converter[SINGLETON]
  -> PhysicalProject[*ANY*]
   -> PhysicalScan[REPLICATED]

1.2) But since the child (PhysicalProject) failed to infer distribution
during rule call, it falls back to ANY distribution. In order to satisfy
SINGLETON distribution of a parent, we inject an exchange in the final plan:
-> PhysicalRoot[SINGLETON]
* -> Exchange[SINGLETON]*
  -> PhysicalProject[*ANY*]
   -> PhysicalScan[REPLICATED]

2) But this is a suboptimal plan. The optimal plan is:
-> PhysicalRoot[SINGLETON]
 -> PhysicalProject[REPLICATED]
  -> PhysicalScan[REPLICATED]

You may observe it in my tests:
1)
https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L46
-
works as you described and produces not optimal plan with exchange
2)
https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L30
-
rely on AbstractConverter-s and produce an optimal plan with bottom-up
trait propagation at the cost of significantly increased planning time

Regards,
Vladimir.

пт, 8 нояб. 2019 г. в 16:15, XING JIN :

> Hi Vladimir,
>
> I think the way PlannerTests#GoodSingleRule and EnumerableXXXRule work may
> help you ~
> They work by a top-down fashion, but when matching parent, they convert the
> children explicitly.
> You may try below steps:
> 1. Construct a rule LogicalParentRule to match the LogicalParent without
> distribution/physical requirement for the LogicalChild;
> 2. In this rule, you call 'planner.changeTraits' on the LogicalChild to
> build a new child with physical convention. Note that at this moment only
> an empty RelSubset is created and no PhysicalChild exists.
> 3. Then set the RelNode to be the new input of LogicalParent;
>
> By above steps, you can build a parent-child relationship between
> LogicalParent and PhysicalChild, and at last the PhysicalParentRule will be
> fired based on this relationship.
>
> I have a commit to illustrate my idea, check VolcanoPlannerTest#testDEV in
> below link, hope it may help you ~
> https://github.com/jinxing64/calcite/tree/demo
>
> Also I'm +1 with Seliverstov that to get all parents of a set, which
> against the current check in RelSubset#getParentRels
>
> Best,
> Jin
>
> Vladimir Ozerov  于2019年11月5日周二 下午6:41写道：
>
> > Hi Xiening,
> >
> > I read the thread about on-demand trait requests. It seems pretty similar
> > to what I am trying to achieve, as it facilitates the bottom-up
> propagation
> > of physical traits. In fact, both your and my strategy propagate traits
> > bottom-up, but I do this through rules, which also fire bottom-up, while
> in
> > your case only the traits are propagated bottom-up, while rules continue
> > working in a top-down fashion.
> >
> > However, I am thinking of how I would potentially implement my optimizer
> > with your approach, and it feels like with on-demand traits resulting
> > implementation of metadata queries may become very complex to that point
> > that it will look like another set of rules, parallel to the already
> > existing ruleset. For example, consider that I have a couple of
> distributed
> > tables in an OLTP application. These tables have a number of indexes,
> and I
> > would like to join them. First, I have a number of choices on how to join
> > tables with respect to distribution. Then, I have a number of choices on
> > which access method to use. Because sometimes it is beneficial to pick
> > index scans instead of table scans even without index conditions, for
> > example, to preserve a comfortable collation. So when my logical scan
> > receives such metadata request, it typically cannot return all possible
> > combinations, because there are too many of them. Instead, some
> heuristical
> > or cost-based logic will be used to calculate a couple of most
> prospective
> > ones. But it seems that we will have to duplicate the same logic in the
> > corresponding rule, aren't we?
> >
> > I would love to read your design because this is a really interesting
> > topic, and it is of great importance for the distributed engines
> developed
> > on top of Calcite since proper use of distribution and collation is the
> key
> > success 

Re: Re: Re: Problem with converters and possibly rule matching

[Vladimir Ozerov]

search place* = search space

чт, 14 нояб. 2019 г. в 13:10, Vladimir Ozerov :

> Hi Haisheng,
>
> I double-checked the code. My original version returned false for some
> cases, but it didn't affect number of rules calls anyway, so I changed it
> to always return true. Please note that if I change the code as you
> suggested, the test started failing, because bottom-up propagation of rule
> calls no longer work: when the child is converted to physical form, the
> parent logical node is not notified. This is the very problem I address
> with that weird physical-to-logical conversions: they do not make sense,
> and converter expansion does not produce any new rels, but their existence
> allow for logical rule re-trigger which ultimately allow the plan to
> compile.
>
> Regarding two conventions - I agree that it may look strange, but I do not
> see any problems from the correctness perspective. Separation of logical
> and physical planning helps me avoid excessive expansion of the search
> place: I do not want my physical rules to produce new rels from not
> optimized logical rels. Do you see any problems with that approach?
>
> Regards,
> Vladimir
>
> ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan :
>
>> Hi Vladimir,
>>
>> The code in PHYSICAL convention L44 looks weird, I think it always
>> returns true.
>>
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
>>
>> Try this:
>>
>> fromTraits.containsIfApplicable(Convention.PHYSICAL)
>> && toTraits.containsIfApplicable(Convention.PHYSICAL);
>>
>>
>> Adding a AbstractConverter on logical operators is meaningless. Calcite
>> is mixing the concept of logical and physical together, which is sad.
>>
>> BTW, using 2 conventions is not appropriate and wrong.
>>
>> - Haisheng
>>
>> --
>> 发件人：Vladimir Ozerov
>> 日 期：2019年11月05日 18:02:15
>> 收件人：Haisheng Yuan
>> 抄 送：dev@calcite.apache.org (dev@calcite.apache.org)<
>> dev@calcite.apache.org>
>> 主 题：Re: Re: Problem with converters and possibly rule matching
>>
>> Hi Haisheng,
>>
>>  think I already tried something very similar to what you explained, but
>> it gave not an optimal plan. Please let me describe what I did. I would
>> appreciate your feedback.
>>
>> 1) We start with a simple operator tree Root <- Project <- Scan, where
>> the root is a final aggregator in the distributed query engine:
>> -> LogicalRoot
>>  -> LogicalProject
>>   -> LogicalScan
>>
>> 2) First, we convert the Root and enforce SINGLETON distribution on a
>> child:
>> *-> PhysicalRoot[SINGLETON]*
>> * -> Enforcer#1[SINGLETON]*
>>   -> LogicalProject
>>-> LogicalScan
>>
>> 3) Then the project's rule is invoked. It doesn't know the distribution
>> of the input, so it requests ANY distribution. Note that we have to set ANY
>> to the project as well since we do not know the distribution of the input:
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>> *  -> PhysicalProject[ANY]*
>> *   -> Enforcer#2[ANY]*
>> -> LogicalScan
>>
>> 4) Finally, the physical scan is created and its distribution is
>> resolved. Suppose that it is REPLICATED, i.e. the whole result set is
>> located on all nodes.
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>>   -> PhysicalProject[ANY]
>>-> Enforcer#2[ANY]
>> *-> PhysicalScan[REPLICATED]*
>>
>> 5) Now as all logical nodes are converted, we start resolving enforcers.
>> The second one is no-op, since REPLICATED satisfies ANY:
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>>   -> PhysicalProject[ANY]
>>-> PhysicalScan[REPLICATED]
>>
>> 6) But the first enforcer now requires an Exchange, since ANY doesn't
>> satisfy SINGLETON!
>> -> PhysicalRoot[SINGLETON]
>> * -> SingletonExchange[SINGLETON]*
>>   -> PhysicalProject[ANY] // <= unresolved!
>>-> PhysicalScan[REPLICATED]
>>
>> The resulting plan requires data movement only because we didn't know
>> precise distribution of the PhysicalProject when it was created. But should
>> I enable Convention.Impl.canConvertConvention, bottom-up propagation
>> kicks in, and the correct plan is produced because now LogicalProject
>> has a chance to be converted to PhysicalProject with the concrete
>> distribution. The optimized plan looks like this (since REPLICATED
>> satisfies SINGLETON):
>> -> PhysicalRoot[SINGLETON]
>>  -> PhysicalProject[REPLICATED]
>>   -> PhysicalScan[REPLICATED]
>>
>> You may see this in action in my reproducer:
>> 1) Test producing "bad" plan:
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
>> 2) Root enforces SINGLETON on Project:
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
>> 3) Project enforces default (ANY) distribution on Scan:
>> 

Re: Re: Re: Problem with converters and possibly rule matching

[Vladimir Ozerov]

Hi Haisheng,

I double-checked the code. My original version returned false for some
cases, but it didn't affect number of rules calls anyway, so I changed it
to always return true. Please note that if I change the code as you
suggested, the test started failing, because bottom-up propagation of rule
calls no longer work: when the child is converted to physical form, the
parent logical node is not notified. This is the very problem I address
with that weird physical-to-logical conversions: they do not make sense,
and converter expansion does not produce any new rels, but their existence
allow for logical rule re-trigger which ultimately allow the plan to
compile.

Regarding two conventions - I agree that it may look strange, but I do not
see any problems from the correctness perspective. Separation of logical
and physical planning helps me avoid excessive expansion of the search
place: I do not want my physical rules to produce new rels from not
optimized logical rels. Do you see any problems with that approach?

Regards,
Vladimir

ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan :

> Hi Vladimir,
>
> The code in PHYSICAL convention L44 looks weird, I think it always returns
> true.
>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
>
> Try this:
>
> fromTraits.containsIfApplicable(Convention.PHYSICAL)
> && toTraits.containsIfApplicable(Convention.PHYSICAL);
>
>
> Adding a AbstractConverter on logical operators is meaningless. Calcite is
> mixing the concept of logical and physical together, which is sad.
>
> BTW, using 2 conventions is not appropriate and wrong.
>
> - Haisheng
>
> --
> 发件人：Vladimir Ozerov
> 日 期：2019年11月05日 18:02:15
> 收件人：Haisheng Yuan
> 抄 送：dev@calcite.apache.org (dev@calcite.apache.org) >
> 主 题：Re: Re: Problem with converters and possibly rule matching
>
> Hi Haisheng,
>
>  think I already tried something very similar to what you explained, but
> it gave not an optimal plan. Please let me describe what I did. I would
> appreciate your feedback.
>
> 1) We start with a simple operator tree Root <- Project <- Scan, where
> the root is a final aggregator in the distributed query engine:
> -> LogicalRoot
>  -> LogicalProject
>   -> LogicalScan
>
> 2) First, we convert the Root and enforce SINGLETON distribution on a
> child:
> *-> PhysicalRoot[SINGLETON]*
> * -> Enforcer#1[SINGLETON]*
>   -> LogicalProject
>-> LogicalScan
>
> 3) Then the project's rule is invoked. It doesn't know the distribution of
> the input, so it requests ANY distribution. Note that we have to set ANY to
> the project as well since we do not know the distribution of the input:
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
> *  -> PhysicalProject[ANY]*
> *   -> Enforcer#2[ANY]*
> -> LogicalScan
>
> 4) Finally, the physical scan is created and its distribution is resolved.
> Suppose that it is REPLICATED, i.e. the whole result set is located on all
> nodes.
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
>   -> PhysicalProject[ANY]
>-> Enforcer#2[ANY]
> *-> PhysicalScan[REPLICATED]*
>
> 5) Now as all logical nodes are converted, we start resolving enforcers.
> The second one is no-op, since REPLICATED satisfies ANY:
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
>   -> PhysicalProject[ANY]
>-> PhysicalScan[REPLICATED]
>
> 6) But the first enforcer now requires an Exchange, since ANY doesn't
> satisfy SINGLETON!
> -> PhysicalRoot[SINGLETON]
> * -> SingletonExchange[SINGLETON]*
>   -> PhysicalProject[ANY] // <= unresolved!
>-> PhysicalScan[REPLICATED]
>
> The resulting plan requires data movement only because we didn't know
> precise distribution of the PhysicalProject when it was created. But should
> I enable Convention.Impl.canConvertConvention, bottom-up propagation
> kicks in, and the correct plan is produced because now LogicalProject has
> a chance to be converted to PhysicalProject with the concrete
> distribution. The optimized plan looks like this (since REPLICATED
> satisfies SINGLETON):
> -> PhysicalRoot[SINGLETON]
>  -> PhysicalProject[REPLICATED]
>   -> PhysicalScan[REPLICATED]
>
> You may see this in action in my reproducer:
> 1) Test producing "bad" plan:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
> 2) Root enforces SINGLETON on Project:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
> 3) Project enforces default (ANY) distribution on Scan:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
>
> Please let me know if this flow is similar to what you meant.
>
> Regards,
> Vladimir.
>
> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan :
>
>> Hi Vladimir,
>>
>> This is still can be done through top-down request approach.
>>
>>