Re: Re: [DISCUSS] Support Sql Hint for Calcite

[Haisheng Yuan]

Julian,
Your concern is very valid, and that is also our main concern.
I was thinking whether we can put hint into the MEMO group, so that both 
logical and physical expression in the same group can share the same hint, 
without copying the hint explicitly. But for newly generated expression that 
doesn't belong to the original group, we still need to copy hints. What's 
worse, in HepPlanner, there is no such concept, we may still need to copy hints 
explicity in planner rules, if we want to keep the hint, which is burdensome.

- Haisheng

--
发件人：Danny Chan
日　期：2019年10月16日 14:54:46
收件人：
主　题：Re: [DISCUSS] Support Sql Hint for Calcite

Thanks for the clarification.

I understand you worried. Yes, the effort/memory would be wasted or meaningless 
if hints are not used. This is just what a hint does, it is a “hint” and 
non-mandatory, but we should give the chance to let user see them, it is the 
use that decide if to use the hints and how to use them. For big queries I have 
no confidence to cover the corner cases. So can we mark this feature as 
experimental and used for simple queries(no decorrelation) first ?

For “reversible”, during the implementation, I try to make the modifications 
non-invasive with the current codes. That is why I made all the interfaces 
about the hint into one class named RelWithHInt. Different with trait, I didn’t 
force users to pass in the hints in the RelNode constructor. I think if is not 
a bigwork if we want to remove the API.

Best,
Danny Chan
在 2019年10月16日 +0800 AM11:14，Julian Hyde ，写道：
> By “skeptical” I mean that I think we can come up with a mechanism to copy 
> hints when applying planner rules, but even when we have implemented that 
> mechanism there will be many cases where people want a hint and that hint is 
> not copied to the RelNode where it is needed, and many other cases where we 
> spend the effort/memory of copying the hint to a RelNode and the hint is not 
> used.
>
> By “reversible” I mean if we come up with an API that does not work, how do 
> we change or remove that API without people complaining?
>
> Julian
>
>
> > On Oct 15, 2019, at 7:11 PM, Danny Chan  wrote:
> >
> > Thanks Julian
> >
> > > I am skeptical that RelWithHint will work for large queries.
> >
> > For “skeptical” do you mean how to transfer the hints during rule planning 
> > ? I’m also not that confident yet.
> >
> > > How do we introduce it in a reversible way
> > Do you mean transform the RelWithHint back into the SqlHint ? I didn’t 
> > implement it in current patch, but I think we have the ability to do that 
> > because we have a inheritPath for each RelWithHint, we can collect all the 
> > hints together and merge them into the SqlHints, then propagate these 
> > SqlHints to the SqlNodes.
> >
> > > What are the other options?
> > Do you mean the way to transfer hints during planning ? I have no other 
> > options yet.
> >
> > Best,
> > Danny Chan
> > 在 2019年10月16日 +0800 AM8:03，dev@calcite.apache.org，写道：
> > >
> > > I am skeptical that RelWithHint will work for large queries.
>

Re: [DISCUSSION] Extension of Metadata Query

[Xiening Dai]

I have seen both patterns in current code base. In most places, for example 
SubQueryRemoveRule, AggregateUnionTrasposeRule SortJoinTransposeRule, etc., 
RelOptCluster.getMetadataQuery() is used. And there are a few other places 
where new RelMetadataQuery instance is created, which Haisheng attempts to fix.

Currently RelOptCluster.invalidateMetadataQuery() is called at the end of 
RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery is guaranteed 
to be within a RelOptCall. I think Haisheng’s fix is safe.


> On Oct 16, 2019, at 1:53 AM, Danny Chan  wrote:
> 
> This is the reason I was struggling for the discussion.
> 
> Best,
> Danny Chan
> 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道：
>> 
>> RelMetadataQuery

Apache Calcite meetup group

[Julian Hyde]

If you’re a member of the Apache Calcite meetup group[1], you probably just 
received an email saying that the group is shutting down. I set it up a few 
years ago, but I never find time to organize meetups, so I decided to stop 
paying the annual fee to meetup.com .

I’m not particularly sad that it’s closing down, given that it has been 
inactive, and we as a community seem to find other ways to talk to each other. 
But if someone in the community would like to organize some meetups and is 
prepared to pay the fees, I’m happy to hand over the reins.

Julian

[1] https://www.meetup.com/Apache-Calcite/ 
 

[jira] [Created] (CALCITE-3422) reOpen CALCITE-1751 tests in PigRelBuilderStyleTest.

[xzh_dz (Jira)]

xzh_dz created CALCITE-3422:
---

 Summary: reOpen CALCITE-1751 tests in PigRelBuilderStyleTest.
 Key: CALCITE-3422
 URL: https://issues.apache.org/jira/browse/CALCITE-3422
 Project: Calcite
  Issue Type: Wish
Reporter: xzh_dz


reOpen CALCITE-1751 tests in PigRelBuilderStyleTest after disable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: CassandraAdapter (Add Type) and WHERE statement.

[Michael Mior]

You're right that there are several types which are not supported by
the Cassandra adapter. We would happily accept pull requests to add
support for new types.

You're also correct that Cassandra cannot efficiently execute queries
which do not specify the partition key. Calcite will make those
queries more efficient, but it can make it easier to execute queries
that CQL does not directly support. Ultimately data is still stored
based on the partition key, so if your query does not specify a
partition key, Calcite will still need to issue an expensive
cross-partition query to Cassandra.
--
Michael Mior
mm...@apache.org

Le mer. 16 oct. 2019 à 07:57, Yanna elina  a écrit :
>
> Hi guys ,
>
> I study Calcite the benefits that a Cassandra-Calcite Adapter can bring ,
> as for example brings the possibility of join.
>
> the problem type defined into CassandraSchema.getRelDataType(..) is very
> limited
> some important type are missing  boolean / array ect...
>
> I thought inherited from the class CassandraSchema for Override  this
> method and add more type but this method is used inside CassandraTable too.
>
> i would like to avoid  to re-write fully this adapter  :)
>
> do you have suggestions?
>
> My second question  is : Cassandra is not optimized to have WHERE on key
> not defined on cluster/partition key. I was wondering if calcite could play
> a role without this mechanism to improve performance
>
>
> Thank !
>
> Yanna

CassandraAdapter (Add Type) and WHERE statement.

[Yanna elina]

Hi guys ,

I study Calcite the benefits that a Cassandra-Calcite Adapter can bring ,
as for example brings the possibility of join.

the problem type defined into CassandraSchema.getRelDataType(..) is very
limited
some important type are missing  boolean / array ect...

I thought inherited from the class CassandraSchema for Override  this
method and add more type but this method is used inside CassandraTable too.

i would like to avoid  to re-write fully this adapter  :)

do you have suggestions?

My second question  is : Cassandra is not optimized to have WHERE on key
not defined on cluster/partition key. I was wondering if calcite could play
a role without this mechanism to improve performance


Thank !

Yanna

[jira] [Created] (CALCITE-3421) Reuse relMetadataQuery.

[xzh_dz (Jira)]

xzh_dz created CALCITE-3421:
---

 Summary: Reuse relMetadataQuery.
 Key: CALCITE-3421
 URL: https://issues.apache.org/jira/browse/CALCITE-3421
 Project: Calcite
  Issue Type: Wish
Reporter: xzh_dz


Reuse relMetadataQuery in 
TraitPropagationTest,VolcanoPlannerTraitTest,RelMetadataTest,PlannerTest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3420) NullPointerException throws for implicit type coercion of nested SET operations

[Danny Chen (Jira)]

Danny Chen created CALCITE-3420:
---

 Summary: NullPointerException throws for implicit type coercion of 
nested SET operations
 Key: CALCITE-3420
 URL: https://issues.apache.org/jira/browse/CALCITE-3420
 Project: Calcite
  Issue Type: Improvement
  Components: core
Affects Versions: 1.21.0
Reporter: Danny Chen
Assignee: Danny Chen
 Fix For: 1.22.0


Check this sql in TypeCoercionTest:

{code:sql}
select t1_varchar20 from t1 
  union select t2_int from t2
  union select t1_int from t1
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3419) Tools for converting List to SQL expression string

[jamie12221 (Jira)]

jamie12221 created CALCITE-3419:
---

 Summary: Tools for converting List  to SQL expression 
string
 Key: CALCITE-3419
 URL: https://issues.apache.org/jira/browse/CALCITE-3419
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jamie12221


'org.apache.calcite.schema.ProjectableFilterableTable' Is a simple tool that 
can apply filter expressions.On a datasource that uses SQL as a query command, 
the org.apache.calcite.schema.ProjectableFilterableTable#scan method applies a 
filter expression, and the filter expression may be converted to SQL. I have 
not found a way to convert List to SQL. In particular, OR expressions 
may involve some complicated operations.Is there a better solution?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: [DISCUSSION] Extension of Metadata Query

[Danny Chan]

This is the reason I was struggling for the discussion.

Best,
Danny Chan
在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道：
>
> RelMetadataQuery

Re: [DISCUSS] Make Enumerable operators responsive to interrupts

[Vladimir Sitnikov]

Statamis,
"cooperative to interrupt" sounds a nice idea, however, I have been bitten
multiple times by improper interrupt handling (not really with Calcite, but
with other projects).

In other words, it is good when everybody supports that.
However, the other libraries might receive unexpected
"interruptedexception", and they might go off rails.

For example, suppose you are implementing logger.info(...). What do you do
if you get an exception while logging?
Do you attempt to log it again? Do you attempt to log to System.err?
That is puzzling, and I have seen a case when the software performed 3
attempts, then it stopped logging completely
because it thought "the logfile is broken".

So:
1) It might worth adding "interrupted" checks in the executors
2) If Calcite ever uses .interrupt(), then it should be configurable (e.g.
to avoid cases when .interrupt() kills not that well-prepared code)

Vladimir

Re: [DISCUSS] Support Sql Hint for Calcite

[Danny Chan]

Thanks for the clarification.

I understand you worried. Yes, the effort/memory would be wasted or meaningless 
if hints are not used. This is just what a hint does, it is a “hint” and 
non-mandatory, but we should give the chance to let user see them, it is the 
use that decide if to use the hints and how to use them. For big queries I have 
no confidence to cover the corner cases. So can we mark this feature as 
experimental and used for simple queries(no decorrelation) first ?

For “reversible”, during the implementation, I try to make the modifications 
non-invasive with the current codes. That is why I made all the interfaces 
about the hint into one class named RelWithHInt. Different with trait, I didn’t 
force users to pass in the hints in the RelNode constructor. I think if is not 
a bigwork if we want to remove the API.

Best,
Danny Chan
在 2019年10月16日 +0800 AM11:14，Julian Hyde ，写道：
> By “skeptical” I mean that I think we can come up with a mechanism to copy 
> hints when applying planner rules, but even when we have implemented that 
> mechanism there will be many cases where people want a hint and that hint is 
> not copied to the RelNode where it is needed, and many other cases where we 
> spend the effort/memory of copying the hint to a RelNode and the hint is not 
> used.
>
> By “reversible” I mean if we come up with an API that does not work, how do 
> we change or remove that API without people complaining?
>
> Julian
>
>
> > On Oct 15, 2019, at 7:11 PM, Danny Chan  wrote:
> >
> > Thanks Julian
> >
> > > I am skeptical that RelWithHint will work for large queries.
> >
> > For “skeptical” do you mean how to transfer the hints during rule planning 
> > ? I’m also not that confident yet.
> >
> > > How do we introduce it in a reversible way
> > Do you mean transform the RelWithHint back into the SqlHint ? I didn’t 
> > implement it in current patch, but I think we have the ability to do that 
> > because we have a inheritPath for each RelWithHint, we can collect all the 
> > hints together and merge them into the SqlHints, then propagate these 
> > SqlHints to the SqlNodes.
> >
> > > What are the other options?
> > Do you mean the way to transfer hints during planning ? I have no other 
> > options yet.
> >
> > Best,
> > Danny Chan
> > 在 2019年10月16日 +0800 AM8:03，dev@calcite.apache.org，写道：
> > >
> > > I am skeptical that RelWithHint will work for large queries.
>