[jira] [Commented] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152601#comment-17152601
]
Christian Beikov commented on CALCITE-3840:
---
I'm sorry [~julianhyde] but I posted a link to the PR which contains a
reference to the commit. Thought that would be enough. So in the future, always
link the commit in the issue?
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
> Fix For: 1.23.0
>
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064696#comment-17064696
]
Christian Beikov commented on CALCITE-3840:
---
Fixed via 893d41fa4b79dbac6b613f214de6b4fbab9e6773
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
> Fix For: 1.23.0
>
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3810) Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter
[ https://issues.apache.org/jira/browse/CALCITE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064695#comment-17064695 ] Christian Beikov commented on CALCITE-3810: --- Fixed via 79d5001b945cbb9c47046b1b28b2345f78a72dc8 > Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter > -- > > Key: CALCITE-3810 > URL: https://issues.apache.org/jira/browse/CALCITE-3810 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Christian Beikov >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, when constructing an ANTI or SEMI join in the relational algebra, > it's not possible to render to an Sql AST. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Beikov resolved CALCITE-3840.
---
Fix Version/s: 1.23.0
Resolution: Fixed
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
> Fix For: 1.23.0
>
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3810) Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter
[ https://issues.apache.org/jira/browse/CALCITE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Beikov resolved CALCITE-3810. --- Fix Version/s: 1.23.0 Resolution: Fixed > Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter > -- > > Key: CALCITE-3810 > URL: https://issues.apache.org/jira/browse/CALCITE-3810 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Christian Beikov >Priority: Major > Labels: pull-request-available > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, when constructing an ANTI or SEMI join in the relational algebra, > it's not possible to render to an Sql AST. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055619#comment-17055619
]
Christian Beikov commented on CALCITE-3840:
---
Allright, I updated the PR with your suggested changes. If you could give me
one last confirmation, I can push it.
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055619#comment-17055619
]
Christian Beikov edited comment on CALCITE-3840 at 3/10/20, 6:10 AM:
-
Allright, I updated the PR with your suggested changes. If you could give me
one last confirmation that this is ok, I can push it.
was (Author: cbeikov):
Allright, I updated the PR with your suggested changes. If you could give me
one last confirmation, I can push it.
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
[
https://issues.apache.org/jira/browse/CALCITE-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054823#comment-17054823
]
Christian Beikov commented on CALCITE-3840:
---
Hey [~donnyzone] I already have a test and fix for it here:
https://github.com/apache/calcite/pull/1819
> Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC
> adapter
>
>
> Key: CALCITE-3840
> URL: https://issues.apache.org/jira/browse/CALCITE-3840
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.21.0
>Reporter: Christian Beikov
>Priority: Major
>
> Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
> (VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g.
> joining with such a VALUES, the RelToSqlConverter tries to re-alias this with
> a unique alias, but fails because it produces \{{FROM (VALUES((1))) AS
> t(col_alias) AS newAlias}}.
> The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3840) Re-aliasing of VALUES that has column aliases produces wrong SQL in the JDBC adapter
Christian Beikov created CALCITE-3840:
-
Summary: Re-aliasing of VALUES that has column aliases produces
wrong SQL in the JDBC adapter
Key: CALCITE-3840
URL: https://issues.apache.org/jira/browse/CALCITE-3840
Project: Calcite
Issue Type: Bug
Components: jdbc-adapter
Affects Versions: 1.21.0
Reporter: Christian Beikov
Rendering a VALUES relnode to e.g. PostgreSQL will produce \{{FROM
(VALUES((1))) AS t(col_alias)}} where "t" is a static alias. When e.g. joining
with such a VALUES, the RelToSqlConverter tries to re-alias this with a unique
alias, but fails because it produces \{{FROM (VALUES((1))) AS t(col_alias) AS
newAlias}}.
The fix is to replace the static table alias instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3810) Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter
[ https://issues.apache.org/jira/browse/CALCITE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041724#comment-17041724 ] Christian Beikov commented on CALCITE-3810: --- There you go, I updated the PR. > Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter > -- > > Key: CALCITE-3810 > URL: https://issues.apache.org/jira/browse/CALCITE-3810 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Christian Beikov >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, when constructing an ANTI or SEMI join in the relational algebra, > it's not possible to render to an Sql AST. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3810) Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter
[ https://issues.apache.org/jira/browse/CALCITE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Beikov updated CALCITE-3810: -- Summary: Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter (was: Implement Rel-to-Sql translation for ANTI and SEMI join) > Render ANTI and SEMI join to NOT EXISTS and EXISTS in the JDBC adapter > -- > > Key: CALCITE-3810 > URL: https://issues.apache.org/jira/browse/CALCITE-3810 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Christian Beikov >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, when constructing an ANTI or SEMI join in the relational algebra, > it's not possible to render to an Sql AST. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3810) Implement Rel-to-Sql translation for ANTI and SEMI join
Christian Beikov created CALCITE-3810: - Summary: Implement Rel-to-Sql translation for ANTI and SEMI join Key: CALCITE-3810 URL: https://issues.apache.org/jira/browse/CALCITE-3810 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.21.0 Reporter: Christian Beikov Currently, when constructing an ANTI or SEMI join in the relational algebra, it's not possible to render to an Sql AST. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2141) Implement query rewrite based on sharding configuration
[
https://issues.apache.org/jira/browse/CALCITE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030081#comment-17030081
]
Christian Beikov commented on CALCITE-2141:
---
I thought of something similar for querying i.e. introduce a sub-table per
partition and combine that into a view through UNION ALL like Julian proposed.
I think that's a pretty simple and straighforward implementation, but it
depends on the optimizer to push down predicates and be able to remove
unnecessary set operands, which is great as implementing these optimizations
will help in general.
If your student would also like to work on the writing part, it would be
awesome if writing could use the same technique i.e. implement writable views.
Internally, the expansion could be represented through writable CTEs.
{{UPDATE sometable t SET t.col = t.col || t.partiton WHERE MOD(t.partition,
2)=0}} would be rewritten to something like the following
{code:java}
WITH update1 AS(
UPDATE sometable_1 t SET t.col = t.col || t.partiton WHERE MOD(1, 2)=0
RETURNING *
), update2 AS(
UPDATE sometable_2 t SET t.col = t.col || t.partiton WHERE MOD(2, 2)=0
RETURNING *
)
SELECT SUM(t.c) FROM (
SELECT COUNT(*) c FROM update1 u1
UNION ALL
SELECT COUNT(*) FROM update2 u2
) t
{code}
> Implement query rewrite based on sharding configuration
> ---
>
> Key: CALCITE-2141
> URL: https://issues.apache.org/jira/browse/CALCITE-2141
> Project: Calcite
> Issue Type: New Feature
>Reporter: Christian Beikov
>Priority: Minor
>
> Based on topology changes, it should be possible to dynamically update a
> sharding configuration for calcite. The effect of such a configuration is,
> that a query involving sharded tables is rewritten to a more optimal form
> possibly targetting mutliple different datasources.
> This is an interesting building block for distributed databases but also for
> applications since it enables the implementation of a static sharding scheme.
> Doing the shard rewriting on a client is also a lot better when using a
> distributed database as that eliminates the need for a coordinator node
> through which data is tunneled.
> Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
> implementation.
> So imagine a topology with a master node and 2 worker nodes, one having
> shards with even numbers and the other one having shards with odd numbers.
> Table "A" is sharded by the column "tenant_id" into e.g. 30 shards. So the
> sharding configuration for table "A" would contain the information "worker 1
> has shards 1,3,5,..." and "worker 2 has shards 0,2,4,...". It also specifies
> that the sharding strategy should use a hash function {{shard = tenant_id %
> 30}}.
> When an application sends a query like e.g. {{select * from A where tenant_id
> = 1}}, normally the master/coordinator would do the shard rewriting, but
> doing this already at the client can eliminate the master as bottleneck for
> many scenarios. It is clear that the query can be completely fulfilled by
> worker 1 since it owns shard 1. The query rewriting therefore simply pushes
> the query to that worker. Note that there might be cases where a shard is
> replicated to other workers so it might be beneficial to make it configurable
> whether or when replicas should be used for querying.
> A query like \{{select * from A where tenant_id in(1,2)}} could be
> transformed to {{select * from worker1.A where tenant_id = 1 union all select
> * from worker2.A where tenant_id = 2}}. One optimization could be to target a
> single worker if it contains at least a replica of all required shards, but
> that would need to be configurable again since replicas might lag behind.
>
> DML statements obviously should be handled as well but at first, I would
> simply forbid the use of multiple workers within one transaction. Supporting
> multiple workers in a transaction will essentially require a 2PC and I'm not
> sure it's always a good idea to let an application be the coordinator for
> such a transaction. There should definitely be an option to let a
> master/coordinator node of a distributed database handle the details of the
> 2PC by configuring that DML statements should always be pushed to the
> master/coordinator node.
> The sharding-jdbc implementation only offers _BASE_ guarantees. I guess there
> are cases where this makes sense so making the transaction handling pluggable
> to allow other strategies would definitely be nice as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417895#comment-16417895
]
Christian Beikov commented on CALCITE-1965:
---
I'm lacking time as well. IMO the only reasonable way to implement this is by
doing it like in the papers I posted. I can review it as well and help with
tests, but getting the implementation right is going to be very time consuming.
Unless you have time to dig into this, I'm not sure there is much you could do.
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>Priority: Major
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2141) Implement query rewrite based on sharding configuration
[
https://issues.apache.org/jira/browse/CALCITE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Beikov updated CALCITE-2141:
--
Description:
Based on topology changes, it should be possible to dynamically update a
sharding configuration for calcite. The effect of such a configuration is, that
a query involving sharded tables is rewritten to a more optimal form possibly
targetting mutliple different datasources.
This is an interesting building block for distributed databases but also for
applications since it enables the implementation of a static sharding scheme.
Doing the shard rewriting on a client is also a lot better when using a
distributed database as that eliminates the need for a coordinator node through
which data is tunneled.
Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
implementation.
So imagine a topology with a master node and 2 worker nodes, one having shards
with even numbers and the other one having shards with odd numbers. Table "A"
is sharded by the column "tenant_id" into e.g. 30 shards. So the sharding
configuration for table "A" would contain the information "worker 1 has shards
1,3,5,..." and "worker 2 has shards 0,2,4,...". It also specifies that the
sharding strategy should use a hash function {{shard = tenant_id % 30}}.
When an application sends a query like e.g. {{select * from A where tenant_id =
1}}, normally the master/coordinator would do the shard rewriting, but doing
this already at the client can eliminate the master as bottleneck for many
scenarios. It is clear that the query can be completely fulfilled by worker 1
since it owns shard 1. The query rewriting therefore simply pushes the query to
that worker. Note that there might be cases where a shard is replicated to
other workers so it might be beneficial to make it configurable whether or when
replicas should be used for querying.
A query like \{{select * from A where tenant_id in(1,2)}} could be transformed
to {{select * from worker1.A where tenant_id = 1 union all select * from
worker2.A where tenant_id = 2}}. One optimization could be to target a single
worker if it contains at least a replica of all required shards, but that would
need to be configurable again since replicas might lag behind.
DML statements obviously should be handled as well but at first, I would simply
forbid the use of multiple workers within one transaction. Supporting multiple
workers in a transaction will essentially require a 2PC and I'm not sure it's
always a good idea to let an application be the coordinator for such a
transaction. There should definitely be an option to let a master/coordinator
node of a distributed database handle the details of the 2PC by configuring
that DML statements should always be pushed to the master/coordinator node.
The sharding-jdbc implementation only offers _BASE_ guarantees. I guess there
are cases where this makes sense so making the transaction handling pluggable
to allow other strategies would definitely be nice as well.
was:
Based on topology changes, it should be possible to dynamically update a
sharding configuration for calcite. The effect of such a configuration is, that
a query involving sharded tables is rewritten to a more optimal form possibly
targetting mutliple different datasources.
This is an interesting building block for distributed databases but also for
applications since it enables the implementation of a static sharding scheme.
Doing the shard rewriting on a client is also a lot better when using a
distributed database as that eliminates the need for a coordinator node through
which data is tunneled.
Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
implementation.
So imagine a topology with a master node and 2 worker nodes, one having shards
with even numbers and the other one having shards with odd numbers. A table "A"
is sharded by the column "tenant_id" into e.g. 30 shards. So the sharding
configuration for table "A" would contain the information "worker 1 has shards
1,3,5,..." and "worker 2 has shards 0,2,4,...". It also specifies that the
sharding strategy should use a hash function "shard = tenant_id % 30".
When an application sends a query like e.g. \{{select * from table where
tenant_id = 1}}, normally the master/coordinator
> Implement query rewrite based on sharding configuration
> ---
>
> Key: CALCITE-2141
> URL: https://issues.apache.org/jira/browse/CALCITE-2141
> Project: Calcite
> Issue Type: New Feature
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>Priority: Minor
>
> Based on topology changes, it should be possible to dynamically update a
> sharding configuration for calcite. The effect of such a configuration is,
> that a
[jira] [Updated] (CALCITE-2141) Implement query rewrite based on sharding configuration
[
https://issues.apache.org/jira/browse/CALCITE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Beikov updated CALCITE-2141:
--
Description:
Based on topology changes, it should be possible to dynamically update a
sharding configuration for calcite. The effect of such a configuration is, that
a query involving sharded tables is rewritten to a more optimal form possibly
targetting mutliple different datasources.
This is an interesting building block for distributed databases but also for
applications since it enables the implementation of a static sharding scheme.
Doing the shard rewriting on a client is also a lot better when using a
distributed database as that eliminates the need for a coordinator node through
which data is tunneled.
Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
implementation.
So imagine a topology with a master node and 2 worker nodes, one having shards
with even numbers and the other one having shards with odd numbers. A table "A"
is sharded by the column "tenant_id" into e.g. 30 shards. So the sharding
configuration for table "A" would contain the information "worker 1 has shards
1,3,5,..." and "worker 2 has shards 0,2,4,...". It also specifies that the
sharding strategy should use a hash function "shard = tenant_id % 30".
When an application sends a query like e.g. \{{select * from table where
tenant_id = 1}}, normally the master/coordinator
was:
Based on topology changes, it should be possible to dynamically update a
sharding configuration for calcite. The effect of such a configuration is, that
a query involving sharded tables is rewritten to a more optimal form possibly
targetting mutliple different datasources.
This is an interesting building block for distributed databases but also for
applications since it enables the implementation of a static sharding scheme.
Doing the shard rewriting on a client is also a lot better when using a
distributed database as that eliminates the need for a coordinator node through
which data is tunneled.
Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
implementation.
> Implement query rewrite based on sharding configuration
> ---
>
> Key: CALCITE-2141
> URL: https://issues.apache.org/jira/browse/CALCITE-2141
> Project: Calcite
> Issue Type: New Feature
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>Priority: Minor
>
> Based on topology changes, it should be possible to dynamically update a
> sharding configuration for calcite. The effect of such a configuration is,
> that a query involving sharded tables is rewritten to a more optimal form
> possibly targetting mutliple different datasources.
> This is an interesting building block for distributed databases but also for
> applications since it enables the implementation of a static sharding scheme.
> Doing the shard rewriting on a client is also a lot better when using a
> distributed database as that eliminates the need for a coordinator node
> through which data is tunneled.
> Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing
> implementation.
> So imagine a topology with a master node and 2 worker nodes, one having
> shards with even numbers and the other one having shards with odd numbers. A
> table "A" is sharded by the column "tenant_id" into e.g. 30 shards. So the
> sharding configuration for table "A" would contain the information "worker 1
> has shards 1,3,5,..." and "worker 2 has shards 0,2,4,...". It also specifies
> that the sharding strategy should use a hash function "shard = tenant_id %
> 30".
> When an application sends a query like e.g. \{{select * from table where
> tenant_id = 1}}, normally the master/coordinator
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2141) Implement query rewrite based on sharding configuration
Christian Beikov created CALCITE-2141: - Summary: Implement query rewrite based on sharding configuration Key: CALCITE-2141 URL: https://issues.apache.org/jira/browse/CALCITE-2141 Project: Calcite Issue Type: New Feature Reporter: Christian Beikov Assignee: Julian Hyde Based on topology changes, it should be possible to dynamically update a sharding configuration for calcite. The effect of such a configuration is, that a query involving sharded tables is rewritten to a more optimal form possibly targetting mutliple different datasources. This is an interesting building block for distributed databases but also for applications since it enables the implementation of a static sharding scheme. Doing the shard rewriting on a client is also a lot better when using a distributed database as that eliminates the need for a coordinator node through which data is tunneled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2141) Implement query rewrite based on sharding configuration
[ https://issues.apache.org/jira/browse/CALCITE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Beikov updated CALCITE-2141: -- Description: Based on topology changes, it should be possible to dynamically update a sharding configuration for calcite. The effect of such a configuration is, that a query involving sharded tables is rewritten to a more optimal form possibly targetting mutliple different datasources. This is an interesting building block for distributed databases but also for applications since it enables the implementation of a static sharding scheme. Doing the shard rewriting on a client is also a lot better when using a distributed database as that eliminates the need for a coordinator node through which data is tunneled. Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing implementation. was: Based on topology changes, it should be possible to dynamically update a sharding configuration for calcite. The effect of such a configuration is, that a query involving sharded tables is rewritten to a more optimal form possibly targetting mutliple different datasources. This is an interesting building block for distributed databases but also for applications since it enables the implementation of a static sharding scheme. Doing the shard rewriting on a client is also a lot better when using a distributed database as that eliminates the need for a coordinator node through which data is tunneled. > Implement query rewrite based on sharding configuration > --- > > Key: CALCITE-2141 > URL: https://issues.apache.org/jira/browse/CALCITE-2141 > Project: Calcite > Issue Type: New Feature >Reporter: Christian Beikov >Assignee: Julian Hyde >Priority: Minor > > Based on topology changes, it should be possible to dynamically update a > sharding configuration for calcite. The effect of such a configuration is, > that a query involving sharded tables is rewritten to a more optimal form > possibly targetting mutliple different datasources. > This is an interesting building block for distributed databases but also for > applications since it enables the implementation of a static sharding scheme. > Doing the shard rewriting on a client is also a lot better when using a > distributed database as that eliminates the need for a coordinator node > through which data is tunneled. > Also see [https://github.com/shardingjdbc/sharding-jdbc] for an existing > implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2079) [Not working with ES5] java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
[
https://issues.apache.org/jira/browse/CALCITE-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279243#comment-16279243
]
Christian Beikov commented on CALCITE-2079:
---
It depends on whether we add automated tests to verify this. I'm not sure how
an automated test could look like, but I guess it'd involve starting a separate
JVM and excercising it a bit to ensure no exceptions are thrown due to class
loading issues. It's certainly possible, but since dependencies change
seldomly, I'd personally also be ok with not having tests for that and doing a
quick fix release in case something was missed. What do you think [~julianhyde]?
> [Not working with ES5] java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/Logger
> --
>
> Key: CALCITE-2079
> URL: https://issues.apache.org/jira/browse/CALCITE-2079
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Affects Versions: 1.15.0
>Reporter: Michael Despotopoulos
>Assignee: Julian Hyde
> Labels: NoClassDefFoundError, elasticsearch, log4j
>
> Initially I had this problem described here:
> http://mail-archives.apache.org/mod_mbox/calcite-issues/201709.mbox/%3cjira.13106083.1506702636000.238618.1506710701...@atlassian.jira%3E
> Then I removed every occurrence of Elasticsearch2 in pom.xml and sqlline
> script and moved on to the current error I am facing which is the following:
> {code:java}
> sqlline version 1.3.0
> sqlline> !connect
> jdbc:calcite:model=elasticsearch5/src/test/resources/model.json admin admin
> java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
> at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:101)
> at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.(AbstractXContentParser.java:57)
> at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.(JsonXContentParser.java:44)
> at
> org.elasticsearch.common.xcontent.json.JsonXContent.createParser(JsonXContent.java:103)
> at
> org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:848)
> at
> org.elasticsearch.common.settings.Setting.lambda$listSetting$27(Setting.java:802)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:807)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:802)
> at
> org.elasticsearch.common.network.NetworkService.(NetworkService.java:50)
> at
> org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:98)
> at
> org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)
> at
> org.elasticsearch.client.transport.TransportClient.(TransportClient.java:268)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:127)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:113)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:103)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.open(Elasticsearch5Schema.java:119)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.(Elasticsearch5Schema.java:74)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5SchemaFactory.create(Elasticsearch5SchemaFactory.java:56)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:270)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:196)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:88)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
> at sqlline.Commands.connect(Commands.java:1095)
> at sqlline.Commands.connect(Commands.java:1001)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:791)
> at sqlline.SqlLine.begin(SqlLine.java:668)
> at sqlline.SqlLine.start(SqlLine.java:373)
>
[jira] [Commented] (CALCITE-2079) [Not working with ES5] java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
[
https://issues.apache.org/jira/browse/CALCITE-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279079#comment-16279079
]
Christian Beikov commented on CALCITE-2079:
---
Basically, all the modules we have in calcite, like core, es2, es5 etc. should
have such a module.xml file. Every runtime dependency of a module should either
be satisfied via a jar defined via a resource-root or via a module dependency.
If you have a module for core, then the es5 module will obviously have a module
dependency on the core module rather than include the jar in it's module
content and refer to it via the resource-root.
When you have all the module.xml files and assembly procedures, we can modify
sqlline to use [JBoss
Modules|https://jboss-modules.github.io/jboss-modules/manual/]. It will use the
filesystem module repository built by the assembly procedures to load modules
in isolation. After that, we only need to update the scripts and should be good
to go.
> [Not working with ES5] java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/Logger
> --
>
> Key: CALCITE-2079
> URL: https://issues.apache.org/jira/browse/CALCITE-2079
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Affects Versions: 1.15.0
>Reporter: Michael Despotopoulos
>Assignee: Julian Hyde
> Labels: NoClassDefFoundError, elasticsearch, log4j
>
> Initially I had this problem described here:
> http://mail-archives.apache.org/mod_mbox/calcite-issues/201709.mbox/%3cjira.13106083.1506702636000.238618.1506710701...@atlassian.jira%3E
> Then I removed every occurrence of Elasticsearch2 in pom.xml and sqlline
> script and moved on to the current error I am facing which is the following:
> {code:java}
> sqlline version 1.3.0
> sqlline> !connect
> jdbc:calcite:model=elasticsearch5/src/test/resources/model.json admin admin
> java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
> at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:101)
> at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.(AbstractXContentParser.java:57)
> at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.(JsonXContentParser.java:44)
> at
> org.elasticsearch.common.xcontent.json.JsonXContent.createParser(JsonXContent.java:103)
> at
> org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:848)
> at
> org.elasticsearch.common.settings.Setting.lambda$listSetting$27(Setting.java:802)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:807)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:802)
> at
> org.elasticsearch.common.network.NetworkService.(NetworkService.java:50)
> at
> org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:98)
> at
> org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)
> at
> org.elasticsearch.client.transport.TransportClient.(TransportClient.java:268)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:127)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:113)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:103)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.open(Elasticsearch5Schema.java:119)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.(Elasticsearch5Schema.java:74)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5SchemaFactory.create(Elasticsearch5SchemaFactory.java:56)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:270)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:196)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:88)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
> at sqlline.Commands.connect(Commands.java:1095)
> at sqlline.Commands.connect(Commands.java:1001)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (CALCITE-2079) [Not working with ES5] java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
[
https://issues.apache.org/jira/browse/CALCITE-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279012#comment-16279012
]
Christian Beikov commented on CALCITE-2079:
---
In CALCITE-1994 you can find a lengthy discussion about the topic. I personally
don't plan to do this any time soon, yet I will do it eventually if nobody else
steps up to do it.
I can guide you if you want to give it a try. It shouldn't be very complex,
it's just time consuming and I'm on a pretty tight schedule currently.
> [Not working with ES5] java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/Logger
> --
>
> Key: CALCITE-2079
> URL: https://issues.apache.org/jira/browse/CALCITE-2079
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Affects Versions: 1.15.0
>Reporter: Michael Despotopoulos
>Assignee: Julian Hyde
> Labels: NoClassDefFoundError, elasticsearch, log4j
>
> Initially I had this problem described here:
> http://mail-archives.apache.org/mod_mbox/calcite-issues/201709.mbox/%3cjira.13106083.1506702636000.238618.1506710701...@atlassian.jira%3E
> Then I removed every occurrence of Elasticsearch2 in pom.xml and sqlline
> script and moved on to the current error I am facing which is the following:
> {code:java}
> sqlline version 1.3.0
> sqlline> !connect
> jdbc:calcite:model=elasticsearch5/src/test/resources/model.json admin admin
> java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
> at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:101)
> at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.(AbstractXContentParser.java:57)
> at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.(JsonXContentParser.java:44)
> at
> org.elasticsearch.common.xcontent.json.JsonXContent.createParser(JsonXContent.java:103)
> at
> org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:848)
> at
> org.elasticsearch.common.settings.Setting.lambda$listSetting$27(Setting.java:802)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:807)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:802)
> at
> org.elasticsearch.common.network.NetworkService.(NetworkService.java:50)
> at
> org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:98)
> at
> org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)
> at
> org.elasticsearch.client.transport.TransportClient.(TransportClient.java:268)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:127)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:113)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:103)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.open(Elasticsearch5Schema.java:119)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.(Elasticsearch5Schema.java:74)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5SchemaFactory.create(Elasticsearch5SchemaFactory.java:56)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:270)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:196)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:88)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
> at sqlline.Commands.connect(Commands.java:1095)
> at sqlline.Commands.connect(Commands.java:1001)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:791)
> at sqlline.SqlLine.begin(SqlLine.java:668)
> at sqlline.SqlLine.start(SqlLine.java:373)
> at sqlline.SqlLine.main(SqlLine.java:265)
> Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger
> at
[jira] [Commented] (CALCITE-2079) [Not working with ES5] java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
[
https://issues.apache.org/jira/browse/CALCITE-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278950#comment-16278950
]
Christian Beikov commented on CALCITE-2079:
---
Without proper class loader isolation we won't be able to get ES5 working for
sqlline.
> [Not working with ES5] java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/Logger
> --
>
> Key: CALCITE-2079
> URL: https://issues.apache.org/jira/browse/CALCITE-2079
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Affects Versions: 1.15.0
>Reporter: Michael Despotopoulos
>Assignee: Julian Hyde
> Labels: NoClassDefFoundError, elasticsearch, log4j
>
> Initially I had this problem described here:
> http://mail-archives.apache.org/mod_mbox/calcite-issues/201709.mbox/%3cjira.13106083.1506702636000.238618.1506710701...@atlassian.jira%3E
> Then I removed every occurrence of Elasticsearch2 in pom.xml and sqlline
> script and moved on to the current error I am facing which is the following:
> {code:java}
> sqlline version 1.3.0
> sqlline> !connect
> jdbc:calcite:model=elasticsearch5/src/test/resources/model.json admin admin
> java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
> at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:101)
> at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.(AbstractXContentParser.java:57)
> at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.(JsonXContentParser.java:44)
> at
> org.elasticsearch.common.xcontent.json.JsonXContent.createParser(JsonXContent.java:103)
> at
> org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:848)
> at
> org.elasticsearch.common.settings.Setting.lambda$listSetting$27(Setting.java:802)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:807)
> at
> org.elasticsearch.common.settings.Setting.listSetting(Setting.java:802)
> at
> org.elasticsearch.common.network.NetworkService.(NetworkService.java:50)
> at
> org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:98)
> at
> org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)
> at
> org.elasticsearch.client.transport.TransportClient.(TransportClient.java:268)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:127)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:113)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:103)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.open(Elasticsearch5Schema.java:119)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.(Elasticsearch5Schema.java:74)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5SchemaFactory.create(Elasticsearch5SchemaFactory.java:56)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:270)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:196)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:88)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
> at sqlline.Commands.connect(Commands.java:1095)
> at sqlline.Commands.connect(Commands.java:1001)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:791)
> at sqlline.SqlLine.begin(SqlLine.java:668)
> at sqlline.SqlLine.start(SqlLine.java:373)
> at sqlline.SqlLine.main(SqlLine.java:265)
> Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262041#comment-16262041 ] Christian Beikov commented on CALCITE-1940: --- I haven't yet looked into the JdbcConvention issue and I'm not sure I will have time for that before mid-december, so don't think this will make it into the release. > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1994) Elasticsearch5 not working using sqlline
[
https://issues.apache.org/jira/browse/CALCITE-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209142#comment-16209142
]
Christian Beikov commented on CALCITE-1994:
---
That's because you need at least an empty user config like
{noformat}
"userConfig": "{}"
{noformat}
When trying to reproduce this, I went through a lot of trouble regarding
classpath problems. The problems unfortunately won't stop if you specify the
user config correctly.
The next problem will be regarding some missing netty methods because of
mutliple netty versions being on the classpath.
All in all, I'm not sure it is so easy to get sqlline to work correctly for
elasticsearch. The sqlline scripts require a major overhaul to make this work.
What do you say [~julianhyde]?
> Elasticsearch5 not working using sqlline
>
>
> Key: CALCITE-1994
> URL: https://issues.apache.org/jira/browse/CALCITE-1994
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Affects Versions: 1.13.0
>Reporter: Kunwar Deep Singh
>Assignee: Julian Hyde
>Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I have been trying to connect to elasticsearch 5.1.1 server using sqlline.
> The following error is generated.
> The following is the error:
> {noformat}
> java.lang.NoSuchMethodError:
> org.elasticsearch.transport.client.PreBuiltTransportClient.addPlugins(Ljava/util/Collectio
> n;Ljava/util/Collection;)Ljava/util/Collection;
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:130)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:116)
> at
> org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:106)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.open(Elasticsearch5Schema.java:119)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5Schema.(Elasticsearch5Schema.java:74)
> at
> org.apache.calcite.adapter.elasticsearch5.Elasticsearch5SchemaFactory.create(Elasticsearch5SchemaFactory.java:56)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:269)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:195)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:87)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
> at sqlline.Commands.connect(Commands.java:1095)
> at sqlline.Commands.connect(Commands.java:1001)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:791)
> at sqlline.SqlLine.begin(SqlLine.java:668)
> at sqlline.SqlLine.start(SqlLine.java:373)
> at sqlline.SqlLine.main(SqlLine.java:265)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1998) Hive - Version specific handling for NULLS FIRST/ NULLS LAST
[ https://issues.apache.org/jira/browse/CALCITE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200047#comment-16200047 ] Christian Beikov commented on CALCITE-1998: --- Looks nice, I added some comments to the PR. When that's done, I'm fine with merging this. > Hive - Version specific handling for NULLS FIRST/ NULLS LAST > > > Key: CALCITE-1998 > URL: https://issues.apache.org/jira/browse/CALCITE-1998 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.15.0 >Reporter: Abbas Gadhia >Assignee: Julian Hyde >Priority: Minor > Fix For: 1.15.0 > > > This JIRA is for PR https://github.com/apache/calcite/pull/545 > We are making the HiveSqlDialect version aware for the NULLS FIRST and NULLS > LAST feature. In https://issues.apache.org/jira/browse/HIVE-12994 , the > authors clarified that the default NullCollation for Hive is > "NullDirection.LOW". Currently, the DEFAULT set in Calcite for Hive is > NullCollation.HIGH > In this PR, we are making 2 changes. > # Change the default NullCollation from HIGH to LOW > # Add NullCollation emulation in the HiveSqlDialect when the version of the > dialect is less that 2.1.0 or when the version is blank. > We're also adding a new Dialect "BigQuerySqlDialect" with the "quoteString" > as "`" based on > https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1998) Hive - Version specific handling for NULLS FIRST/ NULLS LAST
[ https://issues.apache.org/jira/browse/CALCITE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194276#comment-16194276 ] Christian Beikov commented on CALCITE-1998: --- I don't think it's a good idea to put the product version in the dialect. Just let the constructor of the dialect determine capabilities based on version rather then recheck the capabilities based on version every time. I'm also not sure about the dependency, the version checking can be easily implemented within the code base. > Hive - Version specific handling for NULLS FIRST/ NULLS LAST > > > Key: CALCITE-1998 > URL: https://issues.apache.org/jira/browse/CALCITE-1998 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.15.0 >Reporter: Abbas Gadhia >Assignee: Julian Hyde >Priority: Minor > Fix For: 1.15.0 > > > This JIRA is for PR https://github.com/apache/calcite/pull/545 > We are making the HiveSqlDialect version aware for the NULLS FIRST and NULLS > LAST feature. In https://issues.apache.org/jira/browse/HIVE-12994 , the > authors clarified that the default NullCollation for Hive is > "NullDirection.LOW". Currently, the DEFAULT set in Calcite for Hive is > NullCollation.HIGH > In this PR, we are making 2 changes. > # Change the default NullCollation from HIGH to LOW > # Add NullCollation emulation in the HiveSqlDialect when the version of the > dialect is less that 2.1.0 or when the version is blank. > We're also adding a new Dialect "BigQuerySqlDialect" with the "quoteString" > as "`" based on > https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172813#comment-16172813 ] Christian Beikov commented on CALCITE-1940: --- I looked through your changes and they look good to me so far, but as I mentioned a few times, the sequence PR isn't ready yet :D I am still struggling with getting the sequence call being executed with the JdbcConvention because of (wrong?) planning, but maybe you have some time to take a look and find a solution for this. I'm a bit busy with other stuff now until the weekend anyway, so just tell me when you are ready so that I can pick up work from that point. > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1913) Include DB version in SqlDialect
[ https://issues.apache.org/jira/browse/CALCITE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169645#comment-16169645 ] Christian Beikov commented on CALCITE-1913: --- Can you review again now please? I updated the PR > Include DB version in SqlDialect > > > Key: CALCITE-1913 > URL: https://issues.apache.org/jira/browse/CALCITE-1913 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.13.0 >Reporter: Jess Balint >Assignee: Jess Balint >Priority: Minor > > It would be useful to have the DB version # in the SqlDialect for unparsing -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1913) Include DB version in SqlDialect
[
https://issues.apache.org/jira/browse/CALCITE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169632#comment-16169632
]
Christian Beikov commented on CALCITE-1913:
---
Dependencies to {{RelDataType}} and {{SqlNode}} are kind of required. I mean we
could do it somehow, it's just not practial IMO especially since we already
have a dependency on other Sql types because of unparsing of calls.
I'll try to remove the usage of {{SqlImplementor.Context}} and the use of
DatabaseMetadata.
> Include DB version in SqlDialect
>
>
> Key: CALCITE-1913
> URL: https://issues.apache.org/jira/browse/CALCITE-1913
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.13.0
>Reporter: Jess Balint
>Assignee: Jess Balint
>Priority: Minor
>
> It would be useful to have the DB version # in the SqlDialect for unparsing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160314#comment-16160314 ] Christian Beikov commented on CALCITE-1940: --- [~julianhyde] I don't think it makes sense as that would require quite a rewrite and might not be fully what you agree with either. Apart from that, the sequence calls are still not pushed down to the DBMS yet so this is still a bit away from being done. > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1913) Include DB version in SqlDialect
[ https://issues.apache.org/jira/browse/CALCITE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160312#comment-16160312 ] Christian Beikov commented on CALCITE-1913: --- What does it mean to "behave like Postgres"? For most parts, some forks could be considered to "behave like Postgres" but don't support feature X,Y,Z. Having to introduce additional enum entries(maybe even for various versions of a "product") and to touch switch-code later because of such a wrong assumption is IMO wrong. The mere fact that we expose this vaguely defined thing called "DatabaseProduct" is a problem. People will base assumptions on that, which at a later point might become wrong. Having methods that report capabilities are unambiguous and since we intend to allow users to extend SqlDailect, they can choose to report less, or more capabilities based on their needs. Think about what people might have to do to avoid a single capability to be choosen. Maybe they'd have to return a wrong DatabaseProduct just to get a desired behavior. Also, this stuff is not backwards compatibility friendly. If we choose later to enable the use of a capability for existing DatabaseProducts that users don't want, they have no possibility to revert that rather than forking and building their own calcite version. I agree with you that it's generally harder to determine(requires looking through classes), what SqlDialects support a specific feature, but we'd gain the ability to make this extendible. Maybe others could comment on what they prefer or think about the matter? > Include DB version in SqlDialect > > > Key: CALCITE-1913 > URL: https://issues.apache.org/jira/browse/CALCITE-1913 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.13.0 >Reporter: Jess Balint >Assignee: Jess Balint >Priority: Minor > > It would be useful to have the DB version # in the SqlDialect for unparsing -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1913) Include DB version in SqlDialect
[ https://issues.apache.org/jira/browse/CALCITE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157280#comment-16157280 ] Christian Beikov commented on CALCITE-1913: --- I implemented part of what you describe as part of a PR that adds sequence support. Maybe you could comment on that? [~jbal...@gmail.com] https://github.com/apache/calcite/pull/514 By allowing a user to provide a custom SqlDialectFactory, a user could make use of a custom dialect implementation. That way, a consumer of calcite can overrule if a feature is used or not. Still, the default implementation can make a decision about whether a feature is used by doing version comparisons. That obviously requires that SqlDialect is the "gate-keeper" to what a dialect actually supports rather than handling that through checks against DatabaseProduct. My prime example for when the DatabaseProduct approach will break is MySQL. With version 8 it will support CTEs thus we could push down such SQL, yet we only have a single MYSQL enum entry. We could introduce multiple entries for respective versions(e.g. MYSQL5, MYSQL8) to handle such cases, but that won't handle cases where a user might not want to use a specific feature. Allowing SqlDialect to handle unparsing or return whether a DBMS has a specific capability also allows for adding new dialects that don't fit into the current DatabaseProduct scheme. What if there is a MySQL fork out there that doesn't support feature X,Y,Z of our "standard MySQL" version we defined through a DatabaseProduct? These people would be out of luck right now. Maybe you ([~julianhyde]) could comment further about what he doesn't like about this approach so we could tackle specifics? > Include DB version in SqlDialect > > > Key: CALCITE-1913 > URL: https://issues.apache.org/jira/browse/CALCITE-1913 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.13.0 >Reporter: Jess Balint >Assignee: Jess Balint >Priority: Minor > > It would be useful to have the DB version # in the SqlDialect for unparsing -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[
https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156072#comment-16156072
]
Christian Beikov commented on CALCITE-1940:
---
The idea is to encapsulate all the dialect specifics and at the same time
making the system pluggable. By doing the non-standard unparsing stuff through
dialect methods, it is possible for consumers of calcite to define a custom
dialect targeted for their needs. Right now, only sequence fetch rendering is
handled that way, but I plan to remove all checks against the database product
of a dialect like e.g. {{dialect.getDatabaseProduct() ==
DatabaseProduct.MYSQL}} that are currently spread around the code base. I'd
rather have methods that return whether a dialect supports a specific feature
or handle rendering/unparsing an expression, clause or whole query than checks
in the main code base that only work for specific database products.
One of the most recent changes of supported features is MySQL since version 8
will have support for e.g. CTEs. How would the current code handle that? The
way I am proposing it, the MysqlDialect can self configure during
instantiation, detect based on the version if a feature is supported and make
the capability accessible to the code that handles the unparsing.
Apart from that, I'd still need to know how to actually make the planner choose
the correct plan. Since there is no Jdbc input to the Project node, the query
isn't rendered with the JdbcConvention. Any idea how I could achieve that?
> Implement dialect specific support for sequences
>
>
> Key: CALCITE-1940
> URL: https://issues.apache.org/jira/browse/CALCITE-1940
> Project: Calcite
> Issue Type: Improvement
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> The Calcite parser and validator already supports sequences but the push down
> to the JDBC level is currently limited. SInce sequences are not supported on
> all DBMS every sql dialect should have the possibility to render it's own way
> of extracting sequence information or incrementing the value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[
https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152922#comment-16152922
]
Christian Beikov commented on CALCITE-1940:
---
No worries. I didn't bet on this getting into 1.14 especially since I wanted to
tackle the sql dialect specific handling as part of this.
The lifecycle doesn't change, it's still bound to the schema, the handler was
just merged into the dialect and dialect is now self configuring in the
constructor. This is only a first step, so please don't look to deep into that
yet.
For now, I would just like to make a {{RexSeqCall}} be executed with the
appropriate convention. Currently, the call is implemented via RexImpTable and
I have no idea how to force that expression be implemented by the
JdbcConvention. Any idea how to do that?
After that, I would replace all the static {{DatabaseProduct}} usages with
appropriate methods in the dialect to handle implementing the SQL translation.
When I'm done with that, you can review it as a whole, but the current state is
still incomplete.
> Implement dialect specific support for sequences
>
>
> Key: CALCITE-1940
> URL: https://issues.apache.org/jira/browse/CALCITE-1940
> Project: Calcite
> Issue Type: Improvement
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> The Calcite parser and validator already supports sequences but the push down
> to the JDBC level is currently limited. SInce sequences are not supported on
> all DBMS every sql dialect should have the possibility to render it's own way
> of extracting sequence information or incrementing the value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151573#comment-16151573 ] Christian Beikov commented on CALCITE-1940: --- I introduced a custom RexCall subclass since I need to store the RelNode representing the sequence table, but now I'd need to enforce the usage of the "correct" plan. Currently the optimizer generates plans that use the linq4j implementation which is "wrong". Maybe you could take a look at the PR and tell me how to do that best? > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[
https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151561#comment-16151561
]
Christian Beikov commented on CALCITE-1940:
---
Ok I understand how that works now, I was just confused because the SQL string
was generated(apparently for no reason?) within the Prepare class. Anyway, I'm
having a different problem now with the sequence implementation. The plan looks
like {{LogicalProject(NEXT VALUE FOR S2) -> LogicalValues( (0) )}} and during
rule registration, no JDBC rules are registered since there is no JDBC relation
used in RelNode. Any hints on how I can achieve that the plan gets converted to
{{JdbcProject(NEXT VALUE FOR S2) -> JdbcValues( (0) )}}?
> Implement dialect specific support for sequences
>
>
> Key: CALCITE-1940
> URL: https://issues.apache.org/jira/browse/CALCITE-1940
> Project: Calcite
> Issue Type: Improvement
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> The Calcite parser and validator already supports sequences but the push down
> to the JDBC level is currently limited. SInce sequences are not supported on
> all DBMS every sql dialect should have the possibility to render it's own way
> of extracting sequence information or incrementing the value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[
https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148807#comment-16148807
]
Christian Beikov commented on CALCITE-1940:
---
I need the dialect since I have to call the unparse method on it. If we have a
query like e.g. "select next value for s1.seq, next value for s2.seq" and s1
and s2 are schemas that have different dialects, I have to unparse the sequence
call once for the dialect of s1 and once for s2. I somehow need to know in the
{{SqlWriter}} which dialect a schema uses so that I can call the appropriate
unparse method.
Could you([~chris-baynes]) maybe comment on the mailing list thread
"JdbcAdapter Sort Limit and Offset" which is somewhat related in the overall
discussion about how to handle the different dialect quirks. I'd like to
essentially merge the {{Handler}} and other methods into a DBMS specific
{{SqlDialect}} implementation and do the dialect construction through a
configurable factory.
> Implement dialect specific support for sequences
>
>
> Key: CALCITE-1940
> URL: https://issues.apache.org/jira/browse/CALCITE-1940
> Project: Calcite
> Issue Type: Improvement
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> The Calcite parser and validator already supports sequences but the push down
> to the JDBC level is currently limited. SInce sequences are not supported on
> all DBMS every sql dialect should have the possibility to render it's own way
> of extracting sequence information or incrementing the value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148071#comment-16148071 ] Christian Beikov commented on CALCITE-1940: --- Depends on what you want changed. I was kind of waiting for a comment from [~chris-baynes] on how to do that best. The part about retrieving the dialect during unparsing is still unclear to me. How should that be done? Currently I am, as you stated, wrongly assuming a single dialect. > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-1971) Use embedded elastic search in tests
Christian Beikov created CALCITE-1971: - Summary: Use embedded elastic search in tests Key: CALCITE-1971 URL: https://issues.apache.org/jira/browse/CALCITE-1971 Project: Calcite Issue Type: Improvement Reporter: Christian Beikov Assignee: Julian Hyde As discussed in CALCITE-1967 we should use the embedded elastic search server instead to be able to run integration tests locally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145529#comment-16145529
]
Christian Beikov commented on CALCITE-1965:
---
The paper is a bit hard to follow for me, the first one(from Goldstein) was a
lot less formal and easier to read. I guess I understand some of the basic
concepts, but I am unsure if it's that easy to implement.
Maybe you([~jcamachorodriguez]) understand it better and have an idea of how to
implement it? I could try to give it a shot next week or so, but I'd first like
to hear your opinion on the paper and the approach.
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145256#comment-16145256
]
Christian Beikov edited comment on CALCITE-1965 at 8/29/17 1:16 PM:
By the way, found a paper from a co-author of the paper we were discussing that
seems to provide an approach for outer joins. Reading it now:
http://webcache.googleusercontent.com/search?q=cache:U7Uamrnk1f0J:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.85.9467%26rep%3Drep1%26type%3Dpdf+=2=de=clnk=at=lang_de%7Clang_en
This one here is about maintenance:
http://webcache.googleusercontent.com/search?q=cache:gT2fJv28YvwJ:citeseerx.ist.psu.edu/viewdoc/download%3Bjsessionid%3DA83B59DDEFE8F2BEC86FA2D77880AC6A%3Fdoi%3D10.1.1.85.7733%26rep%3Drep1%26type%3Dpdf+=6=de=clnk=at=lang_de%7Clang_en
was (Author: christian.beikov):
By the way, found a paper from a co-author of the paper we were discussing that
seems to provide an approach for outer joins. Reading it now:
http://webcache.googleusercontent.com/search?q=cache:gT2fJv28YvwJ:citeseerx.ist.psu.edu/viewdoc/download%3Bjsessionid%3DA83B59DDEFE8F2BEC86FA2D77880AC6A%3Fdoi%3D10.1.1.85.7733%26rep%3Drep1%26type%3Dpdf+=6=de=clnk=at=lang_de%7Clang_en
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145256#comment-16145256
]
Christian Beikov commented on CALCITE-1965:
---
By the way, found a paper from a co-author of the paper we were discussind that
seems to provide an approach for outer joins. Reading it now:
http://webcache.googleusercontent.com/search?q=cache:gT2fJv28YvwJ:citeseerx.ist.psu.edu/viewdoc/download%3Bjsessionid%3DA83B59DDEFE8F2BEC86FA2D77880AC6A%3Fdoi%3D10.1.1.85.7733%26rep%3Drep1%26type%3Dpdf+=6=de=clnk=at=lang_de%7Clang_en
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145256#comment-16145256
]
Christian Beikov edited comment on CALCITE-1965 at 8/29/17 1:13 PM:
By the way, found a paper from a co-author of the paper we were discussing that
seems to provide an approach for outer joins. Reading it now:
http://webcache.googleusercontent.com/search?q=cache:gT2fJv28YvwJ:citeseerx.ist.psu.edu/viewdoc/download%3Bjsessionid%3DA83B59DDEFE8F2BEC86FA2D77880AC6A%3Fdoi%3D10.1.1.85.7733%26rep%3Drep1%26type%3Dpdf+=6=de=clnk=at=lang_de%7Clang_en
was (Author: christian.beikov):
By the way, found a paper from a co-author of the paper we were discussind that
seems to provide an approach for outer joins. Reading it now:
http://webcache.googleusercontent.com/search?q=cache:gT2fJv28YvwJ:citeseerx.ist.psu.edu/viewdoc/download%3Bjsessionid%3DA83B59DDEFE8F2BEC86FA2D77880AC6A%3Fdoi%3D10.1.1.85.7733%26rep%3Drep1%26type%3Dpdf+=6=de=clnk=at=lang_de%7Clang_en
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145198#comment-16145198
]
Christian Beikov commented on CALCITE-1965:
---
To support matching between queries containing equivalent subqueries/derived
tables we could do a nested matching. Imagine the following view
{noformat}
select *
from a
left join (
select b.a as x
from b
left join c on b.b = c.b
) on a.a = x
{noformat}
EQ={}
OEQ={a.a, x}
=> x is a marker for nested matching
xEQ={}
xOEQ={b.b, c.b}
If such a marker is encountered during matching, then an nested matching is
invoked on the subqueries. The nested matching is a bit different from the
normal matching. The query's xEQ and view's xEQ must be equal. If only
non-nullable FK joins are allowed, it should be sufficient that qxEQ is a
subset of vxEQ. The query's xOEQ must be a strict(including type equality)
prefix of the view's xOEQ.
What do you say? Think this could work?
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145106#comment-16145106
]
Christian Beikov commented on CALCITE-1965:
---
To handle full outer joins I think it should be enough to mark the subsets of
OEQ with a type as being either "full" or "left". The matching of qOEQ with
vOEQ remains based on order and will additionally match the type, whereas
prefix matching of qEQ against vOEQ can handle full outer joins by introducing
an additional compensation predicate for the right side.
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1967) Add support for elastic search 5
[ https://issues.apache.org/jira/browse/CALCITE-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145080#comment-16145080 ] Christian Beikov commented on CALCITE-1967: --- Separate PR or can I include these changes into the existing one? Where should I put the test data file to? Or should I setup custom test data within the test that uses the same schema? > Add support for elastic search 5 > > > Key: CALCITE-1967 > URL: https://issues.apache.org/jira/browse/CALCITE-1967 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The elastic search adapter seems to target versions before 5.x but it would > be nice to also have an adapter for ES5+ so I created one that is quite > similar to the existing adapter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145024#comment-16145024
]
Christian Beikov edited comment on CALCITE-1965 at 8/29/17 10:07 AM:
-
In order to support right joins I'd need this normalization step with an
addition that right joins are rewritten to left joins. I guess the check of the
structure can be easily done during OEQ determination.
I haven't thought about full outer joins, derived tables/subqueries or union
rewrites yet, but I tried to come up with some examples for left and right
outer joins.
Essentially OEQ must be an ordered set of ordered sets. The ordering is given
by doing a depth-first search in the join tree, collecting the predicates of
joins when going upwards. You will understand what I mean when you see examples.
During matching, we would do a prefix match of the query's OEQ against the
view's OEQ. Since left joins are row preserving, a view can join more than a
query and still being applicable. The matching would work like this:
1. Assert that the query's EQ contains all of the view's EQ and let qEQ1 = qEQ
- vEQ
2. Try to find a suffix match for qOEQ in vOEQ and assert qEQ1 contains the
prefix pOEQ. let vOEQ1 = vOEQ - pOEQ
3. Check that qOEQ is contained in the right order in vOEQ1
I think this should handle the equivalences for joins.
As for your examples, this approach wouldn't match:
{noformat}
View1: select a.id, b.id, c.id from a left join b on a.a = b.a left join c on
b.a = c.c;
EQ={} OEQ={a.a, b.a, c.c}
Query1: select a.id, b.id, c.id from a left join c on a.a = c.c left join b on
a.a = b.a;
EQ={} OEQ={a.a, c.c, b.a}
=> doesn't match order
View2: select a.id, b.id, c.id from a left join b on a.a = b.a right join c on
b.a = c.c;
View2': select a.id, b.id, c.id from c left join (a left join b on a.a = b.a)
on c.c = b.a;
EQ={} OEQ={a.a, b.a, c.c}
Query2: select a.id, b.id, c.id from a right join c on a.a = c.c left join b on
a.a = b.a;
Query2': select a.id, b.id, c.id from c left join a on c.c = a.a left join b on
a.a = b.a;
EQ={} OEQ={c.c, a.a, b.a}
=> doesn't match order
{noformat}
The examples I did so far were to illustrate how mixing left and right joins
plays out
*View 1*
{noformat}
select *
from a
left join b on a.a = b.a
left join c on b.b = c.b
ljoin(a.a = b.a)
/\
table(a) ljoin(b.b = c.b)
/ \
table(b)table(c)
EQ={}
OEQ={a.a, b.a}{b.b, c.b}
{noformat}
_Query 1.1_
{noformat}
select *
from a
join b on a.a = b.a
left join c on b.b = c.b
ijoin(a.a = b.a)
/\
table(a) ljoin(b.b = c.b)
/ \
table(b)table(c)
EQ={a.a, b.a}
OEQ={b.b, c.b}
=> since the query's EQ {a.a, b.a} matched as prefix in the view's OEQ, we need
a is not null predicate
select *
from v1
where b.a is not null
{noformat}
_Query 1.2_
{noformat}
select *
from a
join b on a.a = b.a
join c on b.b = c.b
ijoin(a.a = b.a)
/\
table(a) ijoin(b.b = c.b)
/ \
table(b)table(c)
EQ={a.a, b.a}{b.b, c.b}
OEQ={}
=> query's EQ matched as prefix in the view's OEQ
select *
from v1
where b.a is not null and c.b is not null
{noformat}
_Query 1.3_
{noformat}
select *
from a
join b on a.a = b.a
ijoin(a.a = b.a)
/\
table(a) table(b)
EQ={a.a, b.a}
OEQ={}
=> query's EQ matched as prefix in the view's OEQ => pOEQ = vEQ
=> query's OEQ trivially also matches the view's OEQ - pOEQ
select *
from v1
where b.a is not null
{noformat}
_Query 1.4_
{noformat}
select *
from a
join d on a.a = d.a
join b on a.a = b.a
left join c on b.b = c.b
ijoin(a.a = b.a)
/\
ijoin(a.a = d.a) ljoin(b.b = c.b)
/ \ / \
table(a) table(d)table(b)table(c)
EQ={a.a, b.a, d.a}
OEQ={b.b, c.b}
=> since the query's EQ {a.a, b.a} matched as prefix in the view's OEQ, we need
a is not null predicate
=> we also need a compensation join since d.a didn't match
select *
from v1
join d on a.a = d.a
where b.a is not null
{noformat}
*View 2*
{noformat}
select *
from a
left join b on a.a = b.a
right join c on b.b = c.b
=> needs join order rewriting
parenthesis are just to clarify precedence
select *
from c
left join (
a left join b on a.a = b.a
) on b.b = c.b
ljoin(c.b = b.b)
/\
table(c) ljoin(a.a = b.a)
/ \
table(a)table(b)
EQ={}
OEQ={c.b, b.b}{a.a, b.a}
=> since join order changed, the sets changed order too, just like elements
=> the first element is always the left join source
{noformat}
_Query 2.1_
{noformat}
select *
from c
left join b on c.b = b.b
left join a on b.a = a.a
ljoin(c.b = b.b)
/\
table(c) ljoin(b.a = a.a)
/ \
table(b)table(a)
EQ={}
OEQ={c.b,
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145024#comment-16145024
]
Christian Beikov commented on CALCITE-1965:
---
In order to support right joins I'd need this normalization step with an
addition that right joins are rewritten to left joins. I guess the check of the
structure can be easily done during OEQ determination.
I haven't thought about full outer joins, derived tables/subqueries or union
rewrites yet, but I tried to come up with some examples for left and right
outer joins.
Essentially OEQ must be an ordered set of ordered sets. The ordering is given
by doing a depth-first search in the join tree, collecting the predicates of
joins when going upwards. You will understand what I mean when you see examples.
During matching, we would do a prefix match of the query's OEQ against the
view's OEQ. Since left joins are row preserving, a view can join more than a
query and still being applicable. The matching would work like this:
1. Assert that the query's EQ contains all of the view's EQ and let qEQ1 = qEQ
- vEQ
2. Try to find a suffix match for qOEQ in vOEQ and assert qEQ1 contains the
prefix pOEQ. let vOEQ1 = vOEQ - pOEQ
3. Check that qOEQ is contained in the right order in vOEQ1
I think this should handle the equivalences for joins.
As for your examples, this approach wouldn't match:
{noformat}
View1: select a.id, b.id, c.id from a left join b on a.a = b.a left join c on
b.a = c.c;
EQ={} OEQ={a.a, b.a, c.c}
Query1: select a.id, b.id, c.id from a left join c on a.a = c.c left join b on
a.a = b.a;
EQ={} OEQ={a.a, c.c, b.a}
=> doesn't match order
View2: select a.id, b.id, c.id from a left join b on a.a = b.a right join c on
b.a = c.c;
View2': select a.id, b.id, c.id from c left join (a left join b on a.a = b.a)
on c.c = b.a;
EQ={} OEQ={a.a, b.a, c.c}
Query2: select a.id, b.id, c.id from a right join c on a.a = c.c left join b on
a.a = b.a;
Query2': select a.id, b.id, c.id from c left join a on c.c = a.a left join b on
a.a = b.a;
EQ={} OEQ={c.c, a.a, b.a}
=> doesn't match order
{noformat}
The examples I did so far were to illustrate how mixing left and right joins
plays out
*View 1*
{noformat}
select *
from a
left join b on a.a = b.a
left join c on b.b = c.b
ljoin(a.a = b.a)
/\
table(a) ljoin(b.b = c.b)
/ \
table(b)table(c)
EQ={}
OEQ={a.a, b.a}{b.b, c.b}
{noformat}
_Query 1.1_
{noformat}
select *
from a
join b on a.a = b.a
left join c on b.b = c.b
ijoin(a.a = b.a)
/\
table(a) ljoin(b.b = c.b)
/ \
table(b)table(c)
EQ={a.a, b.a}
OEQ={b.b, c.b}
=> since the query's EQ {a.a, b.a} matched as prefix in the view's OEQ, we need
a is not null predicate
select *
from v1
where b.a is not null
{noformat}
_Query 1.2_
{noformat}
select *
from a
join b on a.a = b.a
join c on b.b = c.b
ijoin(a.a = b.a)
/\
table(a) ijoin(b.b = c.b)
/ \
table(b)table(c)
EQ={a.a, b.a}{b.b, c.b}
OEQ={}
=> query's EQ matched as prefix in the view's OEQ
select *
from v1
where b.a is not null and c.b is not null
{noformat}
_Query 1.3_
{noformat}
select *
from a
join b on a.a = b.a
ijoin(a.a = b.a)
/\
table(a) table(b)
EQ={a.a, b.a}
OEQ={}
=> query's EQ matched as prefix in the view's OEQ => pOEQ = vEQ
=> query's OEQ trivially also matches the view's OEQ - pOEQ
select *
from v1
where b.a is not null
{noformat}
*View 2*
{noformat}
select *
from a
left join b on a.a = b.a
right join c on b.b = c.b
=> needs join order rewriting
parenthesis are just to clarify precedence
select *
from c
left join (
a left join b on a.a = b.a
) on b.b = c.b
ljoin(c.b = b.b)
/\
table(c) ljoin(a.a = b.a)
/ \
table(a)table(b)
EQ={}
OEQ={c.b, b.b}{a.a, b.a}
=> since join order changed, the sets changed order too, just like elements
=> the first element is always the left join source
{noformat}
_Query 2.1_
{noformat}
select *
from c
left join b on c.b = b.b
left join a on b.a = a.a
ljoin(c.b = b.b)
/\
table(c) ljoin(b.a = a.a)
/ \
table(b)table(a)
EQ={}
OEQ={c.b, b.b}{b.a, a.a}
=> no rewrite since the order of the OEQ is different i.e. {a.a, b.a}!={b.a,
a.a}
{noformat}
_Query 2.2_
{noformat}
select *
from c
left join b on c.b = b.b
ljoin(c.b = b.b)
/\
table(c) table(b)
EQ={}
OEQ={c.b, b.b}
=> rewrite since qOEQ is a prefix of vOEQ
select *
from v2
{noformat}
*View 3*
{noformat}
select *
from a
left join b on a.a = b.a
right join c on b.b = c.b
left join d on c.c = d.c
=> needs join order rewriting
parenthesis are just to clarify precedence
select *
from c
left join (
a left join b on a.a = b.a
) on b.b = c.b
left join d on
[jira] [Commented] (CALCITE-1966) Support for normal views to act as materialization table
[ https://issues.apache.org/jira/browse/CALCITE-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143086#comment-16143086 ] Christian Beikov commented on CALCITE-1966: --- Hey [~maryannxue] I updated the PR with the comments and also added a test. Hope it's good to go like that? > Support for normal views to act as materialization table > > > Key: CALCITE-1966 > URL: https://issues.apache.org/jira/browse/CALCITE-1966 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Christian Beikov >Assignee: Julian Hyde > > Currently, it seems a materialized view can only have it's materialization in > a real table, although it might be nice to be able to use a view instead. > Since external data stores like e.g. elasticsearch aren't might have > different data representations, the mismatch has to be overcome by doing > casts in a view. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1966) Support for normal views to act as materialization table
[ https://issues.apache.org/jira/browse/CALCITE-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142959#comment-16142959 ] Christian Beikov commented on CALCITE-1966: --- [~maryannxue] I pushed the requested changes. Tests are coming tomorrow. Is there anything else I should take care of? > Support for normal views to act as materialization table > > > Key: CALCITE-1966 > URL: https://issues.apache.org/jira/browse/CALCITE-1966 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Christian Beikov >Assignee: Julian Hyde > > Currently, it seems a materialized view can only have it's materialization in > a real table, although it might be nice to be able to use a view instead. > Since external data stores like e.g. elasticsearch aren't might have > different data representations, the mismatch has to be overcome by doing > casts in a view. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1967) Add support for elastic search 5
[ https://issues.apache.org/jira/browse/CALCITE-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141332#comment-16141332 ] Christian Beikov commented on CALCITE-1967: --- I updated the PR, but there is one other thing I'd like to do if you agree it is a good idea. I'd like to use ES embedded for the tests to make them independent of an external DB. That will allow to run the tests on a CI server more easily and also allow me to do easier testing. Would that be ok or do you have a specific reason to expect an already running ES server? As an additional bonus, this will allow for testing insert/update/delete operation implementations which is something I'd like to look into in the near future. > Add support for elastic search 5 > > > Key: CALCITE-1967 > URL: https://issues.apache.org/jira/browse/CALCITE-1967 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The elastic search adapter seems to target versions before 5.x but it would > be nice to also have an adapter for ES5+ so I created one that is quite > similar to the existing adapter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140715#comment-16140715
]
Christian Beikov commented on CALCITE-1965:
---
Gonna write a unit test for that tomorrow for the PR and ping you when it's
done.
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140624#comment-16140624
]
Christian Beikov commented on CALCITE-1965:
---
The impl in {{getAllPredicates}} depends on the change in the lineage code, so
I am not sure I can apply that independently.
Maybe you can explain the expected lineage concept to me so that I understand
why this might be problematic? I must know the _lineage_ in the outer join case
of {{getAllPredicates}} since I have to add {{ISNULL}} predicates for all
expressions. I guess if I don't add these predicates, the joins would really be
treated like a inner join and rewrites would happen where they shouldn't.
I thought really hard and even enumerated the scenarios for 4 relations with
alternating join types and a materialization covering 2 relations, but couldn't
find a correctnes violation of my approach, of course assuming I have the
correct understand of the rewriting that is happening. Maybe you can enlighten
me? :)
As always, time problems. Know them very good :D
Thanks in advance for all the assistance! Looking forward to see the outer join
support for MVs in master.
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1966) Support for normal views to act as materialization table
[ https://issues.apache.org/jira/browse/CALCITE-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140500#comment-16140500 ] Christian Beikov commented on CALCITE-1966: --- Great, as soon as you give me the ok, I'll add some tests and notify you when I'm done. > Support for normal views to act as materialization table > > > Key: CALCITE-1966 > URL: https://issues.apache.org/jira/browse/CALCITE-1966 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Christian Beikov >Assignee: Julian Hyde > > Currently, it seems a materialized view can only have it's materialization in > a real table, although it might be nice to be able to use a view instead. > Since external data stores like e.g. elasticsearch aren't might have > different data representations, the mismatch has to be overcome by doing > casts in a view. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1967) Add support for elastic search 5
[ https://issues.apache.org/jira/browse/CALCITE-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140492#comment-16140492 ] Christian Beikov commented on CALCITE-1967: --- Yeah I was thinking about depending on the original adapter too or maybe even shading it, but at first I wasn't sure how much I had to rewrite and what you would expect of a new adapter. I don't know if subclassing will work so easy. I can try, but I am afraid that there are some JVMs like the IBM J9 that are pretty strict about binary compatibilty i.e. errors because of non-existing statically referenced methods will be thrown during class loading vs. during first use(Oracle JVMs). And obviously, there are incompatibilities between ES2 and ES5. How about a common base module *calcite-elasticsearch-base* that contains the commonalities with abstract methods to override and two submodules for ES2 and ES5 that each shade the base or just normally depend on it. > Add support for elastic search 5 > > > Key: CALCITE-1967 > URL: https://issues.apache.org/jira/browse/CALCITE-1967 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The elastic search adapter seems to target versions before 5.x but it would > be nice to also have an adapter for ES5+ so I created one that is quite > similar to the existing adapter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1967) Add support for elastic search 5
[ https://issues.apache.org/jira/browse/CALCITE-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140087#comment-16140087 ] Christian Beikov commented on CALCITE-1967: --- See the PR https://github.com/apache/calcite/pull/528 > Add support for elastic search 5 > > > Key: CALCITE-1967 > URL: https://issues.apache.org/jira/browse/CALCITE-1967 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The elastic search adapter seems to target versions before 5.x but it would > be nice to also have an adapter for ES5+ so I created one that is quite > similar to the existing adapter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1966) Support for normal views to act as materialization table
[ https://issues.apache.org/jira/browse/CALCITE-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140077#comment-16140077 ] Christian Beikov commented on CALCITE-1966: --- See the PR: https://github.com/apache/calcite/pull/527 > Support for normal views to act as materialization table > > > Key: CALCITE-1966 > URL: https://issues.apache.org/jira/browse/CALCITE-1966 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Christian Beikov >Assignee: Julian Hyde > > Currently, it seems a materialized view can only have it's materialization in > a real table, although it might be nice to be able to use a view instead. > Since external data stores like e.g. elasticsearch aren't might have > different data representations, the mismatch has to be overcome by doing > casts in a view. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-1966) Support for normal views to act as materialization table
Christian Beikov created CALCITE-1966: - Summary: Support for normal views to act as materialization table Key: CALCITE-1966 URL: https://issues.apache.org/jira/browse/CALCITE-1966 Project: Calcite Issue Type: Improvement Components: core Reporter: Christian Beikov Assignee: Julian Hyde Currently, it seems a materialized view can only have it's materialization in a real table, although it might be nice to be able to use a view instead. Since external data stores like e.g. elasticsearch aren't might have different data representations, the mismatch has to be overcome by doing casts in a view. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-1965) Support outer joins for materialized views
[
https://issues.apache.org/jira/browse/CALCITE-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140060#comment-16140060
]
Christian Beikov commented on CALCITE-1965:
---
See the PR here: https://github.com/apache/calcite/pull/526
> Support outer joins for materialized views
> --
>
> Key: CALCITE-1965
> URL: https://issues.apache.org/jira/browse/CALCITE-1965
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> Currently, only inner joins are supported for materialized view
> substitutions. The support for outer joins involves creating new pulled up
> predicates in case of outer joins that represent semantics of the join. For a
> join predicate like "a.id = b.id" the inner join just pulls up that
> predicate. When having a left join like e.g. {{select * from a left join b on
> a.id = b.id}}, the actual pulled up predicate would be {{OR(=(a.id,
> b.id),ISNULL(b.id))}}. For a right join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id))}} and for a full outer join it would be {{OR(=(a.id,
> b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CALCITE-1965) Support outer joins for materialized views
Christian Beikov created CALCITE-1965:
-
Summary: Support outer joins for materialized views
Key: CALCITE-1965
URL: https://issues.apache.org/jira/browse/CALCITE-1965
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Christian Beikov
Assignee: Julian Hyde
Currently, only inner joins are supported for materialized view substitutions.
The support for outer joins involves creating new pulled up predicates in case
of outer joins that represent semantics of the join. For a join predicate like
"a.id = b.id" the inner join just pulls up that predicate. When having a left
join like e.g. {{select * from a left join b on a.id = b.id}}, the actual
pulled up predicate would be {{OR(=(a.id, b.id),ISNULL(b.id))}}. For a right
join it would be {{OR(=(a.id, b.id),ISNULL(a.id))}} and for a full outer join
it would be {{OR(=(a.id, b.id),ISNULL(a.id),ISNULL(b.id))}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[
https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139487#comment-16139487
]
Christian Beikov commented on CALCITE-1940:
---
Seems I only use {{JdbcTable.jdbcTableName}} to get a proper toString
representation for the sequence.
I wasn't sure how to get the dialect during unparsing and since the writer
already had a getter, I thought it might be appropriate to just get the proper
one in. I suppose the writer should be extended to be able to return a dialect
for a schema name?
I'll adapt the PR to the requested changes and let you know when it's done.
> Implement dialect specific support for sequences
>
>
> Key: CALCITE-1940
> URL: https://issues.apache.org/jira/browse/CALCITE-1940
> Project: Calcite
> Issue Type: Improvement
>Reporter: Christian Beikov
>Assignee: Julian Hyde
>
> The Calcite parser and validator already supports sequences but the push down
> to the JDBC level is currently limited. SInce sequences are not supported on
> all DBMS every sql dialect should have the possibility to render it's own way
> of extracting sequence information or incrementing the value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1940) Implement dialect specific support for sequences
[ https://issues.apache.org/jira/browse/CALCITE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134428#comment-16134428 ] Christian Beikov commented on CALCITE-1940: --- See the following PR for a fix: https://github.com/apache/calcite/pull/514 > Implement dialect specific support for sequences > > > Key: CALCITE-1940 > URL: https://issues.apache.org/jira/browse/CALCITE-1940 > Project: Calcite > Issue Type: Improvement >Reporter: Christian Beikov >Assignee: Julian Hyde > > The Calcite parser and validator already supports sequences but the push down > to the JDBC level is currently limited. SInce sequences are not supported on > all DBMS every sql dialect should have the possibility to render it's own way > of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-1940) Implement dialect specific support for sequences
Christian Beikov created CALCITE-1940: - Summary: Implement dialect specific support for sequences Key: CALCITE-1940 URL: https://issues.apache.org/jira/browse/CALCITE-1940 Project: Calcite Issue Type: Improvement Reporter: Christian Beikov Assignee: Julian Hyde The Calcite parser and validator already supports sequences but the push down to the JDBC level is currently limited. SInce sequences are not supported on all DBMS every sql dialect should have the possibility to render it's own way of extracting sequence information or incrementing the value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
