[jira] [Commented] (CALCITE-4335) Aggregate functions for BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212762#comment-17212762
]
Julian Hyde commented on CALCITE-4335:
--
For {{LOGICAL_AND}}, {{LOGICAL_OR}} see CALCITE-2935.
> Aggregate functions for BigQuery
>
>
> Key: CALCITE-4335
> URL: https://issues.apache.org/jira/browse/CALCITE-4335
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Add aggregate functions for BigQuery
> ([spec|https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions]):
> {{ARRAY_AGG}}, {{ARRAY_CONCAT_AGG}}, {{COUNTIF}}, {{LOGICAL_AND}},
> {{LOGICAL_OR}}, {{STRING_AGG}}.
> Calcite already has {{ANY_VALUE}}, {{BIT_AND}}, {{BIT_OR}}, {{BIT_XOR}},
> {{MAX}}, {{MIN}}, {{SUM}} aggregate functions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4335) Aggregate functions for BigQuery
Julian Hyde created CALCITE-4335:
Summary: Aggregate functions for BigQuery
Key: CALCITE-4335
URL: https://issues.apache.org/jira/browse/CALCITE-4335
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add aggregate functions for BigQuery
([spec|https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions]):
{{ARRAY_AGG}}, {{ARRAY_CONCAT_AGG}}, {{COUNTIF}}, {{LOGICAL_AND}},
{{LOGICAL_OR}}, {{STRING_AGG}}.
Calcite already has {{ANY_VALUE}}, {{BIT_AND}}, {{BIT_OR}}, {{BIT_XOR}},
{{MAX}}, {{MIN}}, {{SUM}} aggregate functions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4317) Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the fields
[
https://issues.apache.org/jira/browse/CALCITE-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212686#comment-17212686
]
Julian Hyde edited comment on CALCITE-4317 at 10/12/20, 9:59 PM:
-
I think it's a valid optimization, and it would be useful in this case.
Still, it's more important that {{RelBuilder.aggregate}} does not create
{{RelNode}}s ({{Aggregate}} or otherwise) that have zero fields. Let's fix that
first.
In CALCITE-4334, I discussed how {{RelBuilder.aggregate(groupKey(),
aggregateCall(TRUE_AGG))}} could create {{VALUES TRUE}}.
was (Author: julianhyde):
I think it's a valid optimization, and it would be useful in this case.
Still, it's more important that {{RelBuilder.aggregate}} does not create
{{RelNode}}s ({{Aggregate}} or otherwise) that have zero fields. Let's fix that
first.
> Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the
> fields
>
>
> Key: CALCITE-4317
> URL: https://issues.apache.org/jira/browse/CALCITE-4317
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Rafay A
>Priority: Major
>
> Consider this query:
> {code:java}
> select o_orderkey from
> (SELECT count(*) cnt_star, count(l_orderkey) cnt_ok FROM lineitem)
> cross join orders
> limit 10
> {code}
> which generates this plan:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$2])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}], cnt_star=[COUNT()], cnt_ok=[COUNT($0)])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`, `l_partkey`,
> `l_suppkey`, `l_linenumber`, `l_quantity`, `l_extendedprice`, `l_discount`,
> `l_tax`, `l_returnflag`, `l_linestatus`, `l_shipdate`, `l_commitdate`,
> `l_receiptdate`, `l_shipinstruct`, `l_shipmode`, `l_comment`])
> ScanCrel(table=[orders], columns=[`o_orderkey`, `o_custkey`,
> `o_orderstatus`, `o_totalprice`, `o_orderdate`, `o_orderpriority`, `o_clerk`,
> `o_shippriority`, `o_comment`])
> {code}
> and after we apply RelFieldTrimmer:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$0])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`])
> ScanCrel(table=[orders], columns=[`o_orderkey`])
> {code}
>
> We see that the LogicalAggregate has no groups/functions after applying the
> trimmer. When we apply ProjectJoinTransposeRule or LoptOptimizeJoinRule after
> trimming, we are seeing these exceptions:
> For LoptOptimizeJoinRule:
> {code:java}
> (java.lang.ArrayIndexOutOfBoundsException) 0
> com.google.common.collect.RegularImmutableList.get():75
> org.apache.calcite.rel.metadata.RelMdColumnOrigins.getColumnOrigins():77
> sun.reflect.NativeMethodAccessorImpl.invoke0():-2
> sun.reflect.NativeMethodAccessorImpl.invoke():62
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():178
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.RelMetadataQuery$MetadataInvocationHandler.invoke():110
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> org.apache.calcite.rel.metadata.RelMetadataQuery.getColumnOrigins():293
> org.apache.calcite.rel.metadata.RelMetadataQuery.getTableOrigin():348
>
[jira] [Created] (CALCITE-4334) TRUE_AGG, an aggregate function that always returns TRUE
Julian Hyde created CALCITE-4334:
Summary: TRUE_AGG, an aggregate function that always returns TRUE
Key: CALCITE-4334
URL: https://issues.apache.org/jira/browse/CALCITE-4334
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
It would be useful to have an (internal) aggregate function that has no
arguments and returns a constant value, regardless of how many rows are in the
group. We suggest {{TRUE_AGG}}, which always returns the {{BOOLEAN}} value
{{TRUE}}.
If there are reasons why it would be better to return other constants (such as
0, 1, FALSE or the empty string) we will consider them.
For example, when rewriting sub-queries (see SubQueryRemoveRule) we introduce
add "true as indicator" to the SELECT clause of sub-queries. It can be used to
detect rows generated by an outer join. If it is an aggregate query, we would
have to write "min(true) as indicator", which necessitates an extra {{Project}}
below the {{Aggregate}} to provide the "true" value.
Another example. We would like to make {{RelBuilder.aggregate(groupKey())}}
throw when given an empty group key and no aggregate calls. (Because it would
create an {{Aggregate}} that has zero fields, and that is problematic elsewhere
in Calcite.) But we would also like a pattern to generate a constant single-row
relational expression. So, {{RelBulder.aggregate(groupKey(),
aggregateCall(TRUE_AGG))}} should generate {{VALUES TRUE}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4317) Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the fields
[
https://issues.apache.org/jira/browse/CALCITE-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212686#comment-17212686
]
Julian Hyde commented on CALCITE-4317:
--
I think it's a valid optimization, and it would be useful in this case.
Still, it's more important that {{RelBuilder.aggregate}} does not create
{{RelNode}}s ({{Aggregate}} or otherwise) that have zero fields. Let's fix that
first.
> Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the
> fields
>
>
> Key: CALCITE-4317
> URL: https://issues.apache.org/jira/browse/CALCITE-4317
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Rafay A
>Priority: Major
>
> Consider this query:
> {code:java}
> select o_orderkey from
> (SELECT count(*) cnt_star, count(l_orderkey) cnt_ok FROM lineitem)
> cross join orders
> limit 10
> {code}
> which generates this plan:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$2])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}], cnt_star=[COUNT()], cnt_ok=[COUNT($0)])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`, `l_partkey`,
> `l_suppkey`, `l_linenumber`, `l_quantity`, `l_extendedprice`, `l_discount`,
> `l_tax`, `l_returnflag`, `l_linestatus`, `l_shipdate`, `l_commitdate`,
> `l_receiptdate`, `l_shipinstruct`, `l_shipmode`, `l_comment`])
> ScanCrel(table=[orders], columns=[`o_orderkey`, `o_custkey`,
> `o_orderstatus`, `o_totalprice`, `o_orderdate`, `o_orderpriority`, `o_clerk`,
> `o_shippriority`, `o_comment`])
> {code}
> and after we apply RelFieldTrimmer:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$0])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`])
> ScanCrel(table=[orders], columns=[`o_orderkey`])
> {code}
>
> We see that the LogicalAggregate has no groups/functions after applying the
> trimmer. When we apply ProjectJoinTransposeRule or LoptOptimizeJoinRule after
> trimming, we are seeing these exceptions:
> For LoptOptimizeJoinRule:
> {code:java}
> (java.lang.ArrayIndexOutOfBoundsException) 0
> com.google.common.collect.RegularImmutableList.get():75
> org.apache.calcite.rel.metadata.RelMdColumnOrigins.getColumnOrigins():77
> sun.reflect.NativeMethodAccessorImpl.invoke0():-2
> sun.reflect.NativeMethodAccessorImpl.invoke():62
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():178
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.RelMetadataQuery$MetadataInvocationHandler.invoke():110
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> org.apache.calcite.rel.metadata.RelMetadataQuery.getColumnOrigins():293
> org.apache.calcite.rel.metadata.RelMetadataQuery.getTableOrigin():348
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.getSimpleFactors():377
>
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.findRemovableSelfJoins():297
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.onMatch():126
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule():317
> org.apache.calcite.plan.hep.HepPlanner.applyRule():556
> org.apache.calcite.plan.hep.HepPlanner.applyRules():415
>
[jira] [Commented] (CALCITE-4317) Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the fields
[
https://issues.apache.org/jira/browse/CALCITE-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212671#comment-17212671
]
Rafay A commented on CALCITE-4317:
--
[~julianhyde], that could be one possible solution. But since a zero field
Aggregate should just return one row, i think it would be a good optimization
to truncate the entire subtree. Any thoughts?
> Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the
> fields
>
>
> Key: CALCITE-4317
> URL: https://issues.apache.org/jira/browse/CALCITE-4317
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Rafay A
>Priority: Major
>
> Consider this query:
> {code:java}
> select o_orderkey from
> (SELECT count(*) cnt_star, count(l_orderkey) cnt_ok FROM lineitem)
> cross join orders
> limit 10
> {code}
> which generates this plan:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$2])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}], cnt_star=[COUNT()], cnt_ok=[COUNT($0)])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`, `l_partkey`,
> `l_suppkey`, `l_linenumber`, `l_quantity`, `l_extendedprice`, `l_discount`,
> `l_tax`, `l_returnflag`, `l_linestatus`, `l_shipdate`, `l_commitdate`,
> `l_receiptdate`, `l_shipinstruct`, `l_shipmode`, `l_comment`])
> ScanCrel(table=[orders], columns=[`o_orderkey`, `o_custkey`,
> `o_orderstatus`, `o_totalprice`, `o_orderdate`, `o_orderpriority`, `o_clerk`,
> `o_shippriority`, `o_comment`])
> {code}
> and after we apply RelFieldTrimmer:
> {code:java}
> LogicalSort(fetch=[10])
> LogicalProject(o_orderkey=[$0])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalAggregate(group=[{}])
> LogicalProject(l_orderkey=[$0])
> ScanCrel(table=[lineitem], columns=[`l_orderkey`])
> ScanCrel(table=[orders], columns=[`o_orderkey`])
> {code}
>
> We see that the LogicalAggregate has no groups/functions after applying the
> trimmer. When we apply ProjectJoinTransposeRule or LoptOptimizeJoinRule after
> trimming, we are seeing these exceptions:
> For LoptOptimizeJoinRule:
> {code:java}
> (java.lang.ArrayIndexOutOfBoundsException) 0
> com.google.common.collect.RegularImmutableList.get():75
> org.apache.calcite.rel.metadata.RelMdColumnOrigins.getColumnOrigins():77
> sun.reflect.NativeMethodAccessorImpl.invoke0():-2
> sun.reflect.NativeMethodAccessorImpl.invoke():62
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():178
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> sun.reflect.GeneratedMethodAccessor144.invoke():-1
> sun.reflect.DelegatingMethodAccessorImpl.invoke():43
> java.lang.reflect.Method.invoke():498
>
> org.apache.calcite.rel.metadata.RelMetadataQuery$MetadataInvocationHandler.invoke():110
> com.sun.proxy.$Proxy175.getColumnOrigins():-1
> org.apache.calcite.rel.metadata.RelMetadataQuery.getColumnOrigins():293
> org.apache.calcite.rel.metadata.RelMetadataQuery.getTableOrigin():348
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.getSimpleFactors():377
>
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.findRemovableSelfJoins():297
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.onMatch():126
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule():317
> org.apache.calcite.plan.hep.HepPlanner.applyRule():556
> org.apache.calcite.plan.hep.HepPlanner.applyRules():415
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction():280
>
[jira] [Commented] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212599#comment-17212599
]
Julian Hyde commented on CALCITE-4332:
--
Sounds fine. The fix should include a test that a given action produces a given
error message.
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-4332:
-
Summary: Improve error when planning rule produces a relational expression
with wrong row type (was: Improve error when planning rule produces a relation
with a wrong rowtype)
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4034) Implement a MySQL InnoDB adapter
[
https://issues.apache.org/jira/browse/CALCITE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212340#comment-17212340
]
neoremind commented on CALCITE-4034:
Thanks for refining the doc and many code, really appreciate your help and
admire your professionalism.
I feel very happy to contribute, I will continue to be responsible for any
questions in the future related to this adapter.
> Implement a MySQL InnoDB adapter
>
>
> Key: CALCITE-4034
> URL: https://issues.apache.org/jira/browse/CALCITE-4034
> Project: Calcite
> Issue Type: New Feature
> Components: innodb-adapter
>Affects Versions: 1.23.0
>Reporter: neoremind
>Assignee: Julian Hyde
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Calcite’s InnoDB adapter allows you to query the data based on InnoDB data
> files directy, data files are also known as .ibd files, this adapter is
> different from JDBC adapter which requires a MySQL server to serve response.
> With .ibd files and the corresponding DDLs, InnoDB adapter is able to work
> like a simple "MySQL server", it accepts SQL query and attempts to compile
> the query based on InnoDB file accessing APIs provided by
> [innodb-java-reader|[https://github.com/alibaba/innodb-java-reader]] (I am
> the author of the library, which is well-tested and production-ready), it
> will exploit projecting, filtering and sorting directly in InnoDB data file
> where possible. What’s more, with DDLs, the adapter is "index aware", it
> leverages rules to choose the right index to scan, for example, using primary
> key or secondary keys to look up data, then it tries to push down some
> conditions into storage engine. Also, the adapter leaves option to provide
> hint as well, so that user can indicate the optimizer to force use one
> specific index.
> The InnoDB adapter can,
> 1) Full scan a table.
> 2) Scan by index such as primary key or secondary key, single column or
> composite keys.
> 3) Push down filtering condition to storage engine, support both point query
> and range query.
> 4) Push down projection.
> 5) Push down ordering if it matches the natural collation of an index.
> 6) Support almost all the commonly used data types.
> 7) Work as a simple MySQL server being able to read .ibd data files directly
> through file system, this can be a tool to query or dump table data by
> offloading from MySQL process under some conditions.
> Before I describe the adapter and its design, I was wondering if it is
> appropriate to deliver the work by contributing the codebase back to Calcite,
> or if it would be better to stay in another project, probably being referred
> from Calcite adapter link. Looking forward to Calcite developer's advice.
>
> Below is the first version of the usage.
> --
> [MySQL]([https://www.mysql.com/]) is the most popular Open Source SQL
> database management system, is developed, distributed, and supported by
> Oracle Corporation. InnoDB is a general-purpose storage engine that balances
> high reliability and high performance in MySQL, since 5.6 InnoDB has become
> the default MySQL storage engine.
> Calcite’s InnoDB adapter allows you to query the data based on InnoDB data
> files directy, data files are also known as .ibd files. It leverages
> [innodb-java-reader]([https://github.com/alibaba/innodb-java-reader]). This
> adapter is different from JDBC adapter which maps a schema in a JDBC data
> source and requires a MySQL server to serve response. With .ibd files and the
> corresponding DDLs, InnoDB adapter is able to work like a simple "MySQL
> server", it accepts SQL query and attempts to compile the query based on
> InnoDB file accessing APIs provided by innodb-java-reader, it will exploit
> projecting, filtering and sorting directly in InnoDB data file where possible.
> What’s more, with DDLs, the adapter is "index aware", it leverages rules to
> choose the right index to scan, for example, using primary key or secondary
> keys to look up data, then it tries to push down some conditions into storage
> engine. Also, the adapter leaves option to provide hint as well, so that user
> can indicate the optimizer to force use one specific index.
> A basic example of a model file is given below, this schema reads from a
> MySQL "scott" database:
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "scott",
> "schemas": [
> {
> "name": "scott",
> "type": "custom",
> "factory": "org.apache.calcite.adapter.innodb.InnodbSchemaFactory",
> "operand": {
> "sqlFilePath": [ "/path/scott.sql" ],
> "ibdDataFileBasePath": "/usr/local/mysql/data/scott"
> }
> }
> ]
> }
> {code}
>
[jira] [Updated] (CALCITE-4325) RexSimplify/Sarg incorrectly simplifies complex expressions with NULL
[
https://issues.apache.org/jira/browse/CALCITE-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Lozynskyi updated CALCITE-4325:
Attachment: (was: rex-simplify-issue-tests.patch)
> RexSimplify/Sarg incorrectly simplifies complex expressions with NULL
> -
>
> Key: CALCITE-4325
> URL: https://issues.apache.org/jira/browse/CALCITE-4325
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
> Environment: All environments
>Reporter: Igor Lozynskyi
>Assignee: Julian Hyde
>Priority: Major
> Attachments: rex-simplify-issue-tests-2.patch
>
>
> In Calcite 1.26, the Rex expressions like follows:
> {code:java}
> (deptno = 20 OR deptno IS NULL) AND deptno = 10{code}
> Are simplified to:
> {code:java}
> deptno IS NULL{code}
> Instead of:
> {code:java}
> FALSE{code}
> Similarly, the following expression:
> {code:java}
> (deptno <> 20 OR deptno IS NULL) AND deptno = 10{code}
> Is simplified to:
> {code:java}
> deptno = 10 OR deptno IS NULL{code}
> Instead of:
> {code:java}
> deptno = 10{code}
> A diff with related unit tests (for RelBuilderTest.java) is in the attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4325) RexSimplify/Sarg incorrectly simplifies complex expressions with NULL
[
https://issues.apache.org/jira/browse/CALCITE-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Lozynskyi updated CALCITE-4325:
Attachment: rex-simplify-issue-tests-2.patch
> RexSimplify/Sarg incorrectly simplifies complex expressions with NULL
> -
>
> Key: CALCITE-4325
> URL: https://issues.apache.org/jira/browse/CALCITE-4325
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
> Environment: All environments
>Reporter: Igor Lozynskyi
>Assignee: Julian Hyde
>Priority: Major
> Attachments: rex-simplify-issue-tests-2.patch,
> rex-simplify-issue-tests.patch
>
>
> In Calcite 1.26, the Rex expressions like follows:
> {code:java}
> (deptno = 20 OR deptno IS NULL) AND deptno = 10{code}
> Are simplified to:
> {code:java}
> deptno IS NULL{code}
> Instead of:
> {code:java}
> FALSE{code}
> Similarly, the following expression:
> {code:java}
> (deptno <> 20 OR deptno IS NULL) AND deptno = 10{code}
> Is simplified to:
> {code:java}
> deptno = 10 OR deptno IS NULL{code}
> Instead of:
> {code:java}
> deptno = 10{code}
> A diff with related unit tests (for RelBuilderTest.java) is in the attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4332) Improve error when planning rule produces a relation with a wrong rowtype
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-4332:
Description:
Current code in {{VolcanoPlanner}}:
{code:java}
assert RelOptUtil.equal(
"rel rowtype",
rel.getRowType(),
"equivRel rowtype",
equivRel.getRowType(),
Litmus.THROW);
equivRel = ensureRegistered(equivRel, null);
{code}
The suggested changes:
1) Use regular error {{IllegalArgumentException}} rather than an assertion
2) Make sure the exception message contains fine-grained information on what
went wrong. For instance: {{name: non nullable -> nullable; description:
character -> int; ...}}
was:
Current coed in {{VolcanoPlanner}}:
{code:java}
assert RelOptUtil.equal(
"rel rowtype",
rel.getRowType(),
"equivRel rowtype",
equivRel.getRowType(),
Litmus.THROW);
equivRel = ensureRegistered(equivRel, null);
{code}
The suggested changes:
1) Use regular error {{IllegalArgumentException}} rather than an assertion
2) Make sure the exception message contains fine-grained information on what
went wrong. For instance: {{name: non nullable -> nullable; description:
character -> int; ...}}
> Improve error when planning rule produces a relation with a wrong rowtype
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4333) The Sort rel should be decorrelated even though it has fetch or limit when its parent is not a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-4333:
Labels: pull-request-available (was: )
> The Sort rel should be decorrelated even though it has fetch or limit when
> its parent is not a Correlate
>
>
> Key: CALCITE-4333
> URL: https://issues.apache.org/jira/browse/CALCITE-4333
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Check this test in SqlToRelConverterTest:
> {code:java}
> @Test void testSortLimitWithCorrelateInput() {
> final String sql = "" +
> "SELECT deptno, ename\n" +
> "FROM\n" +
> "(SELECT DISTINCT deptno FROM emp) t1,\n" +
> " LATERAL (\n" +
> "SELECT ename, sal\n" +
> "FROM emp\n" +
> "WHERE deptno = t1.deptno)\n" +
> "ORDER BY ename DESC\n" +
> "LIMIT 3";
> sql(sql).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4333) The Sort rel should be decorrelated even though it has fetch or limit when its parent is not a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-4333:
Description:
Check this test in SqlToRelConverterTest:
{code:java}
@Test void testSortLimitWithCorrelateInput() {
final String sql = "" +
"SELECT deptno, ename\n" +
"FROM\n" +
"(SELECT DISTINCT deptno FROM emp) t1,\n" +
" LATERAL (\n" +
"SELECT ename, sal\n" +
"FROM emp\n" +
"WHERE deptno = t1.deptno)\n" +
"ORDER BY ename DESC\n" +
"LIMIT 3";
sql(sql).ok();
}
{code}
> The Sort rel should be decorrelated even though it has fetch or limit when
> its parent is not a Correlate
>
>
> Key: CALCITE-4333
> URL: https://issues.apache.org/jira/browse/CALCITE-4333
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Fix For: 1.27.0
>
>
> Check this test in SqlToRelConverterTest:
> {code:java}
> @Test void testSortLimitWithCorrelateInput() {
> final String sql = "" +
> "SELECT deptno, ename\n" +
> "FROM\n" +
> "(SELECT DISTINCT deptno FROM emp) t1,\n" +
> " LATERAL (\n" +
> "SELECT ename, sal\n" +
> "FROM emp\n" +
> "WHERE deptno = t1.deptno)\n" +
> "ORDER BY ename DESC\n" +
> "LIMIT 3";
> sql(sql).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4333) The Sort rel should be decorrelated even though it has fetch or limit when its parent is not a Correlate
Danny Chen created CALCITE-4333: --- Summary: The Sort rel should be decorrelated even though it has fetch or limit when its parent is not a Correlate Key: CALCITE-4333 URL: https://issues.apache.org/jira/browse/CALCITE-4333 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.26.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.27.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4331) RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
[
https://issues.apache.org/jira/browse/CALCITE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212264#comment-17212264
]
Vladimir Sitnikov commented on CALCITE-4331:
Just for the reference, the added {{isNullable()}} check triggers no new
failures in CI.
> RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
> --
>
> Key: CALCITE-4331
> URL: https://issues.apache.org/jira/browse/CALCITE-4331
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
> That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
> NULLABLE}} in a planner rule, even though, the planner typically does not
> allow to loosen field nullability (e.g. transform from {{CHARACTER NON NULL}}
> to {{CHARACTER NULLABLE}} typically fails with rel {{rowtype...equivRel
> rowtype}} assertion error).
> What I got was silent wrong results {{count(name)}} was transformed to
> {{count(\*)}} since Calcite believed the field was non-nullable, and it did
> not care that I provided nullable rex for the implementation. Frankly
> speaking, I assumed "no asserts => the rowtypes in call.transformTo(..) are
> ok"
> Current code:
> {code:java}
> for (Pair pair : Pair.zip(f1, f2)) {
> final RelDataType type1 = pair.left.getType();
> final RelDataType type2 = pair.right.getType();
> // If one of the types is ANY comparison should succeed
> if (type1.getSqlTypeName() == SqlTypeName.ANY
> || type2.getSqlTypeName() == SqlTypeName.ANY) {
> continue;
> }
> if (!type1.equals(type2)) {
> return false;
> }
> {code}
> I'm inclined add a check that both types have the same {{isNullable()}} value.
> ANY checks were added in
> https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
> [~jacques], [~julianhyde], do you know if it is OK to add the following?
> {code:java}
> if (type1.isNullable() != type2.isNullable()) {
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4331) RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
[
https://issues.apache.org/jira/browse/CALCITE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-4331:
Labels: pull-request-available (was: )
> RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
> --
>
> Key: CALCITE-4331
> URL: https://issues.apache.org/jira/browse/CALCITE-4331
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
> That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
> NULLABLE}} in a planner rule, even though, the planner typically does not
> allow to loosen field nullability (e.g. transform from {{CHARACTER NON NULL}}
> to {{CHARACTER NULLABLE}} typically fails with rel {{rowtype...equivRel
> rowtype}} assertion error).
> What I got was silent wrong results {{count(name)}} was transformed to
> {{count(\*)}} since Calcite believed the field was non-nullable, and it did
> not care that I provided nullable rex for the implementation. Frankly
> speaking, I assumed "no asserts => the rowtypes in call.transformTo(..) are
> ok"
> Current code:
> {code:java}
> for (Pair pair : Pair.zip(f1, f2)) {
> final RelDataType type1 = pair.left.getType();
> final RelDataType type2 = pair.right.getType();
> // If one of the types is ANY comparison should succeed
> if (type1.getSqlTypeName() == SqlTypeName.ANY
> || type2.getSqlTypeName() == SqlTypeName.ANY) {
> continue;
> }
> if (!type1.equals(type2)) {
> return false;
> }
> {code}
> I'm inclined add a check that both types have the same {{isNullable()}} value.
> ANY checks were added in
> https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
> [~jacques], [~julianhyde], do you know if it is OK to add the following?
> {code:java}
> if (type1.isNullable() != type2.isNullable()) {
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3679) Allow lambda expressions in SQL queries
[ https://issues.apache.org/jira/browse/CALCITE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212235#comment-17212235 ] Ritesh commented on CALCITE-3679: - Hey [~hyuan] I really wanted to work on this issue. But due to some personal reasons I might not be able to work on it this year. > Allow lambda expressions in SQL queries > --- > > Key: CALCITE-3679 > URL: https://issues.apache.org/jira/browse/CALCITE-3679 > Project: Calcite > Issue Type: New Feature >Reporter: Ritesh >Assignee: Ritesh >Priority: Major > Labels: pull-request-available > Attachments: [CALCITE-3679]_Basic_implementation.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://teradata.github.io/presto/docs/0.167-t/functions/lambda.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4332) Improve error when planning rule produces a relation with a wrong rowtype
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212228#comment-17212228
]
Chunwei Lei commented on CALCITE-4332:
--
That would be great. I met the same confusing error message couple of times. It
is hard to figure out which field went wrong in the current message.
> Improve error when planning rule produces a relation with a wrong rowtype
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current coed in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4332) Improve error when planning rule produces a relation with a wrong rowtype
Vladimir Sitnikov created CALCITE-4332:
--
Summary: Improve error when planning rule produces a relation with
a wrong rowtype
Key: CALCITE-4332
URL: https://issues.apache.org/jira/browse/CALCITE-4332
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.26.0
Reporter: Vladimir Sitnikov
Current coed in {{VolcanoPlanner}}:
{code:java}
assert RelOptUtil.equal(
"rel rowtype",
rel.getRowType(),
"equivRel rowtype",
equivRel.getRowType(),
Litmus.THROW);
equivRel = ensureRegistered(equivRel, null);
{code}
The suggested changes:
1) Use regular error {{IllegalArgumentException}} rather than an assertion
2) Make sure the exception message contains fine-grained information on what
went wrong. For instance: {{name: non nullable -> nullable; description:
character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4331) RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
[
https://issues.apache.org/jira/browse/CALCITE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Sitnikov updated CALCITE-4331:
---
Description:
Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
NULLABLE}} in a planner rule, even though, the planner typically does not allow
to loosen field nullability (e.g. transform from {{CHARACTER NON NULL}} to
{{CHARACTER NULLABLE}} typically fails with rel {{rowtype...equivRel rowtype}}
assertion error).
What I got was silent wrong results {{count(name)}} was transformed to
{{count(\*)}} since Calcite believed the field was non-nullable, and it did not
care that I provided nullable rex for the implementation. Frankly speaking, I
assumed "no asserts => the rowtypes in call.transformTo(..) are ok"
Current code:
{code:java}
for (Pair pair : Pair.zip(f1, f2)) {
final RelDataType type1 = pair.left.getType();
final RelDataType type2 = pair.right.getType();
// If one of the types is ANY comparison should succeed
if (type1.getSqlTypeName() == SqlTypeName.ANY
|| type2.getSqlTypeName() == SqlTypeName.ANY) {
continue;
}
if (!type1.equals(type2)) {
return false;
}
{code}
I'm inclined add a check that both types have the same {{isNullable()}} value.
ANY checks were added in
https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
[~jacques], [~julianhyde], do you know if it is OK to add the following?
{code:java}
if (type1.isNullable() != type2.isNullable()) {
return false;
}
{code}
was:
Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
NULLABLE}} in a planner rule, even though, the planner does not allow to loosen
field nullability.
What I got was silent wrong results {{count(name)}} was transformed to
{{count(\*)}} since Calcite believed the field was non-nullable, and it did not
care that I provided nullable rex for the implementation. Frankly speaking, I
assumed "no asserts => the rowtypes in call.transformTo(..) are ok"
Current code:
{code:java}
for (Pair pair : Pair.zip(f1, f2)) {
final RelDataType type1 = pair.left.getType();
final RelDataType type2 = pair.right.getType();
// If one of the types is ANY comparison should succeed
if (type1.getSqlTypeName() == SqlTypeName.ANY
|| type2.getSqlTypeName() == SqlTypeName.ANY) {
continue;
}
if (!type1.equals(type2)) {
return false;
}
{code}
I'm inclined add a check that both types have the same {{isNullable()}} value.
ANY checks were added in
https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
[~jacques], [~julianhyde], do you know if it is OK to add the following?
{code:java}
if (type1.isNullable() != type2.isNullable()) {
return false;
}
{code}
> RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
> --
>
> Key: CALCITE-4331
> URL: https://issues.apache.org/jira/browse/CALCITE-4331
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
> That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
> NULLABLE}} in a planner rule, even though, the planner typically does not
> allow to loosen field nullability (e.g. transform from {{CHARACTER NON NULL}}
> to {{CHARACTER NULLABLE}} typically fails with rel {{rowtype...equivRel
> rowtype}} assertion error).
> What I got was silent wrong results {{count(name)}} was transformed to
> {{count(\*)}} since Calcite believed the field was non-nullable, and it did
> not care that I provided nullable rex for the implementation. Frankly
> speaking, I assumed "no asserts => the rowtypes in call.transformTo(..) are
> ok"
> Current code:
> {code:java}
> for (Pair pair : Pair.zip(f1, f2)) {
> final RelDataType type1 = pair.left.getType();
> final RelDataType type2 = pair.right.getType();
> // If one of the types is ANY comparison should succeed
> if (type1.getSqlTypeName() == SqlTypeName.ANY
> || type2.getSqlTypeName() == SqlTypeName.ANY) {
> continue;
> }
> if (!type1.equals(type2)) {
> return false;
> }
> {code}
> I'm inclined add a check that both types have the same {{isNullable()}} value.
> ANY checks were added in
> https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
> [~jacques], [~julianhyde], do you know if it is
[jira] [Updated] (CALCITE-4331) RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
[
https://issues.apache.org/jira/browse/CALCITE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Sitnikov updated CALCITE-4331:
---
Description:
Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
NULLABLE}} in a planner rule, even though, the planner does not allow to loosen
field nullability.
What I got was silent wrong results {{count(name)}} was transformed to
{{count(\*)}} since Calcite believed the field was non-nullable, and it did not
care that I provided nullable rex for the implementation. Frankly speaking, I
assumed "no asserts => the rowtypes in call.transformTo(..) are ok"
Current code:
{code:java}
for (Pair pair : Pair.zip(f1, f2)) {
final RelDataType type1 = pair.left.getType();
final RelDataType type2 = pair.right.getType();
// If one of the types is ANY comparison should succeed
if (type1.getSqlTypeName() == SqlTypeName.ANY
|| type2.getSqlTypeName() == SqlTypeName.ANY) {
continue;
}
if (!type1.equals(type2)) {
return false;
}
{code}
I'm inclined add a check that both types have the same {{isNullable()}} value.
ANY checks were added in
https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
[~jacques], [~julianhyde], do you know if it is OK to add the following?
{code:java}
if (type1.isNullable() != type2.isNullable()) {
return false;
}
{code}
was:
Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
NULLABLE}} in a planner rule, even though, the planner does not allow to loosen
field nullability.
What I got was silent wrong results {{count(name)}} was transformed to
{{count(*)}} since Calcite believed the field was non-nullable, and it did not
care that I provided nullable rex for the implementation. Frankly speaking, I
assumed "no asserts => the rowtypes in call.transformTo(..) are ok"
Current code:
{code:java}
for (Pair pair : Pair.zip(f1, f2)) {
final RelDataType type1 = pair.left.getType();
final RelDataType type2 = pair.right.getType();
// If one of the types is ANY comparison should succeed
if (type1.getSqlTypeName() == SqlTypeName.ANY
|| type2.getSqlTypeName() == SqlTypeName.ANY) {
continue;
}
if (!type1.equals(type2)) {
return false;
}
{code}
I'm inclined add a check that both types have the same {{isNullable()}} value.
ANY checks were added in
https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
[~jacques], [~julianhyde], do you know if it is OK to add {{if
(type1.isNullable() != type2.isNullable()) { return false; } }} ?
> RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
> --
>
> Key: CALCITE-4331
> URL: https://issues.apache.org/jira/browse/CALCITE-4331
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
> That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
> NULLABLE}} in a planner rule, even though, the planner does not allow to
> loosen field nullability.
> What I got was silent wrong results {{count(name)}} was transformed to
> {{count(\*)}} since Calcite believed the field was non-nullable, and it did
> not care that I provided nullable rex for the implementation. Frankly
> speaking, I assumed "no asserts => the rowtypes in call.transformTo(..) are
> ok"
> Current code:
> {code:java}
> for (Pair pair : Pair.zip(f1, f2)) {
> final RelDataType type1 = pair.left.getType();
> final RelDataType type2 = pair.right.getType();
> // If one of the types is ANY comparison should succeed
> if (type1.getSqlTypeName() == SqlTypeName.ANY
> || type2.getSqlTypeName() == SqlTypeName.ANY) {
> continue;
> }
> if (!type1.equals(type2)) {
> return false;
> }
> {code}
> I'm inclined add a check that both types have the same {{isNullable()}} value.
> ANY checks were added in
> https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
> [~jacques], [~julianhyde], do you know if it is OK to add the following?
> {code:java}
> if (type1.isNullable() != type2.isNullable()) {
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4331) RelOptUtil#areRowTypesEqual should compare type nullability even for ANY types
Vladimir Sitnikov created CALCITE-4331:
--
Summary: RelOptUtil#areRowTypesEqual should compare type
nullability even for ANY types
Key: CALCITE-4331
URL: https://issues.apache.org/jira/browse/CALCITE-4331
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.26.0
Reporter: Vladimir Sitnikov
Calcite ignores data types completely if one of them is {{SqlTypeName.ANY}}.
That enables to transform rowtype from {{name ANY NON NULL}} to {{name ANY
NULLABLE}} in a planner rule, even though, the planner does not allow to loosen
field nullability.
What I got was silent wrong results {{count(name)}} was transformed to
{{count(*)}} since Calcite believed the field was non-nullable, and it did not
care that I provided nullable rex for the implementation. Frankly speaking, I
assumed "no asserts => the rowtypes in call.transformTo(..) are ok"
Current code:
{code:java}
for (Pair pair : Pair.zip(f1, f2)) {
final RelDataType type1 = pair.left.getType();
final RelDataType type2 = pair.right.getType();
// If one of the types is ANY comparison should succeed
if (type1.getSqlTypeName() == SqlTypeName.ANY
|| type2.getSqlTypeName() == SqlTypeName.ANY) {
continue;
}
if (!type1.equals(type2)) {
return false;
}
{code}
I'm inclined add a check that both types have the same {{isNullable()}} value.
ANY checks were added in
https://github.com/apache/calcite/commit/5562fc3952f61d943f109796c3d62b7390a8ac48
[~jacques], [~julianhyde], do you know if it is OK to add {{if
(type1.isNullable() != type2.isNullable()) { return false; } }} ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212173#comment-17212173 ] Chunwei Lei edited comment on CALCITE-4223 at 10/12/20, 7:13 AM: - I would check it. Thank you for your reply( :D Just back from vacation). was (Author: chunwei lei): I would check it. Thank you for your reply(:DJust back from vacation). > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212173#comment-17212173 ] Chunwei Lei commented on CALCITE-4223: -- I would check it. Thank you for your reply(:DJust back from vacation). > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4310) Add API backward compatibility verification
[
https://issues.apache.org/jira/browse/CALCITE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212165#comment-17212165
]
Chunwei Lei commented on CALCITE-4310:
--
Interesting tools(y)
> Add API backward compatibility verification
> ---
>
> Key: CALCITE-4310
> URL: https://issues.apache.org/jira/browse/CALCITE-4310
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.25.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> One of the key questions when preparing a release is "what are the breaking
> changes?", and it would be nice if we could document the changes and minimize
> them.
> Here are the tools that might help:
> * japicmp: https://github.com/melix/japicmp-gradle-plugin (integrated with
> Gradle, enables to configure extra filters so we could exclude
> {{@API(internal)}} from public apis)
> * https://abi-laboratory.pro/java/tracker/ : (no idea how it compares with
> japicmp)
> * Kotlin-specific: https://github.com/Kotlin/binary-compatibility-validator
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
