[jira] [Commented] (CALCITE-2069) RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type
[
https://issues.apache.org/jira/browse/CALCITE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270275#comment-16270275
]
Julian Hyde commented on CALCITE-2069:
--
Reviewing
https://github.com/apache/calcite/pull/574/commits/f113abd7f67291941517abfeb875849ae60bdaaa:
* The fix looks reasonable to me, and it doesn't break any tests.
* [~jnadeau], The fix removes code that you added 4 years ago without adding
any tests. So, it might be worth you reviewing this.
* Can you clean up your naming? You have a type called "intType" which is
actually an ANY.
* Why are you using {{patternField}}? That is a *very* obscure method.
> RexSimplify.removeNullabilityCast() always removes cast for operand with ANY
> type
> -
>
> Key: CALCITE-2069
> URL: https://issues.apache.org/jira/browse/CALCITE-2069
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>
> When a field is received from Dynamic Table, its type left {{ANY}}, and it is
> used in the filter condition with the cast, which actually should produce
> physical cast (for example we are trying to cast varchar to boolean)
> {{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only
> field in condition.
> This test helps to observe this issue:
> {code:java}
> @Test public void testFilterCastAny() {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelDataType intType =
> builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
> RelNode root =
> builder.scan("EMP")
> .filter(
> builder.cast(
> builder.patternField("varchar_field", intType, 0),
> SqlTypeName.BOOLEAN))
> .build();
> assertThat(str(root),
> is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT
> NULL])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n"));
> }
> {code}
> It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
> of its arguments has {{ANY}} type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (CALCITE-1616) Data profiler
[
https://issues.apache.org/jira/browse/CALCITE-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-1616.
--
Resolution: Fixed
Fix Version/s: 1.15.0
Fixed in
[dad58186|http://git-wip-us.apache.org/repos/asf/calcite/commit/dad58186].
Does not work on JDK 7 (due to com.yahoo.data-sketches requiring JDK 8);
applicable tests no-op on JDK 7.
> Data profiler
> -
>
> Key: CALCITE-1616
> URL: https://issues.apache.org/jira/browse/CALCITE-1616
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
> Fix For: 1.15.0
>
>
> Profiling looks at a data set and infers characteristics and constraints
> about the data.
> Some applications:
> * it helps users understand their data,
> * inferred constraints may allow additional optimizations (e.g. a foreign key
> allows semi-join removal),
> * column statistics help the optimizer estimate the selectivity of filters
> and joins,
> * joint cardinalities drive the algorithm that chooses which tiles of a
> lattice to materialize.
> Imagine you ran a profiler on a data set of 1 million rows and columns
> [orderId, gender, state, zipcode, productId, productName, brand]. Here is
> some sample output:
> {noformat}
> {"type": "distribution", "columns": [], "cardinality": 100)},
> {"type": "distribution", "columns": ["gender"], "cardinality": 2, nullCount:
> 0, values: ["F", "M"])},
> {"type": "distribution", "columns": ["state"], "cardinality": 50)},
> {"type": "distribution", "columns": ["zipcode"], "cardinality": 43000,
> "nullCount": 0)},
> {"type": "distribution", "columns": ["state", "zipcode"], "cardinality":
> 43419)},
> {"type": "unique", "columns": ["orderId"]},
> {"type": "fd", "columns": ["productId"], "depend": ["brand", "productName"]},
> {noformat}
> Note:
> * the cardinality of 0 columns is the count of the data set;
> * "nullCount" and "values" are only present for distributions of 1 column;
> * "nullCount" may be is omitted if 0;
> * "values" is only present if there are N or fewer values
> * "distribution" of 2 or more columns is only output if it is "interesting";
> in the case of ["state", "zipcode"] it is interesting because the joint
> cardinality 43,419 is fewer than the cardinality 999,982 that would be
> expected if they were uniform and independent;
> * "fd" and "unique" are minimal. For example, we don't output
> "unique(orderId, productId)" if we have "unique(orderId)".
> Other ideas:
> * Some measure of skewedness. Does one value occur many more times than
> others?
> * Don't compute joint distributions for the power set of columns. This
> requires memory exponential in the number of columns. In pass 1 compute
> single-column distributions. In pass N compute N-column distributions only
> for the combinations of columns that the previous pass indicates will be
> interesting.
> * Use HyperLogLog to compute cardinalities.
> * Add low-cardinality columns to joint distributions. Rather than computing
> cardinality(zipcode, state) compute cardinality(zipcode, state, gender="F")
> and cardinality(zipcode, state, gender="M"). Because HLL rolls up losslessly,
> with 2x the memory you can compute 3 results: cardinality(zipcode, state),
> cardinality(zipcode, gender), cardinality(state, gender).
> * Approximate histograms: approximate quartiles? Or buckets with exact counts
> in each range?
> * Allow passing previous results into the algorithm. If you know the previous
> histogram of the orderDate column it is easier to compute its new histogram
> than if you start from scratch.
> * HyperLogLog is inaccurate for low cardinalities, so keep all values until
> the number of values exceeds a threshold, then transition to buckets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (CALCITE-2005) Core project failed during default-test on Windows
[
https://issues.apache.org/jira/browse/CALCITE-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-2005.
--
Resolution: Fixed
Fix Version/s: 1.15.0
Fixed in
[9bb54006|http://git-wip-us.apache.org/repos/asf/calcite/commit/9bb54006].
> Core project failed during default-test on Windows
> --
>
> Key: CALCITE-2005
> URL: https://issues.apache.org/jira/browse/CALCITE-2005
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.14.0
> Environment: java version "1.8.0_144",
> maven version "3.5.0", windows
>Reporter: Dmitry Shmushkis
>Assignee: Julian Hyde
>Priority: Minor
> Fix For: 1.15.0
>
>
> *The following failure is generated when running "mvn clean install"*
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Calcite SUCCESS [ 2.728
> s]
> [INFO] Calcite Linq4j . SUCCESS [ 13.181
> s]
> [INFO] Calcite Core ... FAILURE [03:31
> min]
> [INFO] Calcite Cassandra .. SKIPPED
> [INFO] Calcite Druid .. SKIPPED
> [INFO] Calcite Elasticsearch .. SKIPPED
> [INFO] Calcite Elasticsearch5 . SKIPPED
> [INFO] Calcite Examples ... SKIPPED
> [INFO] Calcite Example CSV SKIPPED
> [INFO] Calcite Example Function ... SKIPPED
> [INFO] Calcite File ... SKIPPED
> [INFO] Calcite MongoDB SKIPPED
> [INFO] Calcite Pig SKIPPED
> [INFO] Calcite Piglet . SKIPPED
> [INFO] Calcite Plus ... SKIPPED
> [INFO] Calcite Spark .. SKIPPED
> [INFO] Calcite Splunk . SKIPPED
> [INFO] Calcite Ubenchmark . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:47 min
> [INFO] Finished at: 2017-10-09T14:10:17+03:00
> [INFO] Final Memory: 72M/1649M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project calcite-core: There are test failures.
> [ERROR]
> [ERROR] Please refer to D:\Project\git\calcite\core\target\surefire-reports
> for the individual test results.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> (default-test) on project calcite-core: There are test failures.
> Please refer to D:\Project\git\calcite\core\target\surefire-reports for the
> individual test results.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Updated] (CALCITE-2005) Core project failed during default-test on Windows
[
https://issues.apache.org/jira/browse/CALCITE-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-2005:
-
Summary: Core project failed during default-test on Windows (was: Core
project failed during default-test)
> Core project failed during default-test on Windows
> --
>
> Key: CALCITE-2005
> URL: https://issues.apache.org/jira/browse/CALCITE-2005
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.14.0
> Environment: java version "1.8.0_144",
> maven version "3.5.0", windows
>Reporter: Dmitry Shmushkis
>Assignee: Julian Hyde
>Priority: Minor
>
> *The following failure is generated when running "mvn clean install"*
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Calcite SUCCESS [ 2.728
> s]
> [INFO] Calcite Linq4j . SUCCESS [ 13.181
> s]
> [INFO] Calcite Core ... FAILURE [03:31
> min]
> [INFO] Calcite Cassandra .. SKIPPED
> [INFO] Calcite Druid .. SKIPPED
> [INFO] Calcite Elasticsearch .. SKIPPED
> [INFO] Calcite Elasticsearch5 . SKIPPED
> [INFO] Calcite Examples ... SKIPPED
> [INFO] Calcite Example CSV SKIPPED
> [INFO] Calcite Example Function ... SKIPPED
> [INFO] Calcite File ... SKIPPED
> [INFO] Calcite MongoDB SKIPPED
> [INFO] Calcite Pig SKIPPED
> [INFO] Calcite Piglet . SKIPPED
> [INFO] Calcite Plus ... SKIPPED
> [INFO] Calcite Spark .. SKIPPED
> [INFO] Calcite Splunk . SKIPPED
> [INFO] Calcite Ubenchmark . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:47 min
> [INFO] Finished at: 2017-10-09T14:10:17+03:00
> [INFO] Final Memory: 72M/1649M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project calcite-core: There are test failures.
> [ERROR]
> [ERROR] Please refer to D:\Project\git\calcite\core\target\surefire-reports
> for the individual test results.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> (default-test) on project calcite-core: There are test failures.
> Please refer to D:\Project\git\calcite\core\target\surefire-reports for the
> individual test results.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
[jira] [Commented] (CALCITE-2067) RexBuilder can't handle NaN,Infinity double constants
[
https://issues.apache.org/jira/browse/CALCITE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269934#comment-16269934
]
Julian Hyde commented on CALCITE-2067:
--
Reviewing
https://github.com/apache/calcite/pull/572/commits/6c2eed677f5efdf09ad763a74fe7c0baa126dd43.
To be useful to other people besides Drill, this change needs to have larger
scope than just changing RexBuilder. You're effectively changing Calcite's SQL
language, so I think you should add tests and documentation for NaN and
Infinity in the SQL grammar, parser, validator, and operator implementations.
I don't feel comfortable allowing {{Double}} values in
{{RexLiteral.valueMatchesType}}. A lot of code relies upon the value being a
{{BigDecimal}}.
> RexBuilder can't handle NaN,Infinity double constants
> -
>
> Key: CALCITE-2067
> URL: https://issues.apache.org/jira/browse/CALCITE-2067
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Julian Hyde
>
> Currently we are working in Drill to allow support for Nan an Infinity values
> .
> When using such values with functions we see the NumberFormatException
> exception.
> Query example: _select sin(cast('NaN' as float))_
> RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
> Double values, that's why we get NumberFormatException for nan,inf values.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2066) RelOptUtil.splitJoinCondition() could not split condition with case after applying FilterReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269160#comment-16269160
]
Julian Hyde commented on CALCITE-2066:
--
I'll take a look. It won't make it into 1.15, I'm afraid.
> RelOptUtil.splitJoinCondition() could not split condition with case after
> applying FilterReduceExpressionsRule
> --
>
> Key: CALCITE-2066
> URL: https://issues.apache.org/jira/browse/CALCITE-2066
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>
> Query
> {code:sql}
> select * from emp a emp b where coalesce(a.mgr, -1) = coalesce(b.mgr, -1)
> {code}
> has logical plan
> {noformat}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], EMPNO0=[$9], ENAME0=[$10],
> JOB0=[$11], MGR0=[$12], HIREDATE0=[$13], SAL0=[$14], COMM0=[$15],
> DEPTNO0=[$16], SLACKER0=[$17])
> LogicalFilter(condition=[=(CASE(IS NOT NULL($3), $3, -1), CASE(IS NOT
> NULL($12), $12, -1))])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> After applying {{FilterReduceExpressionsRule}} condition in filter is changed
> to
> {noformat}
> CASE(IS NOT NULL($12), CASE(IS NOT NULL($3), =($3, $12), =(-1, $12)), CASE(IS
> NOT NULL($3), =($3, -1), CAST(true):BOOLEAN))
> {noformat}
> When this condition is passed into {{RelOptUtil.splitJoinCondition()}}
> method, {{leftJoinKeys}} and {{rightJoinKeys}} lists are left empty and
> passed condition is returned.
> *Note:* {{RelOptUtil.splitJoinCondition()}} works correctly for the initial
> condition without applying {{FilterReduceExpressionsRule}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-2067) RexBuilder can't handle NaN,Infinity double constants
[
https://issues.apache.org/jira/browse/CALCITE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269046#comment-16269046
]
Volodymyr Tkach edited comment on CALCITE-2067 at 11/28/17 5:04 PM:
[~julianhyde] Sorry, fixed.
The behavior is inconsistent when comparing these queries:
select sin(cast('NaN' as float)); -- fails with NumberFormatException
select sin(num_column) from some_table; -- Succeeds, even if num_column
contains NaN,Infinity doubles
In this [PR|https://github.com/apache/calcite/pull/572], I am currently working
on, I have overloaded RexBuilder#makeApproxLiteral to handle Double values as
well. Will it be appropriate?
was (Author: volodymyr.tkach):
Sorry, fixed.
The behavior is inconsistent when comparing these queries:
select sin(cast('NaN' as float)); -- fails with NumberFormatException
select sin(num_column) from some_table; -- Succeeds, even if num_column
contains NaN,Infinity doubles
In this [PR|https://github.com/apache/calcite/pull/572], I am currently working
on, I have overloaded RexBuilder#makeApproxLiteral to handle Double values as
well. Will it be appropriate?
> RexBuilder can't handle NaN,Infinity double constants
> -
>
> Key: CALCITE-2067
> URL: https://issues.apache.org/jira/browse/CALCITE-2067
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Julian Hyde
>
> Currently we are working in Drill to allow support for Nan an Infinity values
> .
> When using such values with functions we see the NumberFormatException
> exception.
> Query example: _select sin(cast('NaN' as float))_
> RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
> Double values, that's why we get NumberFormatException for nan,inf values.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2067) RexBuilder can't handle NaN,Infinity double constants
[
https://issues.apache.org/jira/browse/CALCITE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269046#comment-16269046
]
Volodymyr Tkach commented on CALCITE-2067:
--
Sorry, fixed.
The behavior is inconsistent when comparing these queries:
select sin(cast('NaN' as float)); -- fails with NumberFormatException
select sin(num_column) from some_table; -- Succeeds, even if num_column
contains NaN,Infinity doubles
In this [PR|https://github.com/apache/calcite/pull/572], I am currently working
on, I have overloaded RexBuilder#makeApproxLiteral to handle Double values as
well. Will it be appropriate?
> RexBuilder can't handle NaN,Infinity double constants
> -
>
> Key: CALCITE-2067
> URL: https://issues.apache.org/jira/browse/CALCITE-2067
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Julian Hyde
>
> Currently we are working in Drill to allow support for Nan an Infinity values
> .
> When using such values with functions we see the NumberFormatException
> exception.
> Query example: _select sin(cast('NaN' as float))_
> RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
> Double values, that's why we get NumberFormatException for nan,inf values.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CALCITE-2067) RexBuilder can't handle NaN,Infinity double constants
[
https://issues.apache.org/jira/browse/CALCITE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Tkach updated CALCITE-2067:
-
Description:
Currently we are working in Drill to allow support for Nan an Infinity values .
When using such values with functions we see the NumberFormatException
exception.
Query example: _select sin(cast('NaN' as float))_
RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
Double values, that's why we get NumberFormatException for nan,inf values.
was:
Currently we are working in Drill to allow support for Nan an Infinity values .
When using such values with functions we see the NumberFormatException
exception.
Query example: select sin(cast('NaN' as float)) from (values(1)).
RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
Double values, that's why we get NumberFormatException for nan,inf values.
> RexBuilder can't handle NaN,Infinity double constants
> -
>
> Key: CALCITE-2067
> URL: https://issues.apache.org/jira/browse/CALCITE-2067
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Julian Hyde
>
> Currently we are working in Drill to allow support for Nan an Infinity values
> .
> When using such values with functions we see the NumberFormatException
> exception.
> Query example: _select sin(cast('NaN' as float))_
> RexBuilder#clean and other RexBuilder's methods use BigDecimal for handling
> Double values, that's why we get NumberFormatException for nan,inf values.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2069) RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type
[
https://issues.apache.org/jira/browse/CALCITE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268943#comment-16268943
]
Volodymyr Vysotskyi commented on CALCITE-2069:
--
I created pull request for this issue:
https://github.com/apache/calcite/pull/574. Could someone please take a look?
> RexSimplify.removeNullabilityCast() always removes cast for operand with ANY
> type
> -
>
> Key: CALCITE-2069
> URL: https://issues.apache.org/jira/browse/CALCITE-2069
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>
> When a field is received from Dynamic Table, its type left {{ANY}}, and it is
> used in the filter condition with the cast, which actually should produce
> physical cast (for example we are trying to cast varchar to boolean)
> {{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only
> field in condition.
> This test helps to observe this issue:
> {code:java}
> @Test public void testFilterCastAny() {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelDataType intType =
> builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
> RelNode root =
> builder.scan("EMP")
> .filter(
> builder.cast(
> builder.patternField("varchar_field", intType, 0),
> SqlTypeName.BOOLEAN))
> .build();
> assertThat(str(root),
> is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT
> NULL])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n"));
> }
> {code}
> It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
> of its arguments has {{ANY}} type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CALCITE-2069) RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type
[
https://issues.apache.org/jira/browse/CALCITE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2069:
-
Description:
When a field is received from Dynamic Table, its type left {{ANY}}, and it is
used in the filter condition with the cast, which actually should produce
physical cast (for example we are trying to cast varchar to boolean)
{{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only field
in condition.
This test helps to observe this issue:
{code:java}
@Test public void testFilterCastAny() {
final RelBuilder builder = RelBuilder.create(config().build());
final RelDataType intType =
builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
RelNode root =
builder.scan("EMP")
.filter(
builder.cast(
builder.patternField("varchar_field", intType, 0),
SqlTypeName.BOOLEAN))
.build();
assertThat(str(root),
is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT
NULL])\n"
+ " LogicalTableScan(table=[[scott, EMP]])\n"));
}
{code}
It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
of its arguments has {{ANY}} type.
was:
When a field is received from Dynamic Table, its type left {{ANY}}, and it is
used in the filter condition with the cast, which actually should produce
physical cast (for example we are trying to cast varchar to boolean)
{{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only field
in condition.
This test helps to observe this issue:
{code:java}
@Test public void testFilterCastAny() {
final RelBuilder builder = RelBuilder.create(config().build());
final RelDataType intType =
builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
RelNode root =
builder.scan("EMP")
.filter(
builder.cast(
builder.patternField("varchar_field", intType, 0),
SqlTypeName.BOOLEAN))
.build();
assertThat(str(root),
is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT NULL])\n"
+ " LogicalTableScan(table=[[scott, EMP]])\n"));
}
{code}
It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
of its arguments has {{ANY}} type.
> RexSimplify.removeNullabilityCast() always removes cast for operand with ANY
> type
> -
>
> Key: CALCITE-2069
> URL: https://issues.apache.org/jira/browse/CALCITE-2069
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>
> When a field is received from Dynamic Table, its type left {{ANY}}, and it is
> used in the filter condition with the cast, which actually should produce
> physical cast (for example we are trying to cast varchar to boolean)
> {{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only
> field in condition.
> This test helps to observe this issue:
> {code:java}
> @Test public void testFilterCastAny() {
> final RelBuilder builder = RelBuilder.create(config().build());
> final RelDataType intType =
> builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
> RelNode root =
> builder.scan("EMP")
> .filter(
> builder.cast(
> builder.patternField("varchar_field", intType, 0),
> SqlTypeName.BOOLEAN))
> .build();
> assertThat(str(root),
> is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT
> NULL])\n"
> + " LogicalTableScan(table=[[scott, EMP]])\n"));
> }
> {code}
> It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
> of its arguments has {{ANY}} type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CALCITE-2069) RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type
Volodymyr Vysotskyi created CALCITE-2069:
Summary: RexSimplify.removeNullabilityCast() always removes cast
for operand with ANY type
Key: CALCITE-2069
URL: https://issues.apache.org/jira/browse/CALCITE-2069
Project: Calcite
Issue Type: Bug
Reporter: Volodymyr Vysotskyi
Assignee: Julian Hyde
When a field is received from Dynamic Table, its type left {{ANY}}, and it is
used in the filter condition with the cast, which actually should produce
physical cast (for example we are trying to cast varchar to boolean)
{{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only field
in condition.
This test helps to observe this issue:
{code:java}
@Test public void testFilterCastAny() {
final RelBuilder builder = RelBuilder.create(config().build());
final RelDataType intType =
builder.getTypeFactory().createSqlType(SqlTypeName.ANY);
RelNode root =
builder.scan("EMP")
.filter(
builder.cast(
builder.patternField("varchar_field", intType, 0),
SqlTypeName.BOOLEAN))
.build();
assertThat(str(root),
is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT NULL])\n"
+ " LogicalTableScan(table=[[scott, EMP]])\n"));
}
{code}
It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any
of its arguments has {{ANY}} type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CALCITE-2068) throws NPE in RelMdPercentageOriginalRows when getPercentageOriginalRows returns null
[
https://issues.apache.org/jira/browse/CALCITE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated CALCITE-2068:
Description:
these two methods will get NEP when mq.getPercentageOriginalRows return null:
{code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
double numerator = 0.0;
double denominator = 0.0;
for (RelNode input : rel.getInputs()) {
double rowCount = mq.getRowCount(input);
double percentage = mq.getPercentageOriginalRows(input);
if (percentage != 0.0) {
denominator += rowCount / percentage;
numerator += rowCount;
}
}
return quotientForPercentage(numerator, denominator);
}
public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
double left = mq.getPercentageOriginalRows(rel.getLeft());
double right = mq.getPercentageOriginalRows(rel.getRight());
return left * right;
}
{code}
assigning null to double (NOT Double type) will get NPE.
was:
these two methods will get NEP:
{code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
double numerator = 0.0;
double denominator = 0.0;
for (RelNode input : rel.getInputs()) {
double rowCount = mq.getRowCount(input);
double percentage = mq.getPercentageOriginalRows(input);
if (percentage != 0.0) {
denominator += rowCount / percentage;
numerator += rowCount;
}
}
return quotientForPercentage(numerator, denominator);
}
public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
double left = mq.getPercentageOriginalRows(rel.getLeft());
double right = mq.getPercentageOriginalRows(rel.getRight());
return left * right;
}
{code}
getPercentageOriginalRows maybe return null, and assigning null to double will
get NPE.
> throws NPE in RelMdPercentageOriginalRows when getPercentageOriginalRows
> returns null
> -
>
> Key: CALCITE-2068
> URL: https://issues.apache.org/jira/browse/CALCITE-2068
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: godfrey he
>Assignee: Julian Hyde
>
> these two methods will get NEP when mq.getPercentageOriginalRows return null:
> {code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
> public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
> double numerator = 0.0;
> double denominator = 0.0;
> for (RelNode input : rel.getInputs()) {
> double rowCount = mq.getRowCount(input);
> double percentage = mq.getPercentageOriginalRows(input);
> if (percentage != 0.0) {
> denominator += rowCount / percentage;
> numerator += rowCount;
> }
> }
> return quotientForPercentage(numerator, denominator);
> }
> public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
> double left = mq.getPercentageOriginalRows(rel.getLeft());
> double right = mq.getPercentageOriginalRows(rel.getRight());
> return left * right;
> }
> {code}
> assigning null to double (NOT Double type) will get NPE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CALCITE-2068) throws NPE in RelMdPercentageOriginalRows when getPercentageOriginalRows returns null
godfrey he created CALCITE-2068:
---
Summary: throws NPE in RelMdPercentageOriginalRows when
getPercentageOriginalRows returns null
Key: CALCITE-2068
URL: https://issues.apache.org/jira/browse/CALCITE-2068
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.13.0
Reporter: godfrey he
Assignee: Julian Hyde
these two method will get NEP:
{code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
double numerator = 0.0;
double denominator = 0.0;
for (RelNode input : rel.getInputs()) {
double rowCount = mq.getRowCount(input);
double percentage = mq.getPercentageOriginalRows(input);
if (percentage != 0.0) {
denominator += rowCount / percentage;
numerator += rowCount;
}
}
return quotientForPercentage(numerator, denominator);
}
public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
double left = mq.getPercentageOriginalRows(rel.getLeft());
double right = mq.getPercentageOriginalRows(rel.getRight());
return left * right;
}
{code}
getPercentageOriginalRows maybe return null, and assigning null to double will
get NPE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CALCITE-2068) throws NPE in RelMdPercentageOriginalRows when getPercentageOriginalRows returns null
[
https://issues.apache.org/jira/browse/CALCITE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated CALCITE-2068:
Description:
these two methods will get NEP:
{code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
double numerator = 0.0;
double denominator = 0.0;
for (RelNode input : rel.getInputs()) {
double rowCount = mq.getRowCount(input);
double percentage = mq.getPercentageOriginalRows(input);
if (percentage != 0.0) {
denominator += rowCount / percentage;
numerator += rowCount;
}
}
return quotientForPercentage(numerator, denominator);
}
public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
double left = mq.getPercentageOriginalRows(rel.getLeft());
double right = mq.getPercentageOriginalRows(rel.getRight());
return left * right;
}
{code}
getPercentageOriginalRows maybe return null, and assigning null to double will
get NPE.
was:
these two method will get NEP:
{code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
double numerator = 0.0;
double denominator = 0.0;
for (RelNode input : rel.getInputs()) {
double rowCount = mq.getRowCount(input);
double percentage = mq.getPercentageOriginalRows(input);
if (percentage != 0.0) {
denominator += rowCount / percentage;
numerator += rowCount;
}
}
return quotientForPercentage(numerator, denominator);
}
public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
double left = mq.getPercentageOriginalRows(rel.getLeft());
double right = mq.getPercentageOriginalRows(rel.getRight());
return left * right;
}
{code}
getPercentageOriginalRows maybe return null, and assigning null to double will
get NPE.
> throws NPE in RelMdPercentageOriginalRows when getPercentageOriginalRows
> returns null
> -
>
> Key: CALCITE-2068
> URL: https://issues.apache.org/jira/browse/CALCITE-2068
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: godfrey he
>Assignee: Julian Hyde
>
> these two methods will get NEP:
> {code:title=RelMdPercentageOriginalRows.java|borderStyle=solid}
> public Double getPercentageOriginalRows(Union rel, RelMetadataQuery mq) {
> double numerator = 0.0;
> double denominator = 0.0;
> for (RelNode input : rel.getInputs()) {
> double rowCount = mq.getRowCount(input);
> double percentage = mq.getPercentageOriginalRows(input);
> if (percentage != 0.0) {
> denominator += rowCount / percentage;
> numerator += rowCount;
> }
> }
> return quotientForPercentage(numerator, denominator);
> }
> public Double getPercentageOriginalRows(Join rel, RelMetadataQuery mq) {
> double left = mq.getPercentageOriginalRows(rel.getLeft());
> double right = mq.getPercentageOriginalRows(rel.getRight());
> return left * right;
> }
> {code}
> getPercentageOriginalRows maybe return null, and assigning null to double
> will get NPE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2066) RelOptUtil.splitJoinCondition() could not split condition with case after applying FilterReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268673#comment-16268673
]
Volodymyr Vysotskyi commented on CALCITE-2066:
--
I think it's better to handle larger expression than fail after.
I created pull request for this Jira: https://github.com/apache/calcite/pull/573
I have disabled {{pushPredicateIntoCase}} only for the case when equal
conditions may be splitted. Could you please take a look?
> RelOptUtil.splitJoinCondition() could not split condition with case after
> applying FilterReduceExpressionsRule
> --
>
> Key: CALCITE-2066
> URL: https://issues.apache.org/jira/browse/CALCITE-2066
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>
> Query
> {code:sql}
> select * from emp a emp b where coalesce(a.mgr, -1) = coalesce(b.mgr, -1)
> {code}
> has logical plan
> {noformat}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], EMPNO0=[$9], ENAME0=[$10],
> JOB0=[$11], MGR0=[$12], HIREDATE0=[$13], SAL0=[$14], COMM0=[$15],
> DEPTNO0=[$16], SLACKER0=[$17])
> LogicalFilter(condition=[=(CASE(IS NOT NULL($3), $3, -1), CASE(IS NOT
> NULL($12), $12, -1))])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> After applying {{FilterReduceExpressionsRule}} condition in filter is changed
> to
> {noformat}
> CASE(IS NOT NULL($12), CASE(IS NOT NULL($3), =($3, $12), =(-1, $12)), CASE(IS
> NOT NULL($3), =($3, -1), CAST(true):BOOLEAN))
> {noformat}
> When this condition is passed into {{RelOptUtil.splitJoinCondition()}}
> method, {{leftJoinKeys}} and {{rightJoinKeys}} lists are left empty and
> passed condition is returned.
> *Note:* {{RelOptUtil.splitJoinCondition()}} works correctly for the initial
> condition without applying {{FilterReduceExpressionsRule}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
