[jira] [Created] (CALCITE-6435) SqlToRel conversion of IN expressions may lead to incorrect simplifications
Zoltan Haindrich created CALCITE-6435:
-
Summary: SqlToRel conversion of IN expressions may lead to
incorrect simplifications
Key: CALCITE-6435
URL: https://issues.apache.org/jira/browse/CALCITE-6435
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
the query must have the following features:
* not all columns are selected
** to enable `RelFieldTrimmer` to start a cycle
* two equivalent eq filters
** one in `IN` form (`ename in ( 'Sebastian' )`)
** a regular `=` (`ename = 'Sebastian'`)
* an unrelated filter like `deptno < 100`
the optimizer should more-or-less start with the `RelFieldTrimmer`
the issue happens like:
* at parse time both literals are parsed as `CHAR(n)`
* the number of values in the `IN` is below `inSubqueryThreshold` - so it gets
converted to a set of `=` filters
** expression is converted to OR form
** during conversion
[SqlToRelConverter#ensureSqlType|#ensureSqlType]([https://github.com/apache/calcite/blob/fb15511e76c660cbd440578421645ebe63941bf7/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1777-L1779])
is called
*** which skips the conversion for `CHAR` / `VARCHAR`
* the `=` filter goes thru the "regular" rex conversion - which involves
calling `rexBuilder#ensureType`
* the filter condition contains `ename = 'Sebastian'` twice; however the types
differ
* `RelFieldTrimmer` start a change cycle ; which induces the simplification of
the filter condition
* `RexSimplify` is executed with predicate elimination disabled (this will be
important)
* simplification compares the two literals with
[equals]([https://github.com/apache/calcite/blob/fb15511e76c660cbd440578421645ebe63941bf7/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1685])
and returns `false`
workarounds:
* disable the conversion by setting `inSubqueryThreshold` to `1`
* run a rule which executes `RexSimplify` with predicate elimination enabled
earlier than the trimmer (ex: `ReduceExpressionsRule`)
** I think this bug remained hidden because this might happen easily
testcase for `RelOptRulesTest`
{code:java}
@Test void testIncorrectInType() {
final String sql = "select ename from emp "
+ " where ename in ( 'Sebastian' ) and ename = 'Sebastian' and deptno
< 100";
sql(sql)
.withTrim(true)
.withRule()
.checkUnchanged();
}
{code}
results in plan
{code:java}
LogicalProject(ENAME=[$0])
LogicalValues(tuples=[[]])
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6020) SqlToRelConverter should not replace windowed SUM with equivalent expression using SUM0
Zoltan Haindrich created CALCITE-6020:
-
Summary: SqlToRelConverter should not replace windowed SUM with
equivalent expression using SUM0
Key: CALCITE-6020
URL: https://issues.apache.org/jira/browse/CALCITE-6020
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
{{SqlToRelConverter}} replaces {{SUM}} with {{SUM0}} around
[here|https://github.com/apache/calcite/blob/e1991e08a225ef08c2402ab35c310d88fff3c222/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5885]
This might have been needed at some point in the past - but I think it will be
better to leave it as {{SUM}} - as in case there is no {{SUM0}} in the system
that will be replaced with a {{COALESCE(SUM(...) , 0 )}} to provide it - as see
[here|https://github.com/apache/calcite/blob/e1991e08a225ef08c2402ab35c310d88fff3c222/core/src/test/java/org/apache/calcite/rel/rel2sql/RelToSqlConverterTest.java#L1288]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
Zoltan Haindrich created CALCITE-5953:
-
Summary: AggregateCaseToFilterRule may make inaccurate SUM
transformation
Key: CALCITE-5953
URL: https://issues.apache.org/jira/browse/CALCITE-5953
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
consider: `sum(case when x = 1 then 2 else 0 end) as b`
notice that this expression may only be null if there are no rows in the table
`AggregateCaseToFilterRule` rewrites the above expression to `sum(1) filter
(where x=2)` which broadens when it could be `null` to when there are no
matches to the filter
* `A` is `0` correctly in this case; but I think it will be still `0` in case
there are 0 input rows
* The result for `B` supposed to be `0` but since there are no matches by the
filter it becomes `null`
* `C` is not touched
```
# Convert CASE to FILTER without matches
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x);
+---+---++
| A | B | C |
+---+---++
| 0 | 0 | -4 |
+---+---++
(1 row)
!ok
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
B=[$t1], C=[$t2])
EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3],
C=[SUM($0)])
EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
!plan
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-4006) Add ordered-set aggregate functions
Zoltan Haindrich created CALCITE-4006: - Summary: Add ordered-set aggregate functions Key: CALCITE-4006 URL: https://issues.apache.org/jira/browse/CALCITE-4006 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich * mode * percentile_cont * percentile_disc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3887) Filter and Join conditions may not need to retain nullability during simplifications
Zoltan Haindrich created CALCITE-3887: - Summary: Filter and Join conditions may not need to retain nullability during simplifications Key: CALCITE-3887 URL: https://issues.apache.org/jira/browse/CALCITE-3887 Project: Calcite Issue Type: Improvement Components: core Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Reported by [~icshuo]: that there are situation in which join conditions are placed inside a redundant `CAST`. https://mail-archives.apache.org/mod_mbox/calcite-dev/202003.mbox/%3CCABYySnvtrMFWaBqM7_FgKedXELdXeCqciRJV7G4MyYN07HUOnQ%40mail.gmail.com%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3215) Simplification may not fully simplify IsNotNull expressions
Zoltan Haindrich created CALCITE-3215:
-
Summary: Simplification may not fully simplify IsNotNull
expressions
Key: CALCITE-3215
URL: https://issues.apache.org/jira/browse/CALCITE-3215
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
CALCITE-2929 have added a safety check to avoid simplifying problematic cases.
The safety check apparently misses some kinds, for example: {{UNARY_PLUS}}
{code}
@Test public void testIsNullSimplificationWithUnaryPlus() {
RexNode expr =
isNotNull(coalesce(unaryPlus(vInt(1)), vIntNotNull(0)));
RexNode s = simplify.simplifyUnknownAs(expr, RexUnknownAs.UNKNOWN);
assertThat(expr.isAlwaysTrue(), is(true));
assertThat(s, is(trueLiteral));
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (CALCITE-2929) Simplification of IS NULL checks are incorrectly assuming that CAST-s are possible
Zoltan Haindrich created CALCITE-2929:
-
Summary: Simplification of IS NULL checks are incorrectly assuming
that CAST-s are possible
Key: CALCITE-2929
URL: https://issues.apache.org/jira/browse/CALCITE-2929
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Example:
{code}
with ax(s) as (values ('xxx'),(cast(null as character varying)))
select cast(s as int) IS NULL from ax;
{code}
returns a result set; which evaluates
however 'xxx' is not a valid integer; so an error should be recieved instead
Another class of almost the same issue:
{code}
select cast('xxx' as int) IS NULL;
{code}
is also problematic; in that case the not nullability is deduced from the fact
that the literal's type is not nullable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2886) Simplification of AND expressions should push negations earlier
Zoltan Haindrich created CALCITE-2886: - Summary: Simplification of AND expressions should push negations earlier Key: CALCITE-2886 URL: https://issues.apache.org/jira/browse/CALCITE-2886 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich * currently simplification decomposes the AND operands * simplifies all operands * runs main simplification * (re) simplifies all operands which are negated Idea is to make the recursion as a first step; which could also push negation further down; and skip the simplification at end completely - as its not connected to a structural change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2851) Simplification: track visited nodes in paranoid mode
Zoltan Haindrich created CALCITE-2851: - Summary: Simplification: track visited nodes in paranoid mode Key: CALCITE-2851 URL: https://issues.apache.org/jira/browse/CALCITE-2851 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Idea is to have an object which could track all the visited nodes, goal is to identify if we have rules which are about to re-visit a previously visited node again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2848) Simplifying a case statement's first branch should disregard its safety
Zoltan Haindrich created CALCITE-2848:
-
Summary: Simplifying a case statement's first branch should
disregard its safety
Key: CALCITE-2848
URL: https://issues.apache.org/jira/browse/CALCITE-2848
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Example: a very simple CASE which is not get rewritten because the first
condition branch contains an unknown UDF
{code:sql}
CASE
WHEN to_date('2001-11-11') = '2000' THEN true
ELSE false
END
{code}
This expression right now retained as is; because the "to_date" was an unknown
UDF; but since the first branch of a CASE is always evaluated and that could
unlock the rewrite of this case to it's boolean form.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2842) Computing digest of IN expressions leads to Exceptions
Zoltan Haindrich created CALCITE-2842:
-
Summary: Computing digest of IN expressions leads to Exceptions
Key: CALCITE-2842
URL: https://issues.apache.org/jira/browse/CALCITE-2842
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Julian Hyde
Testcase:
{code}
/* RexProgramTest */
@Test public void testInDigest() {
RexNode e = in(vInt(), literal(1), literal(2));
assertThat(e.toString(), is("IN(?0.int0, 1, 2)"));
}
/* RexProgramBuilder */
protected RexNode in(Iterable nodes) {
return rexBuilder.makeCall(SqlStdOperatorTable.IN,
ImmutableList.copyOf(nodes));
}
{code}
The exception is due to that it tries to look at the other operand
https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/rex/RexCall.java#L115
[SIMPLE_BINARY_OPS|https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/rex/RexCall.java#L64]
contains
[COMPARISION|https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/sql/SqlKind.java#L1248]
which also contains "IN"
this was introduced in CALCITE-2454 a few days ago; bug not released
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2840) Simplification may use narrower UnknownAs modes during AND/OR simplification
Zoltan Haindrich created CALCITE-2840: - Summary: Simplification may use narrower UnknownAs modes during AND/OR simplification Key: CALCITE-2840 URL: https://issues.apache.org/jira/browse/CALCITE-2840 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Some parts of this was already working in 1.17 ; but 1.18 turned out to be a more conservative in this sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2841) Simplification: push negation into Case expression
Zoltan Haindrich created CALCITE-2841: - Summary: Simplification: push negation into Case expression Key: CALCITE-2841 URL: https://issues.apache.org/jira/browse/CALCITE-2841 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Julian Hyde -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2839) Simplify x = true to x and x = false to not x
Zoltan Haindrich created CALCITE-2839:
-
Summary: Simplify x = true to x and x = false to not x
Key: CALCITE-2839
URL: https://issues.apache.org/jira/browse/CALCITE-2839
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Julian Hyde
Testcase for RexProgramTest
{code}
@Test public void testEqTrue() {
checkSimplify3(eq(vBool(1), trueLiteral),
"?0.bool1",
"?0.bool1",
"?0.bool1");
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2838) Simplification: Remove redundant IS TRUE checks
Zoltan Haindrich created CALCITE-2838:
-
Summary: Simplification: Remove redundant IS TRUE checks
Key: CALCITE-2838
URL: https://issues.apache.org/jira/browse/CALCITE-2838
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
In case simplifcation is already processing in unknownAsFalse mode, {{expr IS
TRUE}} is redundant - and may just prevent further optimizations from happening:
{code}
@Test public void testRedundantIsTrue() {
// in case of unknownAsFalse
// x is TRUE <=> x
checkSimplify3(isTrue(vBool(1)),
"IS TRUE(?0.bool1)",
"?0.bool1",
"IS TRUE(?0.bool1)");
}
{code}
there are some further possibilities
{code}
(unknownAsTrue) x is NOT FALSE <=> x
(unknownAsTrue) x is FALSE <=> not x
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2802) Druid adapter: Usage of range conditions like "2010-01-01 < timestamp" leads to incorrect results
Zoltan Haindrich created CALCITE-2802:
-
Summary: Druid adapter: Usage of range conditions like "2010-01-01
< timestamp" leads to incorrect results
Key: CALCITE-2802
URL: https://issues.apache.org/jira/browse/CALCITE-2802
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Timestamp range conditions when the timestamp is on left hand side work
correctly; however when the literal is on the left hand side results are
missing.
{code}
@Test
public void testRangeCalc() {
final Fixture2 f = new Fixture2();
checkDateRange(f,
f.and(
f.le(f.timestampLiteral(2011, Calendar.JANUARY, 1), f.t),
f.le(f.t, f.timestampLiteral(2012, Calendar.FEBRUARY, 2))),
is("[2011-01-01T00:00:00.000Z/2012-02-02T00:00:00.001Z]"));
}
{code}
Fail:
{code}
java.lang.AssertionError:
Expected: is "[2011-01-01T00:00:00.000Z/2012-02-02T00:00:00.001Z]"
but: was "[1900-01-01T00:00:00.000Z/2011-01-01T00:00:00.001Z]"
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2731) Case statements are simplified with unsafe conditions
Zoltan Haindrich created CALCITE-2731:
-
Summary: Case statements are simplified with unsafe conditions
Key: CALCITE-2731
URL: https://issues.apache.org/jira/browse/CALCITE-2731
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
In CALCITE-1413 we were cautious to not simplify
{code}
CASE a=0 THEN 0 ELSE 1/a END
{code}
and it seemed good; however: {{RexProgramBuilder}} is also invokes simplify -
but the protection at that point is not able to work, because it sees only
references:
{code}
CASE $1 THEN $2 ELSE $3 END
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2695) Simplify casts which are only widening nullability
Zoltan Haindrich created CALCITE-2695:
-
Summary: Simplify casts which are only widening nullability
Key: CALCITE-2695
URL: https://issues.apache.org/jira/browse/CALCITE-2695
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
There are cases in which casts are added; but they are not neccessary.
{code}
@Test public void testSimplifyRedundantCast() {
RexNode expr = cast(isTrue(vBoolNotNull()), tBoolean(true));
assertThat(expr.getType().isNullable(), is(true));
RexNode result = simplify.simplifyUnknownAs(expr, RexUnknownAs.UNKNOWN);
assertThat(result.getType().isNullable(), is(false));
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2688) Improve diagnosability when return type could not be inferred.
Zoltan Haindrich created CALCITE-2688:
-
Summary: Improve diagnosability when return type could not be
inferred.
Key: CALCITE-2688
URL: https://issues.apache.org/jira/browse/CALCITE-2688
Project: Calcite
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Currently when the return type inferenence fails; a {{null}} is returned; while
leads to an NPE very quickly - the resulting Exception is not informative at
all.
{code}
@Test(expected = IllegalArgumentException.class)
public void checkNoCommonReturnTypeException() {
coalesce(vVarchar(1), vInt(2));
}
{code}
example NPE:
{code}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.rex.RexCall.(RexCall.java:59)
at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:242)
at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:254)
at
org.apache.calcite.test.RexProgramBuilderBase.coalesce(RexProgramBuilderBase.java:300)
at
org.apache.calcite.test.RexProgramTest.checkNoCommonReturnTypeException(RexProgramTest.java:753)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2687) Is distinct from could lead to Exceptions in ReduceExpressionRule
Zoltan Haindrich created CALCITE-2687:
-
Summary: Is distinct from could lead to Exceptions in
ReduceExpressionRule
Key: CALCITE-2687
URL: https://issues.apache.org/jira/browse/CALCITE-2687
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Test:
{code}
@Test
public void testReduceConstants4() throws Exception {
HepProgram program = new HepProgramBuilder()
.addRuleInstance(ReduceExpressionsRule.PROJECT_INSTANCE)
.addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
.addRuleInstance(ReduceExpressionsRule.JOIN_INSTANCE)
.build();
String sql = "select e.mgr is not distinct from f.mgr from emp e join emp f
on (e.mgr=f.mgr) where e.mgr is null";
sql(sql).with(program)
.check();
}
{code}
Unfortunately the exception doesn't have a stacktrace by default; just some
type:
{code}
org.apache.calcite.adapter.enumerable.RexToLixTranslator$AlwaysNull
{code}
Translation is not possible at this point:
https://github.com/apache/calcite/blob/d32ee5c320938b5c34ce09df2276c9570c27a301/core/src/main/java/org/apache/calcite/adapter/enumerable/RexToLixTranslator.java#L746
the underlying rexNode is:
{code}
CAST(null):INTEGER NOT NULL
{code}
which is problematic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2685) Bridge cases in which Calcite's type resolution is more stricter than Hive.
Zoltan Haindrich created CALCITE-2685:
-
Summary: Bridge cases in which Calcite's type resolution is more
stricter than Hive.
Key: CALCITE-2685
URL: https://issues.apache.org/jira/browse/CALCITE-2685
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Calcite is more stricter w.r.t common types than Hive.
For example in case a CASE with different types on the branches is not
something Calcite likes:
{code}
CASE
WHEN cond1 THEN booleanCol
WHEN cond2 THEN stringCol
WHEN cond3 THEN floatCol
ELSE doubleCol
END
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2675) ReduceExpressionRule may leave behind altered types w.r.t nullability
Zoltan Haindrich created CALCITE-2675:
-
Summary: ReduceExpressionRule may leave behind altered types w.r.t
nullability
Key: CALCITE-2675
URL: https://issues.apache.org/jira/browse/CALCITE-2675
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
If a simplification could happen after some [ReduceExpression
rewrite|https://github.com/apache/calcite/blob/fcc8bf7f44f92efb3c9a1e1f51ffc1a09cab27b9/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java#L794];
the simplification result may have a slightly different type in nullability.
{code}
@Test public void testReduceCaseNullabilityChange() throws Exception {
HepProgram program = new HepProgramBuilder()
.addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
.addRuleInstance(ReduceExpressionsRule.PROJECT_INSTANCE)
.build();
try (Hook.Closeable a =
Hook.REL_BUILDER_SIMPLIFY.add(Hook.propertyJ(false))) {
checkPlanning(program,
"select case when empno = 1 then 1 when 1 IS NOT NULL then 2 else
null end as qx "
+ "from emp");
}
{code}
Exposed by CALCITE-1413 changes; I'm not sure if there is any other variations
for which the same could happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2652) Fix Rel2Sql conversion in case the join condition contains a boolean column reference
Zoltan Haindrich created CALCITE-2652:
-
Summary: Fix Rel2Sql conversion in case the join condition
contains a boolean column reference
Key: CALCITE-2652
URL: https://issues.apache.org/jira/browse/CALCITE-2652
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
If the join condition contains a boolean column reference in an {{AND}} it
leads to an exception.
Reproducible with the following testcase - by adding it to RelToSqlConvertTest
{code}
@Test
public void testBooleanColInOn() {
final String sql = "SELECT 1 from emps join emp on (emp.deptno
= emps.empno and manager)";
final String expected = "...";
sql(sql).schema(CalciteAssert.SchemaSpec.POST).ok(expected);
}
{code}
exception is:
{code}
java.lang.RuntimeException: While invoking method 'public
org.apache.calcite.rel.rel2sql.SqlImplementor$Result
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(org.apache.calcite.rel.core.Project)'
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.toSql(RelToSqlConverterTest.java:136)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.access$1(RelToSqlConverterTest.java:134)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest$Sql.exec(RelToSqlConverterTest.java:2893)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest$Sql.ok(RelToSqlConverterTest.java:2868)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.testBooleanColInOn(RelToSqlConverterTest.java:998)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
... 30 more
Caused by: java.lang.RuntimeException: While invoking method 'public
org.apache.calcite.rel.rel2sql.SqlImplementor$Result
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(org.apache.calcite.rel.core.Join)'
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103)
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109)
at
[jira] [Created] (CALCITE-2632) Add hashCode and equals implementations to RexNode
Zoltan Haindrich created CALCITE-2632:
-
Summary: Add hashCode and equals implementations to RexNode
Key: CALCITE-2632
URL: https://issues.apache.org/jira/browse/CALCITE-2632
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
Right now RexNode doesn't have any equals or hashCode functions; which makes it
rely on the default implementation.
But when we are writing simplification logics we sometimes forget to use
{{toString()}} during comparisions and may try to rely on pure equals:
* there is a [Set of
RexNode-s|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1104]
during {{AND}} simplification and in [RexUtil as
well|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L321]
* I've by mistake just written rexNode.equals(otherRexNode) during the
implementation of CALCITE-1413
* I've just bumped into the same thing...that
[RexUtil.andNot|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L1888]
is also rely on itand I think those comparisions go back a while (~3years
at least) ; and a bug is not appeared from it because this comparision is in
most cases false.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2631) Address small issues in case simplification
Zoltan Haindrich created CALCITE-2631: - Summary: Address small issues in case simplification Key: CALCITE-2631 URL: https://issues.apache.org/jira/browse/CALCITE-2631 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich followup of CALCITE-1413 https://github.com/apache/calcite/commit/b470a0cd4572c9f6c4c0e9b51926b97c5af58d3f#comments to address the following things: * fuse branch removal logics with case branch simplification * postpone condition simplification during branch compaction removal to avoid re-simplification of the same subtree if multiple branches are removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2611) Code generation fails if one side of an or contains unknown
Zoltan Haindrich created CALCITE-2611:
-
Summary: Code generation fails if one side of an or contains
unknown
Key: CALCITE-2611
URL: https://issues.apache.org/jira/browse/CALCITE-2611
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Julian Hyde
This is rather strange...it needs also an and below which contains the unknown.
Running the following query:
{code}
!connect jdbc:calcite:model=target/test-classes/model.json admin admin
select (EMPID = 3 and unknown) or ( EMPID = 3) from emps;
{code}
results in a compilation exception;
the main problem is that there is an interesting field:
{code}
static final Object $L4J$C$_null = !null;
{code}
full exception:
{code}
0: jdbc:calcite:model=target/test-classes/mod> select (cast(EMPID as integer) =
3 and unknown) or not ( cast(EMPID as integer) = 3) from emps;
Error: Error while executing SQL "select (cast(EMPID as integer) = 3 and
unknown) or not ( cast(EMPID as integer) = 3) from emps": Error while compiling
generated Java code:
org.apache.calcite.DataContext root;
public org.apache.calcite.linq4j.Enumerable bind(final
org.apache.calcite.DataContext root0) {
root = root0;
final org.apache.calcite.rel.RelNode v1stashed =
(org.apache.calcite.rel.RelNode) root.get("v1stashed");
final org.apache.calcite.interpreter.Interpreter interpreter = new
org.apache.calcite.interpreter.Interpreter(
root,
v1stashed);
return new org.apache.calcite.linq4j.AbstractEnumerable(){
public org.apache.calcite.linq4j.Enumerator enumerator() {
return new org.apache.calcite.linq4j.Enumerator(){
public final org.apache.calcite.linq4j.Enumerator inputEnumerator =
interpreter.enumerator();
public void reset() {
inputEnumerator.reset();
}
public boolean moveNext() {
return inputEnumerator.moveNext();
}
public void close() {
inputEnumerator.close();
}
public Object current() {
final Integer inp5_ = (Integer) ((Object[])
inputEnumerator.current())[5];
return inp5_ != null && inp5_.intValue() != 3 ? Boolean.TRUE :
inp5_ != null || $L4J$C$_null || inp5_ == null ? (Boolean) null : Boolean.FALSE;
}
static final Object $L4J$C$_null = !null;
};
}
};
}
public Class getElementType() {
return java.lang.Boolean.class;
} (state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2606) Return type inference for and/or may identify result type incorrectly
Zoltan Haindrich created CALCITE-2606:
-
Summary: Return type inference for and/or may identify result type
incorrectly
Key: CALCITE-2606
URL: https://issues.apache.org/jira/browse/CALCITE-2606
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
In case in an AND expression a null constant appears earlier than a nullable
boolean; the return type is set to NULL instead of boolean.
https://github.com/apache/calcite/blob/c39bfaa02a06ac91575076a6e74f29863923f5eb/core/src/main/java/org/apache/calcite/sql/type/ReturnTypes.java#L198
The problem can be reproduced; however I'm not sure if it would happen
naturally or not - I've discovered it during enhancing case simplification.
{code}
@Test
public void testUsageOfConstantNull() {
RexLiteral constantNull = rexBuilder.constantNull();
RexNode node1 = or(constantNull, vBool());
assertThat(node1.getType(), is(vBool().getType()));
RexNode node2 = or(vBoolNotNull(), constantNull);
assertThat(node2.getType(), is(vBool().getType()));
RexNode node3 = or(constantNull, constantNull);
assertThat(node3.getType(), is(constantNull.getType()));
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2445) RexSimplify x=x
Zoltan Haindrich created CALCITE-2445: - Summary: RexSimplify x=x Key: CALCITE-2445 URL: https://issues.apache.org/jira/browse/CALCITE-2445 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Julian Hyde -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2444) Rel2Sql: support c IN (1,2)
Zoltan Haindrich created CALCITE-2444:
-
Summary: Rel2Sql: support c IN (1,2)
Key: CALCITE-2444
URL: https://issues.apache.org/jira/browse/CALCITE-2444
Project: Calcite
Issue Type: Bug
Components: jdbc-adapter
Affects Versions: 1.17.0
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
It is not possible to convert an Rel tree into an sql string if it contains a
filter which has an IN:
example: `select * from emp where empno in (10,20)`
note: Calcite's sql parser translates INs into either ORs or into a subquery.
The following testcase can be added to RelToSqlConverterTest to reproduce the
issue:
{code}
@Test public void testHiveIn() {
// this can't be tested using "sql" because Calcite's sql parser replaces
INs with ORs or subqueries.
final RelBuilder builder =
RelBuilder.create(RelBuilderTest.config().build());
RexBuilder rexBuilder = builder.getRexBuilder();
RelNode root =
builder.scan("EMP")
.filter(rexBuilder.makeCall(SqlStdOperatorTable.IN,
builder.field("DEPTNO"),
builder.literal(20), builder.literal(21)))
.build();
RelNode rel = root;
SqlDialect dialect = SqlDialect.DatabaseProduct.HIVE.getDialect();
final RelToSqlConverter converter = new RelToSqlConverter(dialect);
final SqlNode sqlNode = converter.visitChild(0, rel).asStatement();
String sqlStr = sqlNode.toSqlString(dialect).getSql();
assertEquals("select * from emp where deptno in (10,20)", sqlStr);
}
{code}
The exception is raised because calcite expects that every IN is a
subquery(because of sql2rel rewrites them)
{code}
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_171]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_171]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:173)
~[calcite-core-1.17.0.jar:1.17.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_171]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_171]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.17.0.jar:1.17.0]
... 61 more
Caused by: java.lang.ClassCastException: org.apache.calcite.rex.RexCall cannot
be cast to org.apache.calcite.rex.RexSubQuery
at
org.apache.calcite.rel.rel2sql.SqlImplementor$Context.toSql(SqlImplementor.java:540)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:166)
~[calcite-core-1.17.0.jar:1.17.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_171]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_171]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109)
~[calcite-core-1.17.0.jar:1.17.0]
at
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:173)
~[calcite-core-1.17.0.jar:1.17.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_171]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_171]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at
[jira] [Created] (CALCITE-2401) Improve RelMdPredicates performance
Zoltan Haindrich created CALCITE-2401: - Summary: Improve RelMdPredicates performance Key: CALCITE-2401 URL: https://issues.apache.org/jira/browse/CALCITE-2401 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich In CALCITE-2384 a simplify feature was disabled - to prevent performance degradation of RelMdPredicates I think that the algorithm which tries out all the combination of the variables could be probably improved to while it's doing the same - it doesn't run RexSimplify that many times. for more details see my comment in the other ticket [here|https://issues.apache.org/jira/browse/CALCITE-2384?focusedCommentId=16530021=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16530021] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2326) Generalize p(x) IS TRUE/FALSE/UNKNOWN handling in RexSimplify
Zoltan Haindrich created CALCITE-2326: - Summary: Generalize p(x) IS TRUE/FALSE/UNKNOWN handling in RexSimplify Key: CALCITE-2326 URL: https://issues.apache.org/jira/browse/CALCITE-2326 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Currently only IS TRUE is handle by the unknownAsFalse field variable. The main goal would be to extend the logic to also handle IS FALSE cases (and possibly IS UNKNOWN too) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2240) Predicate push into CASE statment doesn't happen in some cases
Zoltan Haindrich created CALCITE-2240:
-
Summary: Predicate push into CASE statment doesn't happen in some
cases
Key: CALCITE-2240
URL: https://issues.apache.org/jira/browse/CALCITE-2240
Project: Calcite
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Julian Hyde
[CaseShuttle|https://github.com/apache/calcite/blob/8139acb78b100483fbafb078fb82d07f921abef0/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java#L562]
may make the optimization but the functions return value doesn't depend on
that.
For the following query the =1 is not pushed into the case.
{code}
select empno from emp where case when sal > 1000 then empno else sal end = 1
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
