[jira] [Created] (CALCITE-2134) AggregateJoinTransposeRule fails in optimize phase based on CBO because of the bug in RelMdColumnUniqueness
jingzhang created CALCITE-2134:
--
Summary: AggregateJoinTransposeRule fails in optimize phase based
on CBO because of the bug in RelMdColumnUniqueness
Key: CALCITE-2134
URL: https://issues.apache.org/jira/browse/CALCITE-2134
Project: Calcite
Issue Type: Bug
Reporter: jingzhang
Assignee: Julian Hyde
Attachments: 02-snippet.sql, 02.sql
If runs tpch 02 query(in attachment) or snippet of 02(in attachment), the
following exception will be thrown in AggregateJoinTranspose Rule during
optimized phase based on VolcanoPlanner.
{code}
java.lang.AssertionError
at
org.apache.calcite.sql.SqlSplittableAggFunction$SelfSplitter.topSplit(SqlSplittableAggFunction.java:209)
at
org.apache.calcite.rel.rules.AggregateJoinTransposeRule.onMatch(AggregateJoinTransposeRule.java:310)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
{code}
The failure is caused by the RelMdColumnUniqueness's estimation of RelSubset
type RelNodes, which does not take Join into consideration. I don't figure out
why only includes Aggregate, Filter, Values, TableScan, Project yet.
{code}
public Boolean areColumnsUnique(RelSubset rel, RelMetadataQuery mq,
ImmutableBitSet columns, boolean ignoreNulls) {
int nullCount = 0;
for (RelNode rel2 : rel.getRels()) {
if (rel2 instanceof Aggregate
|| rel2 instanceof Filter
|| rel2 instanceof Values
|| rel2 instanceof TableScan
|| simplyProjects(rel2, columns)) {
try {
final Boolean unique = mq.areColumnsUnique(rel2, columns,
ignoreNulls);
if (unique != null) {
if (unique) {
return true;
}
} else {
++nullCount;
}
} catch (CyclicMetadataException e) {
// Ignore this relational expression; there will be non-cyclic ones
// in this set.
}
}
}
return nullCount == 0 ? false : null;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2133) Allow SqlGroupedWindowFunction to specify returnTypeInference in its constructor
[ https://issues.apache.org/jira/browse/CALCITE-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324916#comment-16324916 ] Shuyi Chen commented on CALCITE-2133: - Hi [~julianhyde], please let me know what you think. Here is the [PR|https://github.com/apache/calcite/pull/606]. > Allow SqlGroupedWindowFunction to specify returnTypeInference in its > constructor > > > Key: CALCITE-2133 > URL: https://issues.apache.org/jira/browse/CALCITE-2133 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.16.0 >Reporter: Shuyi Chen >Assignee: Julian Hyde > Fix For: 1.16.0 > > > This is follow-up of > [Calcite-1867|https://issues.apache.org/jira/browse/CALCITE-1867], we need > toy add a new constructor to allow Flink to pass in the returnTypeInference > parameter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2133) Allow SqlGroupedWindowFunction to specify returnTypeInference in its constructor
Shuyi Chen created CALCITE-2133: --- Summary: Allow SqlGroupedWindowFunction to specify returnTypeInference in its constructor Key: CALCITE-2133 URL: https://issues.apache.org/jira/browse/CALCITE-2133 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.16.0 Reporter: Shuyi Chen Assignee: Julian Hyde Fix For: 1.16.0 This is follow-up of [Calcite-1867|https://issues.apache.org/jira/browse/CALCITE-1867], we need toy add a new constructor to allow Flink to pass in the returnTypeInference parameter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:55 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
{quote}Test case for https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
NullPointerException{{monospaced text}} when extract is called on a NULL date
field.
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995, 1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\", \"birth_date\"
limit 1")
.returns("");
}
{quote}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
Test case for https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
NullPointerException{{monospaced text}} when extract is called on a NULL date
field.
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995, 1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\", \"birth_date\"
limit 1")
.returns("");
bq. }
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:54 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
Test case for https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
NullPointerException{{monospaced text}} when extract is called on a NULL date
field.
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995, 1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\", \"birth_date\"
limit 1")
.returns("");
bq. }
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
bq. Test case for
bq.https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
bq.NullPointerException{{monospaced text}} when extract is called on a NULL
date field.
bq. @Test public void testExtractOnNullDateField(){
bq. CalciteAssert.that()
bq. .with(CalciteAssert.Config.FOODMART_CLONE)
bq. .query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
bq. "from \"foodmart\".\"employee\"\n" +
bq. "where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
bq. "group by extract(year from \"end_date\"),
\"hire_date\", \"birth_date\" limit 1")
bq. .returns("");
bq. }
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:53 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
bq. Test case for
bq.https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
bq.NullPointerException{{monospaced text}} when extract is called on a NULL
date field.
bq. @Test public void testExtractOnNullDateField(){
bq. CalciteAssert.that()
bq. .with(CalciteAssert.Config.FOODMART_CLONE)
bq. .query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
bq. "from \"foodmart\".\"employee\"\n" +
bq. "where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
bq. "group by extract(year from \"end_date\"),
\"hire_date\", \"birth_date\" limit 1")
bq. .returns("");
bq. }
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
Test case for
https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
NullPointerException{{monospaced text}} when extract is called on a NULL
date field.
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:52 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
Test case for
https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
NullPointerException{{monospaced text}} when extract is called on a NULL
date field.
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
{{
/** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException{{monospaced text}} when extract is called on a NULL
date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}
}}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:51 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
{{monospaced text}}
/** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException{{monospaced text}} when extract is called on a NULL
date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}{quote}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
{{monospaced text}}
/** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException{{monospaced text}} when extract is called on a NULL
date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}{quote}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando edited comment on CALCITE-1188 at 1/12/18 10:51 PM:
-
I have tried to reproduce this bug but I could not, here is my attempt:
{{
/** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException{{monospaced text}} when extract is called on a NULL
date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}
}}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
was (Author: asolimando):
I have tried to reproduce this bug but I could not, here is my attempt:
{{monospaced text}}
/** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException{{monospaced text}} when extract is called on a NULL
date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}{quote}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1188) NullPointerException in EXTRACT with WHERE ... IN clause if field has null value
[
https://issues.apache.org/jira/browse/CALCITE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324688#comment-16324688
]
Alessandro Solimando commented on CALCITE-1188:
---
I have tried to reproduce this bug but I could not, here is my attempt:
{quote} /** Test case for
* https://issues.apache.org/jira/browse/CALCITE-1188;>[CALCITE-1188]
* NullPointerException when extract is called on a NULL date field. */
@Test public void testExtractOnNullDateField(){
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("select extract (year from \"end_date\"), \"hire_date\",
\"birth_date\"\n" +
"from \"foodmart\".\"employee\"\n" +
"where extract (year from \"end_date\") in (1994, 1995,
1996)\n" +
"group by extract(year from \"end_date\"), \"hire_date\",
\"birth_date\" limit 1")
.returns("");
}{quote}
All the suggested ingredients are there:
# number of items in WHERE-IN clause must be equal or more than two
# null value for _end_date_ column used in the extract (both in the select and
where clauses)
Any suggestion?
> NullPointerException in EXTRACT with WHERE ... IN clause if field has null
> value
>
>
> Key: CALCITE-1188
> URL: https://issues.apache.org/jira/browse/CALCITE-1188
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Wei Hu
>Assignee: Julian Hyde
> Labels: newbie
>
> if there was a timestamp or date field F1, the queries below will throw
> nullpointer exception if F1 has null value
> select extract(year from F1) from T where extract(year from F1) in (2004,
> 2005)
> select extract(year from F1) , count(0) from T where extract(year from F1) in
> (2004, 2005) group by extract(year from F1)
> the number of items in WHERE-IN clause must be equal or more than two
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2107) Timezone not passed as part of granularity when passing TimeExtractionFunction to Druid
[
https://issues.apache.org/jira/browse/CALCITE-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324596#comment-16324596
]
Nishant Bangarwa commented on CALCITE-2107:
---
Sure, granularity in druid queries determines how data gets bucketed across the
time dimension. http://druid.io/docs/latest/querying/granularities.html has
details on how druid uses it and different available granularities in druid.
Here in this particular case, if the user has set the timezone in calcite
connection config, we want the bucketing for time extraction functions to
happen as per the user specified timezone and hence need druid granularity to
align with the calcite timezone.
Before this patch we were using Simple Granularities string values (e.g
year,month) to Druid which always bucket time as per UTC timezone.
With this patch granularity is an object having corresponding period and
timezone which is then passed as part of druid query. This is needed for the
time bucketing done by druid to align with calcite/hive time bucketing.
Corresponding implementation in Druid -
* Period Granularity
https://github.com/druid-io/druid/blob/master/java-util/src/main/java/io/druid/java/util/common/granularity/PeriodGranularity.java
* Simple Granularities
https://github.com/druid-io/druid/blob/master/java-util/src/main/java/io/druid/java/util/common/granularity/Granularities.java
> Timezone not passed as part of granularity when passing
> TimeExtractionFunction to Druid
> ---
>
> Key: CALCITE-2107
> URL: https://issues.apache.org/jira/browse/CALCITE-2107
> Project: Calcite
> Issue Type: Bug
> Components: druid
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>
> When sending granularity to TimeFormatExtractionFunction we need to add
> timezone to the granularity to align it with the timezone specified in
> calcite connection. If no timezone is specified in granularity, It is
> considered as UTC timezone by druid.
> Note the return value in is as per granularity UTC and not IST as expected -
> {code}
> @Test
> public void testTmeWithFilterOnFloorOnTimeWithTimezoneConversion() {
> final String sql = "Select cast(floor(\"timestamp\" to MONTH) as
> timestamp) as t from "
> + "\"foodmart\" where floor(\"timestamp\" to MONTH) >= '1997-04-30
> 18:30:00 UTC' order by t"
> + " limit 1";
> final String druidQueryPart1 =
> "filter\":{\"type\":\"bound\",\"dimension\":\"__time\","
> + "\"lower\":\"1997-05-01T00:00:00.000Z\",\"lowerStrict\":false,"
> +
> "\"ordering\":\"lexicographic\",\"extractionFn\":{\"type\":\"timeFormat\","
> + "\"format\":\"-MM-dd";
> final String druidQueryPart2 =
> "\"granularity\":\"month\",\"timeZone\":\"IST\","
> +
> "\"locale\":\"en-US\"}},\"dimensions\":[],\"metrics\":[],\"granularity\":\"all\"";
> CalciteAssert.that()
> .enable(enabled())
> .with(ImmutableMap.of("model", FOODMART.getPath()))
> .with(CalciteConnectionProperty.TIME_ZONE.camelName(), "IST")
> .query(sql)
> .runs()
> .queryContains(druidChecker(druidQueryPart1, druidQueryPart2))
> .returnsOrdered("T=1997-05-01 05:30:00");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CALCITE-1054) NPE caused by wrong code generation for Timestamp fields
[
https://issues.apache.org/jira/browse/CALCITE-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324587#comment-16324587
]
Maryann Xue edited comment on CALCITE-1054 at 1/12/18 9:45 PM:
---
After verifying CALCITE-1118 along with CALCITE-1427 and this issue, I found
that the issue also exists with Date/Time related functions. These functions
are implemented by {{NotNullImplementor}} which will use their operands in
primitive forms directly. So the solution should be unboxing the operands
beforehand and meanwhile make sure the unboxed Parameters are handled properly,
which means to call {{RexToLixTranslator.handleNull()}} on the operands when
necessary. I've made extra check-ins in the PR to reflect this change.
was (Author: maryannxue):
After verifying CALCITE-1118 along with CALCITE-1427 and this issue, I found
that the issue also exists with Date/Time related functions. These functions
are implemented by {{NotNullImplementor}}s which will use their operands in
primitive forms directly. So the solution should be unboxing the operands
beforehand and meanwhile make sure the unboxed Parameters are handled properly,
which means to call {{RexToLixTranslator.handleNull()}} on the operands when
necessary. I've made extra check-ins in the PR to reflect this change.
> NPE caused by wrong code generation for Timestamp fields
>
>
> Key: CALCITE-1054
> URL: https://issues.apache.org/jira/browse/CALCITE-1054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.6.0, 1.5.0, 1.4.0-incubating
>Reporter: Frankie Bollaert
>Assignee: Julian Hyde
>Priority: Minor
>
> Problem occurs when:
> - Execute a query containing 2 checks on a Timestamp field
> - Table contains records which have NULL values for this field
> Example query:
> {code}
> select * from aTable where aTimestamp > timestamp '2015-1-1 00:00:00' and
> aTimestamp < timestamp '2015-2-1 00:00:00';
> {code}
> {code}
> /* 48 */ public boolean moveNext() {
> /* 49 */ while (inputEnumerator.moveNext()) {
> /* 50 */ final java.sql.Timestamp inp23_ = (java.sql.Timestamp)
> ((Object[]) inputEnumerator.current())[23];
> /* 51 */ final long v =
> org.apache.calcite.runtime.SqlFunctions.toLong(inp23_);
> /* 52 */ if (inp23_ != null && v > 142007040L && (inp23_ != null
> && v < 142274880L)) {
> /* 53 */ return true;
> /* 54 */ }
> /* 55 */ }
> /* 56 */ return false;
> /* 57 */ }
> {code}
> Stack trace snippet
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1094)
> at
> org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1089)
> at Baz$1$1.moveNext(ANONYMOUS.java:51)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:677)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:103)
> {code}
> The generated code also looks wrong for date fields.
> {code}
> /* 15 */ public boolean moveNext() {
> /* 16 */ while (inputEnumerator.moveNext()) {
> /* 17 */ final java.sql.Date current = (java.sql.Date)
> inputEnumerator.current();
> /* 18 */ final int v =
> org.apache.calcite.runtime.SqlFunctions.toInt(current);
> /* 19 */ if (current != null && v > 2780 && (current != null && v <
> 5290)) {
> /* 20 */ return true;
> /* 21 */ }
> /* 22 */ }
> /* 23 */ return false;
> /* 24 */ }
> {code}
> \\
> Other types of fields do not have this problem. Below is what the generated
> code looks like in the case of a String field. *On line 20 there is a null
> check.* This is the type of check that needs to be generated for Timestamp
> fields as well.
> {code}
> select empno from sales.emps where gender > 'A' and gender < 'Z';
> {code}
> {code}
> /* 17 */ public boolean moveNext() {
> /* 18 */while (inputEnumerator.moveNext()) {
> /* 19 */ final Object[] current = (Object[]) inputEnumerator.current();
> /* 20 */ final String inp3_ = current[3] == null ? (String) null :
> current[3].toString();
> /* 21 */ if (inp3_ != null &&
> org.apache.calcite.runtime.SqlFunctions.gt(inp3_,
> $L4J$C$org_apache_calcite_runtime_SqlFunctions_rtrim_A_) && (inp3_ != null &&
> org.apache.calcite.runtime.SqlFunctions.lt(inp3_,
> $L4J$C$org_apache_calcite_runtime_SqlFunctions_rtrim_Z_))) {
> /* 22 */return true;
> /* 23 */ }
> /* 24 */}
> /* 25 */return false;
> /* 26 */ }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-1054) NPE caused by wrong code generation for Timestamp fields
[
https://issues.apache.org/jira/browse/CALCITE-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324587#comment-16324587
]
Maryann Xue commented on CALCITE-1054:
--
After verifying CALCITE-1118 along with CALCITE-1427 and this issue, I found
that the issue also exists with Date/Time related functions. These functions
are implemented by {{NotNullImplementor}}s which will use their operands in
primitive forms directly. So the solution should be unboxing the operands
beforehand and meanwhile make sure the unboxed Parameters are handled properly,
which means to call {{RexToLixTranslator.handleNull()}} on the operands when
necessary. I've made extra check-ins in the PR to reflect this change.
> NPE caused by wrong code generation for Timestamp fields
>
>
> Key: CALCITE-1054
> URL: https://issues.apache.org/jira/browse/CALCITE-1054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.6.0, 1.5.0, 1.4.0-incubating
>Reporter: Frankie Bollaert
>Assignee: Julian Hyde
>Priority: Minor
>
> Problem occurs when:
> - Execute a query containing 2 checks on a Timestamp field
> - Table contains records which have NULL values for this field
> Example query:
> {code}
> select * from aTable where aTimestamp > timestamp '2015-1-1 00:00:00' and
> aTimestamp < timestamp '2015-2-1 00:00:00';
> {code}
> {code}
> /* 48 */ public boolean moveNext() {
> /* 49 */ while (inputEnumerator.moveNext()) {
> /* 50 */ final java.sql.Timestamp inp23_ = (java.sql.Timestamp)
> ((Object[]) inputEnumerator.current())[23];
> /* 51 */ final long v =
> org.apache.calcite.runtime.SqlFunctions.toLong(inp23_);
> /* 52 */ if (inp23_ != null && v > 142007040L && (inp23_ != null
> && v < 142274880L)) {
> /* 53 */ return true;
> /* 54 */ }
> /* 55 */ }
> /* 56 */ return false;
> /* 57 */ }
> {code}
> Stack trace snippet
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1094)
> at
> org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1089)
> at Baz$1$1.moveNext(ANONYMOUS.java:51)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:677)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:103)
> {code}
> The generated code also looks wrong for date fields.
> {code}
> /* 15 */ public boolean moveNext() {
> /* 16 */ while (inputEnumerator.moveNext()) {
> /* 17 */ final java.sql.Date current = (java.sql.Date)
> inputEnumerator.current();
> /* 18 */ final int v =
> org.apache.calcite.runtime.SqlFunctions.toInt(current);
> /* 19 */ if (current != null && v > 2780 && (current != null && v <
> 5290)) {
> /* 20 */ return true;
> /* 21 */ }
> /* 22 */ }
> /* 23 */ return false;
> /* 24 */ }
> {code}
> \\
> Other types of fields do not have this problem. Below is what the generated
> code looks like in the case of a String field. *On line 20 there is a null
> check.* This is the type of check that needs to be generated for Timestamp
> fields as well.
> {code}
> select empno from sales.emps where gender > 'A' and gender < 'Z';
> {code}
> {code}
> /* 17 */ public boolean moveNext() {
> /* 18 */while (inputEnumerator.moveNext()) {
> /* 19 */ final Object[] current = (Object[]) inputEnumerator.current();
> /* 20 */ final String inp3_ = current[3] == null ? (String) null :
> current[3].toString();
> /* 21 */ if (inp3_ != null &&
> org.apache.calcite.runtime.SqlFunctions.gt(inp3_,
> $L4J$C$org_apache_calcite_runtime_SqlFunctions_rtrim_A_) && (inp3_ != null &&
> org.apache.calcite.runtime.SqlFunctions.lt(inp3_,
> $L4J$C$org_apache_calcite_runtime_SqlFunctions_rtrim_Z_))) {
> /* 22 */return true;
> /* 23 */ }
> /* 24 */}
> /* 25 */return false;
> /* 26 */ }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-508) Reading from ResultSet before calling next() should throw SQLException not NoSuchElementException
[ https://issues.apache.org/jira/browse/CALCITE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324575#comment-16324575 ] Julian Hyde commented on CALCITE-508: - Yes it looks as if the fix is in Avatica, and we'll need tests in Avatica and also to fix the existing test in Calcite's JdbcTest. This ticket will suffice for all. The fix in Avatica will have to be released, and Calcite upgraded to that release, before we can change JdbcTest. As for JIRA cases, this will suffice for all. Maybe there will be a "Upgrade Calcite to Avatica 1.11" case at some point. > Reading from ResultSet before calling next() should throw SQLException not > NoSuchElementException > - > > Key: CALCITE-508 > URL: https://issues.apache.org/jira/browse/CALCITE-508 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde > Labels: newbie > > Reading from ResultSet before calling next() should throw SQLException not > NoSuchElementException. > Each of the Cursor.Accessor.getXxx methods should convert runtime exceptions > to SQLException. > JdbcTest.testExtract currently demonstrates this problem; it passes if there > is a NoSuchElementException, but should look for a SQLException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2107) Timezone not passed as part of granularity when passing TimeExtractionFunction to Druid
[
https://issues.apache.org/jira/browse/CALCITE-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324573#comment-16324573
]
Julian Hyde commented on CALCITE-2107:
--
First of all, can you improve the comments. I don't know what "granularity"
means and reading the comments contained circular definitions. Next, please
explain why time zone belongs in the granularity object when it doesn't occur
in the granularity string.
> Timezone not passed as part of granularity when passing
> TimeExtractionFunction to Druid
> ---
>
> Key: CALCITE-2107
> URL: https://issues.apache.org/jira/browse/CALCITE-2107
> Project: Calcite
> Issue Type: Bug
> Components: druid
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>
> When sending granularity to TimeFormatExtractionFunction we need to add
> timezone to the granularity to align it with the timezone specified in
> calcite connection. If no timezone is specified in granularity, It is
> considered as UTC timezone by druid.
> Note the return value in is as per granularity UTC and not IST as expected -
> {code}
> @Test
> public void testTmeWithFilterOnFloorOnTimeWithTimezoneConversion() {
> final String sql = "Select cast(floor(\"timestamp\" to MONTH) as
> timestamp) as t from "
> + "\"foodmart\" where floor(\"timestamp\" to MONTH) >= '1997-04-30
> 18:30:00 UTC' order by t"
> + " limit 1";
> final String druidQueryPart1 =
> "filter\":{\"type\":\"bound\",\"dimension\":\"__time\","
> + "\"lower\":\"1997-05-01T00:00:00.000Z\",\"lowerStrict\":false,"
> +
> "\"ordering\":\"lexicographic\",\"extractionFn\":{\"type\":\"timeFormat\","
> + "\"format\":\"-MM-dd";
> final String druidQueryPart2 =
> "\"granularity\":\"month\",\"timeZone\":\"IST\","
> +
> "\"locale\":\"en-US\"}},\"dimensions\":[],\"metrics\":[],\"granularity\":\"all\"";
> CalciteAssert.that()
> .enable(enabled())
> .with(ImmutableMap.of("model", FOODMART.getPath()))
> .with(CalciteConnectionProperty.TIME_ZONE.camelName(), "IST")
> .query(sql)
> .runs()
> .queryContains(druidChecker(druidQueryPart1, druidQueryPart2))
> .returnsOrdered("T=1997-05-01 05:30:00");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-2107) Timezone not passed as part of granularity when passing TimeExtractionFunction to Druid
[
https://issues.apache.org/jira/browse/CALCITE-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324566#comment-16324566
]
Nishant Bangarwa commented on CALCITE-2107:
---
[~julianhyde] [~jcamachorodriguez] submitted a patch to allow passing timezone
as part of druid granularity.
https://github.com/apache/calcite/pull/605
Please review.
> Timezone not passed as part of granularity when passing
> TimeExtractionFunction to Druid
> ---
>
> Key: CALCITE-2107
> URL: https://issues.apache.org/jira/browse/CALCITE-2107
> Project: Calcite
> Issue Type: Bug
> Components: druid
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>
> When sending granularity to TimeFormatExtractionFunction we need to add
> timezone to the granularity to align it with the timezone specified in
> calcite connection. If no timezone is specified in granularity, It is
> considered as UTC timezone by druid.
> Note the return value in is as per granularity UTC and not IST as expected -
> {code}
> @Test
> public void testTmeWithFilterOnFloorOnTimeWithTimezoneConversion() {
> final String sql = "Select cast(floor(\"timestamp\" to MONTH) as
> timestamp) as t from "
> + "\"foodmart\" where floor(\"timestamp\" to MONTH) >= '1997-04-30
> 18:30:00 UTC' order by t"
> + " limit 1";
> final String druidQueryPart1 =
> "filter\":{\"type\":\"bound\",\"dimension\":\"__time\","
> + "\"lower\":\"1997-05-01T00:00:00.000Z\",\"lowerStrict\":false,"
> +
> "\"ordering\":\"lexicographic\",\"extractionFn\":{\"type\":\"timeFormat\","
> + "\"format\":\"-MM-dd";
> final String druidQueryPart2 =
> "\"granularity\":\"month\",\"timeZone\":\"IST\","
> +
> "\"locale\":\"en-US\"}},\"dimensions\":[],\"metrics\":[],\"granularity\":\"all\"";
> CalciteAssert.that()
> .enable(enabled())
> .with(ImmutableMap.of("model", FOODMART.getPath()))
> .with(CalciteConnectionProperty.TIME_ZONE.camelName(), "IST")
> .query(sql)
> .runs()
> .queryContains(druidChecker(druidQueryPart1, druidQueryPart2))
> .returnsOrdered("T=1997-05-01 05:30:00");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CALCITE-508) Reading from ResultSet before calling next() should throw SQLException not NoSuchElementException
[ https://issues.apache.org/jira/browse/CALCITE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324563#comment-16324563 ] Alessandro Solimando commented on CALCITE-508: -- Debugging _JdbcTest.testExtract_ I have seen that the concrete implementation of ResultSet is provided by Avatica here, and from the comment on converting runtime exceptions into _Cursor.Accessor.getXxx_ methods I assume that Avatica is the only access to JDBC drivers inside Calcite. Is it correct? I see that Avatica is now a separate repo (calcite-avatica), so I guess this ticket should be handled inside avatica-core and the only modification here should be to update the tests accordingly. Is that correct? > Reading from ResultSet before calling next() should throw SQLException not > NoSuchElementException > - > > Key: CALCITE-508 > URL: https://issues.apache.org/jira/browse/CALCITE-508 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde > Labels: newbie > > Reading from ResultSet before calling next() should throw SQLException not > NoSuchElementException. > Each of the Cursor.Accessor.getXxx methods should convert runtime exceptions > to SQLException. > JdbcTest.testExtract currently demonstrates this problem; it passes if there > is a NoSuchElementException, but should look for a SQLException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2088) More complex end2end tests in Calcite Plus module
[
https://issues.apache.org/jira/browse/CALCITE-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324029#comment-16324029
]
Piotr Bojko commented on CALCITE-2088:
--
Re 1 *done*
Re 3 - See CALCITE-2093 it was not about MacOS but Linux with PL locale instead
of US. If removing sparseness from test is not the way you like - I've managed
to change environment for this test to LC_ALL=en_US. Now it should work ok.
*done*
Re 4 gitignore file is only at the project root lvl *done*
Re 5 *done*
Re 7 *done*
Re 8 *done*
I will try to externalize chinook db and merge QuidemTest with my abstract
class as desired.
> More complex end2end tests in Calcite Plus module
> -
>
> Key: CALCITE-2088
> URL: https://issues.apache.org/jira/browse/CALCITE-2088
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Piotr Bojko
>Assignee: Piotr Bojko
>
> As in following correspondence - we would like to have more tests with more
> complexity. This should lead to lower regressions.
> {quote}
> Yes, please do.
> On Dec 8, 2017, at 7:13 AM, ptr.bo...@gmail.com wrote:
>
> I've taken a look into quidem.
>
> I will log jira for that and assign it to myself, ok?
>
> On Tue, Dec 5, 2017, 22:26 Julian Hyde wrote:
>
> More tests are always welcome.
>
> I would be wary of adding a new approach (assertj-db). Over time we end up
> with as many approaches as there are contributors, and so the code becomes
> hard to maintain. Consider using quidem (see QuidemTest and various .iq
> files in the code base); it combines assertion-based testing with the
> simplicity of script-based tests.
>
> This could be added to the “plus” module, where we don’t mind extra
> dependencies, and don’t mind if the test suite takes a long time.
>
> Julian
>
>
> On Dec 2, 2017, at 3:40 PM, ptr.bo...@gmail.com wrote:
>
> Hello fellow calcite dev team,
>
> I am building a database with use of calcite framework and decided that
> instead of simple unit tests I will go only with integration tests. This
> is
> due the fact that my code only glues the calcite with data and configures
> the whole thing decorating with web api and jdbc access (with avatica ;)
> ).
>
> I have some problems with calcite, possible bugs - some of them in apache
> jira for calcite logged already. Those problem are visible through my
> tests.
>
> And with that in mind I have an idea for a new maven artifact for
> calcite -
> end to end tests for an example h2 database. Database could have some
> tables with data - maybe 100k rows in all tables. Tests with assertj and
> its derivatives, something like I've done in my project - see the
> pastebin
> https://pastebin.com/raw/mevih4k6 .
>
> Such test set can help with lowering regressions establishing a common
> ground for talking about the calcite behaviour on specific cases (which
> can
> be described through end2end tests).
>
> The tech under such maven artifact can be pretty simple:
>
> - h2 as a data source, maybe some other
> - one properly complicated json calcite schema
> - some tech for populating h2 with data (just for having data with some
> descriptive language, not a binary format)
> - assertj-db for DSL in tests
>
> What do You think?
>
> Cheers,
> Pete
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
