[jira] [Commented] (CALCITE-3329) Implement osquery for OS adapter
[
https://issues.apache.org/jira/browse/CALCITE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798465#comment-17798465
]
Julian Hyde commented on CALCITE-3329:
--
[~x1q1j1] In my
[3329-osquery|https://github.com/julianhyde/calcite/tree/3329-osquery] branch I
have added some doc, and fixed the {{.mailmap}} file. Are those commits OK? If
so I will merge.
(By the way, I see you use {{apache.com}} in your email address rather than
{{{}apache.org{}}}. Is this intentional? ASF doesn't own the domain
{{{}apache.com{}}}.)
> Implement osquery for OS adapter
>
>
> Key: CALCITE-3329
> URL: https://issues.apache.org/jira/browse/CALCITE-3329
> Project: Calcite
> Issue Type: New Feature
>Reporter: Forward Xu
>Assignee: Forward Xu
>Priority: Major
> Labels: pull-request-available
> Attachments: 2016021823310.png, 20160218233139164.png,
> image-2019-09-09-08-27-56-974.png, image-2019-09-09-08-33-41-411.png,
> sigar1.pdf, sigar2.pdf
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Implement osquery command for OS adapter, Achieve similar features of
> FaceBook's osquery. E.g:
> select * from os_version;
> select * from system_info;
> select * from mounts;
> select * from interface_addresses
> select * from memory_info;
> select * from interface_addresses;
> select * from cpu_info;
> select * from java_info;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-3679) Allow lambda expressions in SQL queries
[ https://issues.apache.org/jira/browse/CALCITE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798444#comment-17798444 ] hongyu guo edited comment on CALCITE-3679 at 12/19/23 6:33 AM: --- Fixed in [6f64865|https://github.com/apache/calcite/commit/6f64865eb8c71a65bde60a19de2de42775fae4ed]. [~ritesh.kapoor] Thanks for your contribution! [~julianhyde] [~mbudiu] Thanks for your review! was (Author: JIRAUSER300840): Solved in [6f64865|https://github.com/apache/calcite/commit/6f64865eb8c71a65bde60a19de2de42775fae4ed]. [~ritesh.kapoor] Thanks for your contribution! [~julianhyde] [~mbudiu] Thanks for your review! > Allow lambda expressions in SQL queries > --- > > Key: CALCITE-3679 > URL: https://issues.apache.org/jira/browse/CALCITE-3679 > Project: Calcite > Issue Type: New Feature >Reporter: Ritesh >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: [CALCITE-3679]_Basic_implementation.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://teradata.github.io/presto/docs/0.167-t/functions/lambda.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-3679) Allow lambda expressions in SQL queries
[ https://issues.apache.org/jira/browse/CALCITE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798444#comment-17798444 ] hongyu guo commented on CALCITE-3679: - Solved in [6f64865|https://github.com/apache/calcite/commit/6f64865eb8c71a65bde60a19de2de42775fae4ed]. [~ritesh.kapoor] Thanks for your contribution! [~julianhyde] [~mbudiu] Thanks for your review! > Allow lambda expressions in SQL queries > --- > > Key: CALCITE-3679 > URL: https://issues.apache.org/jira/browse/CALCITE-3679 > Project: Calcite > Issue Type: New Feature >Reporter: Ritesh >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: [CALCITE-3679]_Basic_implementation.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://teradata.github.io/presto/docs/0.167-t/functions/lambda.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-3522) sql validator limits decimal literals to 64 bits
[ https://issues.apache.org/jira/browse/CALCITE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798445#comment-17798445 ] Julian Hyde commented on CALCITE-3522: -- I don't think you can just remove the checks. If you do that then some implementations (e.g. those that map DECIMAL values to 64-bit scaled integers) will give incorrect results. > sql validator limits decimal literals to 64 bits > > > Key: CALCITE-3522 > URL: https://issues.apache.org/jira/browse/CALCITE-3522 > Project: Calcite > Issue Type: Test > Components: core >Affects Versions: 1.18.0 >Reporter: Changbo Shu >Priority: Minor > Attachments: code.png > > > [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L2983] > for example: > create table tbl(f1 double), > f1 stores a double's max value. (1.7976931348623157E308) > long max value is 9223372036854775807. > select * from table where f1=value, if value is greater than long max, > sqlvalidator will throw out of range exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-3679) Allow lambda expressions in SQL queries
[ https://issues.apache.org/jira/browse/CALCITE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-3679. - Fix Version/s: 1.37.0 Assignee: hongyu guo (was: Ritesh) Resolution: Fixed > Allow lambda expressions in SQL queries > --- > > Key: CALCITE-3679 > URL: https://issues.apache.org/jira/browse/CALCITE-3679 > Project: Calcite > Issue Type: New Feature >Reporter: Ritesh >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: [CALCITE-3679]_Basic_implementation.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://teradata.github.io/presto/docs/0.167-t/functions/lambda.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6168) RexExecutor can throw during compilation
[
https://issues.apache.org/jira/browse/CALCITE-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798440#comment-17798440
]
Julian Hyde commented on CALCITE-6168:
--
This case is very similar to CALCITE-1439. Is it possible to use the same
behavior?
> RexExecutor can throw during compilation
>
>
> Key: CALCITE-6168
> URL: https://issues.apache.org/jira/browse/CALCITE-6168
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> The RexExecutor is supposed to evaluate expressions at compilation time; if
> an expression causes an exception, it should be caught and the expression
> should be unchanged. The goal is to have the exception be reported at
> runtime. The executor does catch exceptions during its "execute" phase, but
> some exceptions can be caught during its "compile" phase.
> The following SqlOperatorTest will exhibit such an instance:
> {code:java}
> @Test
> void testCast() {
> final SqlOperatorFixture f = fixture();
> f.checkFails("CAST(200 as TINYINT)", "", true);
> }
> }
> {code}
> This is the relevant portion of the stack trace:
> {code}
> at
> org.apache.calcite.linq4j.tree.Primitive.checkRoundedRange(Primitive.java:383)
> at
> org.apache.calcite.linq4j.tree.Primitive.numberValue(Primitive.java:398)
> at
> org.apache.calcite.linq4j.tree.Expressions.constant(Expressions.java:575)
> at
> org.apache.calcite.linq4j.tree.OptimizeShuttle.visit(OptimizeShuttle.java:305)
> at
> org.apache.calcite.linq4j.tree.UnaryExpression.accept(UnaryExpression.java:39)
> at
> org.apache.calcite.linq4j.tree.TernaryExpression.accept(TernaryExpression.java:47)
> at
> org.apache.calcite.linq4j.tree.Expressions.acceptExpressions(Expressions.java:3214)
> at
> org.apache.calcite.linq4j.tree.NewArrayExpression.accept(NewArrayExpression.java:49)
> at
> org.apache.calcite.linq4j.tree.GotoStatement.accept(GotoStatement.java:64)
> at
> org.apache.calcite.linq4j.tree.BlockBuilder.optimize(BlockBuilder.java:455)
> at
> org.apache.calcite.linq4j.tree.BlockBuilder.toBlock(BlockBuilder.java:340)
> at
> org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:102)
> at
> org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:68)
> at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:133)
> at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2265)
> at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:292)
> at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:250)
> at
> org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:189)
> at
> org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:184)
> at
> org.apache.calcite.tools.RelBuilder.lambda$project_$7(RelBuilder.java:2050)
> {code}
> Notice that this happens during the compile stage:
> at
> org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:68)
> The simplest fix is probably to move the try/catch block earlier in the flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6162) Add rule(s) to remove joins with constant single tuple relations
[
https://issues.apache.org/jira/browse/CALCITE-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798437#comment-17798437
]
Julian Hyde commented on CALCITE-6162:
--
That sounds right.
> Add rule(s) to remove joins with constant single tuple relations
>
>
> Key: CALCITE-6162
> URL: https://issues.apache.org/jira/browse/CALCITE-6162
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Stamatis Zampetakis
>Assignee: Hanumath Rao Maduri
>Priority: Major
>
> In various cases SQL users tend to create joins even when it is not really
> necessary. One common pattern is creating joins (or cartesian products) with
> constant relations with exactly one tuple.
> *Q1*
> Before:
> {code:sql}
> select e.empno, e.ename, c.* from emp e cross join (select 5,
> current_timestamp) c;
> {code}
> After:
> {code:sql}
> select e.empno, e.ename, 5, current_timestamp from emp e;
> {code}
> *Q2*
> Before:
> {code:sql}
> select e.empno, e.ename, c.t from emp e inner join (select 7934 as ono,
> current_timestamp as t) c on e.empno=c.ono;
> {code}
> After:
> {code:sql}
> select e.empno, e.ename, current_timestamp from emp e where e.empno=7934;
> {code}
> In the queries outlined above the one side of the join is constant and has
> exactly one tuple so the join can be dropped.
> In a nutshell the new rule(s) should be able to transform the "Before" to
> "After" for the above queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6169) EnumUtils.convert does not implement the correct SQL cast semantics
Mihai Budiu created CALCITE-6169: Summary: EnumUtils.convert does not implement the correct SQL cast semantics Key: CALCITE-6169 URL: https://issues.apache.org/jira/browse/CALCITE-6169 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu Assignee: Mihai Budiu The code generated by EnumUtil for casts uses the Java semantics, where casts just truncate values. The SQL semantics requires a runtime exception if the value does not fit in the target datatype. Some of the bugs in this code generator are masked by the BlockBuilder optimizer which attempts to optimize the generated code using the SQL semantics... But if the code is not optimized and is executed in Java it generates wrong results for most conversions that overflow. One additional wrinkle is that it is not entirely clear whether the bug is in EnumUtils or in the code that invokes EnumUtils. It is not specified which of the two semantics for casts EnumUtils are supposed to implement: Java or SQL? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-3522) sql validator limits decimal literals to 64 bits
[
https://issues.apache.org/jira/browse/CALCITE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798376#comment-17798376
]
Mihai Budiu commented on CALCITE-3522:
--
I have changed the title of this issue. We have the following code in
SqlValidatorImpl:
{code}
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
// Decimal and long have the same precision (as 64-bit integers), so
// the unscaled value of a decimal must fit into a long.
// REVIEW jvs 4-Aug-2004: This should probably be calling over to
// the available calculator implementations to see what they
// support. For now use ESP instead.
//
// jhyde 2006/12/21: I think the limits should be baked into the
// type system, not dependent on the calculator implementation.
BigDecimal bd = literal.getValueAs(BigDecimal.class);
BigInteger unscaled = bd.unscaledValue();
long longValue = unscaled.longValue();
if (!BigInteger.valueOf(longValue).equals(unscaled)) {
// overflow
throw newValidationError(literal,
RESOURCE.numberLiteralOutOfRange(bd.toString()));
}
break;
{code}
This looks wrong. Nothing in the spec says that the precision of Decimals is
limited.
To me it looks like checks can be just removed.
> sql validator limits decimal literals to 64 bits
>
>
> Key: CALCITE-3522
> URL: https://issues.apache.org/jira/browse/CALCITE-3522
> Project: Calcite
> Issue Type: Test
> Components: core
>Affects Versions: 1.18.0
>Reporter: Changbo Shu
>Priority: Minor
> Attachments: code.png
>
>
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L2983]
> for example:
> create table tbl(f1 double),
> f1 stores a double's max value. (1.7976931348623157E308)
> long max value is 9223372036854775807.
> select * from table where f1=value, if value is greater than long max,
> sqlvalidator will throw out of range exception.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-3522) sql validator limits decimal literals to 64 bits
[ https://issues.apache.org/jira/browse/CALCITE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihai Budiu updated CALCITE-3522: - Summary: sql validator limits decimal literals to 64 bits (was: sql validator limits decimal and long have the same precision, so the unscaled value of a decimal must fit into a long) > sql validator limits decimal literals to 64 bits > > > Key: CALCITE-3522 > URL: https://issues.apache.org/jira/browse/CALCITE-3522 > Project: Calcite > Issue Type: Test > Components: core >Affects Versions: 1.18.0 >Reporter: Changbo Shu >Priority: Minor > Attachments: code.png > > > [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L2983] > for example: > create table tbl(f1 double), > f1 stores a double's max value. (1.7976931348623157E308) > long max value is 9223372036854775807. > select * from table where f1=value, if value is greater than long max, > sqlvalidator will throw out of range exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6168) RexExecutor can throw during compilation
Mihai Budiu created CALCITE-6168:
Summary: RexExecutor can throw during compilation
Key: CALCITE-6168
URL: https://issues.apache.org/jira/browse/CALCITE-6168
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The RexExecutor is supposed to evaluate expressions at compilation time; if an
expression causes an exception, it should be caught and the expression should
be unchanged. The goal is to have the exception be reported at runtime. The
executor does catch exceptions during its "execute" phase, but some exceptions
can be caught during its "compile" phase.
The following SqlOperatorTest will exhibit such an instance:
{code:java}
@Test
void testCast() {
final SqlOperatorFixture f = fixture();
f.checkFails("CAST(200 as TINYINT)", "", true);
}
}
{code}
This is the relevant portion of the stack trace:
{code}
at
org.apache.calcite.linq4j.tree.Primitive.checkRoundedRange(Primitive.java:383)
at
org.apache.calcite.linq4j.tree.Primitive.numberValue(Primitive.java:398)
at
org.apache.calcite.linq4j.tree.Expressions.constant(Expressions.java:575)
at
org.apache.calcite.linq4j.tree.OptimizeShuttle.visit(OptimizeShuttle.java:305)
at
org.apache.calcite.linq4j.tree.UnaryExpression.accept(UnaryExpression.java:39)
at
org.apache.calcite.linq4j.tree.TernaryExpression.accept(TernaryExpression.java:47)
at
org.apache.calcite.linq4j.tree.Expressions.acceptExpressions(Expressions.java:3214)
at
org.apache.calcite.linq4j.tree.NewArrayExpression.accept(NewArrayExpression.java:49)
at
org.apache.calcite.linq4j.tree.GotoStatement.accept(GotoStatement.java:64)
at
org.apache.calcite.linq4j.tree.BlockBuilder.optimize(BlockBuilder.java:455)
at
org.apache.calcite.linq4j.tree.BlockBuilder.toBlock(BlockBuilder.java:340)
at
org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:102)
at
org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:68)
at
org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:133)
at
org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2265)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:292)
at
org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:250)
at
org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:189)
at
org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:184)
at
org.apache.calcite.tools.RelBuilder.lambda$project_$7(RelBuilder.java:2050)
{code}
Notice that this happens during the compile stage:
at
org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:68)
The simplest fix is probably to move the try/catch block earlier in the flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6165) Add date_sub and date_diff functions
[ https://issues.apache.org/jira/browse/CALCITE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caican Cai updated CALCITE-6165: Summary: Add date_sub and date_diff functions (was: Add date_sub and date_add functions ) > Add date_sub and date_diff functions > - > > Key: CALCITE-6165 > URL: https://issues.apache.org/jira/browse/CALCITE-6165 > Project: Calcite > Issue Type: Test > Components: tests >Affects Versions: 1.36.0 >Reporter: Caican Cai >Assignee: Caican Cai >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: 2023-12-17 17-04-29屏幕截图-1.png, 2023-12-17 > 17-04-29屏幕截图.png > > > In calcite's ParserTest, there is a lack of test for DateFunction. Currently, > calcite only supports date function from parser.jj: > date_diff, > date_trunc. > It also lacks support for many date functions. For details, you can > see:[https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html] > I supported the two functions date_add and date_sub in Parser.jj, and added > relevant tests about date function in SqlParserTest. The purpose of adding > this datefunctiontest is to tell us which date functions are supported and > which date functions are not supported. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6161) The equalsDeep of sqlCall should compare sqlOperator's sqlKind
[ https://issues.apache.org/jira/browse/CALCITE-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798195#comment-17798195 ] Mingcan Wang commented on CALCITE-6161: --- [~julianhyde] Sorry to bother you, I have started adding tests tonight. But I have some questions about thes tests. * I still create these SqlNodes manually, right? * Does "resolved to MINUS" mean that I can use the SqlStdOperatorTable.MINUS directly? The test I have currently commited already use SqlStdOperatorTable.MINUS. * Does "not resolved" mean I should create a SqlUnresolvedFunction manually? * Is there any difference between -(x, y) and x - y in the first two tests? They are all "(infix operator, not resolved)" > The equalsDeep of sqlCall should compare sqlOperator's sqlKind > -- > > Key: CALCITE-6161 > URL: https://issues.apache.org/jira/browse/CALCITE-6161 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.36.0 >Reporter: Mingcan Wang >Assignee: Mingcan Wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > Here is the situation: > when I create two SqlBasicCalls, one uses UNARY_MINUS operator and the other > uses MINUS operator. But their operandList is the same. However, The result > after calling their equalsDeep() method is true. > I found that the sqlKind of many operators is reused, and the name is not > guaranteed to be unique. Only the combination of kind and name can determine > the unique operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6162) Add rule(s) to remove joins with constant single tuple relations
[
https://issues.apache.org/jira/browse/CALCITE-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798090#comment-17798090
]
Stamatis Zampetakis commented on CALCITE-6162:
--
I haven't examined all possible joins but I have the feeling that we could
simplify most (if not all) of them. For the left join above we could possibly
make use of CASE WHEN operator:
{code:sql}
select c.empno, c.ename, c.t
from (select e.empno, e.ename, CASE WHEN (e.empno = 7934) THEN
current_timestamp ELSE null END as t from emp e) c
{code}
> Add rule(s) to remove joins with constant single tuple relations
>
>
> Key: CALCITE-6162
> URL: https://issues.apache.org/jira/browse/CALCITE-6162
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Stamatis Zampetakis
>Assignee: Hanumath Rao Maduri
>Priority: Major
>
> In various cases SQL users tend to create joins even when it is not really
> necessary. One common pattern is creating joins (or cartesian products) with
> constant relations with exactly one tuple.
> *Q1*
> Before:
> {code:sql}
> select e.empno, e.ename, c.* from emp e cross join (select 5,
> current_timestamp) c;
> {code}
> After:
> {code:sql}
> select e.empno, e.ename, 5, current_timestamp from emp e;
> {code}
> *Q2*
> Before:
> {code:sql}
> select e.empno, e.ename, c.t from emp e inner join (select 7934 as ono,
> current_timestamp as t) c on e.empno=c.ono;
> {code}
> After:
> {code:sql}
> select e.empno, e.ename, current_timestamp from emp e where e.empno=7934;
> {code}
> In the queries outlined above the one side of the join is constant and has
> exactly one tuple so the join can be dropped.
> In a nutshell the new rule(s) should be able to transform the "Before" to
> "After" for the above queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-6163) The spark ARRAY function gives exception when elements contain row(struct) type and NULL
[
https://issues.apache.org/jira/browse/CALCITE-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ran Tao resolved CALCITE-6163.
--
Fix Version/s: 1.37.0
Assignee: Ran Tao
Resolution: Fixed
Fixed via
[https://github.com/apache/calcite/commit/e066266dcde21b3a0de90ec601ff539be1b8d7a3]
> The spark ARRAY function gives exception when elements contain row(struct)
> type and NULL
>
>
> Key: CALCITE-6163
> URL: https://issues.apache.org/jira/browse/CALCITE-6163
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Ran Tao
>Assignee: Ran Tao
>Priority: Major
> Fix For: 1.37.0
>
>
> when we run
> {noformat}
> array(row(1,2), null)
> {noformat}
> It will give 'Parameters must be of the same type' exception.
> {noformat}
> org.apache.calcite.runtime.CalciteContextException: Parameters must be of the
> same type
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:507)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:948)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:933)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5517)
> at
> org.apache.calcite.sql.SqlCallBinding.newValidationError(SqlCallBinding.java:414)
> at
> org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypesImpl(SameOperandTypeChecker.java:100)
> at
> org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypes(SameOperandTypeChecker.java:59)
> at
> org.apache.calcite.sql.SqlOperator.checkOperandTypes(SqlOperator.java:761)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:498)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6575)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6562)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:166)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1926)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1911)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:276)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:275)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1091)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:797)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:161)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:249)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:446)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
> at
> org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:226)
> at
> org.apache.calcite.test.SqlOperatorTest.testArrayFunction(SqlOperatorTest.java:10551)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
[jira] [Resolved] (CALCITE-6127) The spark array function gives NullPointerException when element is row type
[
https://issues.apache.org/jira/browse/CALCITE-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ran Tao resolved CALCITE-6127.
--
Assignee: Ran Tao
Resolution: Fixed
> The spark array function gives NullPointerException when element is row type
>
>
> Key: CALCITE-6127
> URL: https://issues.apache.org/jira/browse/CALCITE-6127
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Ran Tao
>Assignee: Ran Tao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> when we run
> {code:java}
> array(row(1,2))
> {code}
> It will give NullPointerException.
> {noformat}
> java.lang.NullPointerException: array element type family
> at java.base/java.util.Objects.requireNonNull(Objects.java:246)
> at
> org.apache.calcite.sql.fun.SqlLibraryOperators.arrayReturnType(SqlLibraryOperators.java:1070)
> at
> org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:534)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:503)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6575)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6562)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:166)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1926)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1911)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:276)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:275)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1091)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:797)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:161)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:249)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:446)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
> at
> org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:226)
> at
> org.apache.calcite.test.SqlOperatorTest.testArrayFunction(SqlOperatorTest.java:10481)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> org.junit.jupiter.engine.executi
[jira] [Commented] (CALCITE-6127) The spark array function gives NullPointerException when element is row type
[
https://issues.apache.org/jira/browse/CALCITE-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798051#comment-17798051
]
Ran Tao commented on CALCITE-6127:
--
Fixed via
https://github.com/apache/calcite/commit/e066266dcde21b3a0de90ec601ff539be1b8d7a3
> The spark array function gives NullPointerException when element is row type
>
>
> Key: CALCITE-6127
> URL: https://issues.apache.org/jira/browse/CALCITE-6127
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Ran Tao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> when we run
> {code:java}
> array(row(1,2))
> {code}
> It will give NullPointerException.
> {noformat}
> java.lang.NullPointerException: array element type family
> at java.base/java.util.Objects.requireNonNull(Objects.java:246)
> at
> org.apache.calcite.sql.fun.SqlLibraryOperators.arrayReturnType(SqlLibraryOperators.java:1070)
> at
> org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:534)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:503)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6575)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6562)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:166)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1926)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1911)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:276)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:275)
> at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
> at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1091)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:797)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:161)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:249)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:446)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
> at
> org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
> at
> org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:226)
> at
> org.apache.calcite.test.SqlOperatorTest.testArrayFunction(SqlOperatorTest.java:10481)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(Intercepting
[jira] [Closed] (CALCITE-6167) JoinConditionPushRule shouldn't push 'exists' with other table conditions down
[
https://issues.apache.org/jira/browse/CALCITE-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SimonAlexs closed CALCITE-6167.
---
Resolution: Won't Do
It's hard to apply rules to exists or sub-query directly, I will try other ways
to deal exists and sub-query.
> JoinConditionPushRule shouldn't push 'exists' with other table conditions down
> --
>
> Key: CALCITE-6167
> URL: https://issues.apache.org/jira/browse/CALCITE-6167
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: SimonAlexs
>Priority: Major
>
> {code:java}
> SELECT *
> from (select 1 a, 2 b) t
> left join (select 2 a, 3 b) r on exists(select 1 from (select 1 a) s where
> s.a=t.a and s.a=r.a) {code}
> In this sql, the RelNode tree is as below(pay attention to the position of
> exists):
>
> {code:java}
> // 1: with FilterJoinRule.JoinConditionPushRule
> LogicalProject(a=[$0], b=[$1], a0=[$2], b0=[$3])
> LogicalJoin(condition=[true], joinType=[left])
> LogicalValues(tuples=[[{ 1, 2 }]])
> LogicalFilter(condition=[EXISTS({
> LogicalFilter(condition=[AND(=($0, $cor0.a), =($0,
> $cor1.a0))])
> LogicalValues(tuples=[[{ 1 }]])
> })])
> LogicalValues(tuples=[[{ 2, 3 }]])
> // 2: normal RelNode tree, without any rule
> LogicalProject(a=[$0], b=[$1], a0=[$2], b0=[$3])
> LogicalJoin(condition=[EXISTS({
> LogicalFilter(condition=[AND(=($0, $cor0.a), =($0,
> $cor1.a0))])
> LogicalValues(tuples=[[{ 1 }]])
> })], joinType=[left])
> LogicalValues(tuples=[[{ 1, 2 }]])
> LogicalValues(tuples=[[{ 2, 3 }]]){code}
> With JoinConditionPushRule, 'exists' is pushed down to the filter of table
> 'r'.But in 'exists', it has the condition 's.a=t.a' which uses the field of
> table 't'.This 'push down' results in that we cann't get values from table
> 't' easily.
> I think this may be a bug.
>
> Full test code is as below.(If the full code is unnecessary, please tell me,
> i will delete it)
> {code:java}
> String sql = "SELECT * from (select 1 a, 2 b) t " +
> "left join (select 2 a, 3 b) r on exists(select 1 from (select 1 a) s
> where s.a=t.a and s.a=r.a) ";
> Properties properties = new Properties();
> properties.put(CalciteConnectionProperty.MODEL.camelName(), "inline:{\n" +
> " \"version\": \"1.0\",\n" +
> " \"defaultSchema\": \"ps\",\n" +
> " \"schemas\": [ ] }");
> Statement calciteStatement = DriverManager.getConnection("jdbc:calcite:",
> properties).createStatement();
> CalcitePrepare.Context prepareContext =
> calciteStatement.unwrap(CalciteServerStatement.class).createPrepareContext();
> final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.config()
> .withLex(Lex.MYSQL)
> .withConformance(SqlConformanceEnum.MYSQL_5))
> .defaultSchema(prepareContext.getRootSchema().plus())
> .build();
> Planner planner = Frameworks.getPlanner(config);
> SqlNode parsedSql = planner.parse(sql);
> SqlNode validatedSql = planner.validate(parsedSql);
> RelNode originRel = planner.rel(validatedSql).rel;
> // optimize
> HepProgram program = HepProgram.builder()
> .addRuleInstance(CoreRules.JOIN_CONDITION_PUSH)
> .build();
> HepPlanner hepPlanner = new HepPlanner(program);
> hepPlanner.setRoot(originRel);
> RelNode optimizedRel = hepPlanner.findBestExp();
> System.out.println(RelOptUtil.toString(optimizedRel)); {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
