[jira] [Created] (CALCITE-4207) Validation fails for positional aggregate with current_date in 'case' expression
Igor Guzenko created CALCITE-4207:
-
Summary: Validation fails for positional aggregate with
current_date in 'case' expression
Key: CALCITE-4207
URL: https://issues.apache.org/jira/browse/CALCITE-4207
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Igor Guzenko
*Steps to reproduce: *
Add test to SqlValidatorTest.java
{code:java}
@Test void testPositionalAggregateWithExpandedCurrentDateFunction() {
sql("SELECT "
+ "CASE WHEN HIREDATE >= CURRENT_DATE "
+ "THEN 'Forward' "
+ "ELSE 'Actual' END AS fa,"
+ "COUNT(*) "
+ "FROM EMP "
+ "GROUP BY 1").ok();
}
{code}
*Expected result*
Validation passed successfully.
*Actual result*
{code:none}org.apache.calcite.sql.validate.SqlValidatorException: Expression
'HIREDATE' is not being grouped
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:467)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:560)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:883)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:868)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5003)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:113)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:40)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:321)
at
org.apache.calcite.sql.util.SqlBasicVisitor$ArgHandlerImpl.visitChild(SqlBasicVisitor.java:123)
at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:879)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:212)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:40)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql.util.SqlBasicVisitor.visit(SqlBasicVisitor.java:52)
at org.apache.calcite.sql.SqlNodeList.accept(SqlNodeList.java:145)
at
org.apache.calcite.sql.util.SqlBasicVisitor$ArgHandlerImpl.visitChild(SqlBasicVisitor.java:123)
at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:879)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:212)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:40)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql.util.SqlBasicVisitor$ArgHandlerImpl.visitChild(SqlBasicVisitor.java:123)
at
org.apache.calcite.sql.SqlAsOperator.acceptCall(SqlAsOperator.java:121)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:212)
at org.apache.calcite.sql.validate.AggChecker.visit(AggChecker.java:40)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql.validate.AggregatingSelectScope.checkAggregateExpr(AggregatingSelectScope.java:212)
at
org.apache.calcite.sql.validate.AggregatingSelectScope.validateExpr(AggregatingSelectScope.java:221)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Re: [ANNOUNCE] New Calcite PMC chair: Stamatis Zampetakis
Congratulations, Stamatis! On Thu, Dec 19, 2019 at 9:04 AM Alessandro Solimando < alessandro.solima...@gmail.com> wrote: > Congratulations, Stamatis! > > Il Gio 19 Dic 2019, 07:16 Enrico Olivelli ha > scritto: > > > Congratulations Stamatis! > > > > Enrico > > > > Il gio 19 dic 2019, 04:40 Rui Wang ha scritto: > > > > > Congratulations and Thanks Stamatis! > > > > > > > > > > > > -Rui > > > > > > On Wed, Dec 18, 2019 at 6:52 PM XING JIN > > wrote: > > > > > > > Congratulations Stamatis! > > > > > > > > -Jin > > > > > > > > Chunwei Lei 于2019年12月19日周四 上午10:33写道： > > > > > > > > > Congratulations Stamatis! > > > > > > > > > > > > > > > Best, > > > > > Chunwei > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 9:36 AM Danny Chan > > > wrote: > > > > > > > > > > > Congratulations Stamatis! > > > > > > > > > > > > Best, > > > > > > Danny Chan > > > > > > 在 2019年12月19日 +0800 AM7:37，dev@calcite.apache.org，写道： > > > > > > > > > > > > > > Congratulations Stamatis! > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Remove Kotlin
Hello Calcite Team, I believe products are advancing through constant improvements rather than freezing codebase. Why does nobody consider Kotlin as an opportunity to improve Calcite? Indeed, most of the replies sound like: "I don't want to learn Kotlin because it requires extra efforts, etc, etc." Sincerely, Igor On Tue, Dec 17, 2019 at 4:20 PM Michael Mior wrote: > Le mar. 17 déc. 2019 à 03:30, Vladimir Sitnikov > a écrit : > > > > Kevin>Focusing on the technical side of things, I agree that introducing > a > > new > > Kevin>language is of little benefit currently > > > > Kevin, what is your opinion on removing Quidem language? > > Focusing on the technical side, it is a standalone language. > > The language is not Java, it has limited tooling, it has a limited set of > > users, it has 0 or so questions on StackOverflow and so on. > > I'm not Kevin, but removing a language is quite different from > introducing a new language. I wasn't around when Quidem tests were > introduced, but I assume that similar concerns would have been raised. > > > > > Kevin>I agree that introducing a new > > Kevin>language is of little benefit currently > > > > Keeping tests easy to read, easy to edit, easy to create, easy to debug, > > and easy to maintain is hard. > > Making tests easy to read simplifies code reviews that pay off in the > long > > run. > > Agreed on those points. I'm not convinced though that having another > language tests are written in will simplify code review. It may be > true for someone equally literate in Java and Kotlin that in some > instances, Kotlin tests will be easier to read. However, we know our > current contributors are reasonably fluent in Java. I'm not sure about > Kotlin. > > > It is sad you mention "code readability" as "little benefit currently" > item. > > > > Kevin>surprised that a change went in to > > Kevin>switch to Kotlin especially after the discussion that is happening > on > > the > > Kevin>mailing list. > > > > Those are two different items. The discussion re $ is still open, and > > there's no clear answer yet. > > At the same time, in the very same thread, there were explicit opinions > > that "tests that do not use $ might still benefit from improved > > readability". > > > > The change in [2] is not related to the $-discussion, so I don't see why > > people relate them. > > > > Kevin>I agree that introducing a new > > Kevin>language is of little benefit currently > > > > It is not introducing a brand new language. The very same language is > used > > in the build scripts. > > The language was designed for interoperability with Java, and it does > > improve things if used appropriately. > > > > For instance, tests like > > > https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94 > > are > > much easier to read and update than similar tests in Java. > > Assuming you understand, Kotlin, yes. I agree it's not exactly > introducing a new language, but I think using Kotlin in tests is quite > different from using it in build scripts. We expect most contributors > to write tests. If build script editing is a bit less accessible, I > think that's less of a concern. > > > > > Note: Kotlin tests are still regular JUnit5 tests, so they get proper > > statistics in the CI outputs like test count, skipped test count, failure > > count, test duration. > > > > Kevin>In my opinion there are more negatives than positives > > Kevin>currently. > > > > What your opinion re Kotlin is based on besides angst and "introducing a > > new language"? > > I can easily relate how the items play for writing tests COBOL, but, > well, > > I hope we don't consider that :) > > > > Kotlin is a modern language with good tooling. > > It was designed for smooth Java interop which plays well for maintenance. > > The code is easy to read, and it is copy-paste friendly in the same way > as > > almost any other language we currently have in the source repository. > > So newbies could copy-paste Kotlin code or Java code or Quidem code. > > > > For instance, people might still create tests in Java and call methods > > written in Kotlin. They might create "pure Java" tests if they like. > > They might even create Quidem tests, but they won't be able to call > > Java/Kotlin methods (and/or extend classes) then. > > > > It looks like you express an opinion on "let's rewrite everything in > > Kotlin" rather than "let's pick the proper tools on a case by case > basis". > > > > Currently, there are valid reasons against migrating all the tests to > > Kotlin, and I guess we have never discussed that. > > At the same time, we are not discussing "rewrite everything in Kotlin" > here. > > I don't think anyone was discussing rewriting everything in Kotlin. > But I think that's a straw man. Rewriting some things in Kotlin still > has some of the same disadvantages. > > > > > Vladimir >
Re: [Discuss] Make flattening on Struct/Row optional
Hi Rui, I'm glad that the fix was useful. Thanks, Igor On Thu, Dec 12, 2019 at 8:16 PM Rui Wang wrote: > Absolutely. Thanks lgor for the contribution! :) > > > -Rui > > On Wed, Dec 11, 2019 at 10:54 PM Stamatis Zampetakis > wrote: > > > So basically thanks to Igor :) > > > > On Wed, Dec 11, 2019 at 9:56 PM Rui Wang wrote: > > > > > Thanks Stamatis's suggestion. Indeed a recent effort in [1] enhanced > the > > > support that reconstructs ROW in the top SELECT, which is supposed to > > solve > > > the problem. > > > > > > > > > > > > [1]: https://jira.apache.org/jira/browse/CALCITE-3138 > > > > > > On Mon, Dec 9, 2019 at 3:21 PM Rui Wang wrote: > > > > > > > Hello, > > > > > > > > Sorry for the long delay on this thread. Recently I heard about > > requests > > > > on how to deal with STRUCT without flattening it again in BeamSQL. > > Also I > > > > realized Flink has already disabled it in their codebase[1]. I did > try > > to > > > > remove STRUCT flattening and run unit tests of calcite core to see > how > > > many > > > > tests breaks: it was 25, which wasn't that bad. So I would like to > pick > > > up > > > > this effort again. > > > > > > > > Before I do it, I just want to ask if Calcite community supports this > > > > effort (or think if it is a good idea)? > > > > > > > > My current execution plan will be the following: > > > > 1. Add a new flag to FrameworkConfig to specify whether flattening > > > STRUCT. > > > > By default, it is yes. > > > > 2. When disabling struct flatterner, add more tests to test STRUCT > > > support > > > > in general. For example, test STRUCT support on projection, join > > > condition, > > > > filtering, etc. If there is something breaks, try to fix it. > > > > 3. Check the 25 failed tests above and see why they have failed if > > struct > > > > flattener is gone. Duplicate those failed tests but have necessary > > fixes > > > to > > > > make sure they can pass without STRUCT flattening. > > > > > > > > > > > > [1]: > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/calcite/FlinkPlannerImpl.scala#L166 > > > > > > > > > > > > -Rui > > > > > > > > On Wed, Sep 5, 2018 at 11:59 AM Julian Hyde > wrote: > > > > > > > >> It might not be minor, but it’s worth a try. At optimization time we > > > >> treat all fields as fields, regardless of whether they have complex > > > types > > > >> (maps, arrays, multisets, records) so there should not be too many > > > >> problems. The flattening was mainly for the benefit of the runtime. > > > >> > > > >> > > > >> > On Sep 5, 2018, at 11:32 AM, Rui Wang > > > >> wrote: > > > >> > > > > >> > Thanks for your helpful response! It seems like disabling the > > > flattening > > > >> > will at least affect some rules in optimization. It might not be a > > > minor > > > >> > change. > > > >> > > > > >> > > > > >> > -Rui > > > >> > > > > >> > On Wed, Sep 5, 2018 at 4:54 AM Stamatis Zampetakis < > > zabe...@gmail.com > > > > > > > >> > wrote: > > > >> > > > > >> >> Hi Rui, > > > >> >> > > > >> >> Disabling flattening in some cases seems reasonable. > > > >> >> > > > >> >> If I am not mistaken, even in the existing code it is not used > all > > > the > > > >> time > > > >> >> so it makes sense to become configurable. > > > >> >> For example, Calcite prepared statements (CalcitePrepareImpl) are > > > >> using the > > > >> >> flattener only for DDL operations that create materialized views > > (and > > > >> this > > > >> >> is because this code at some point passes from the PlannerImpl). > > > >> >> On the other hand, any query that is using the Planner will also > > pass > > > >> from > > > >> >> the flattener. > > > >> >> > > > >> >> Disabling the flattener does not mean that all rules will work > > > without > > > >> >> problems. The Javadoc of the RelStructuredTypeFlattener at some > > point > > > >> says > > > >> >> "This approach has the benefit that real optimizer and codegen > > rules > > > >> never > > > >> >> have to deal with structured types.". Due to this, it is very > > likely > > > >> that > > > >> >> some rules were written based on the fact that there are no > > > structured > > > >> >> types. > > > >> >> > > > >> >> Best, > > > >> >> Stamatis > > > >> >> > > > >> >> > > > >> >> Στις Τετ, 5 Σεπ 2018 στις 9:48 π.μ., ο/η Julian Hyde < > > > jh...@apache.org > > > >> > > > > >> >> έγραψε: > > > >> >> > > > >> >>> Flattening was introduced mainly because the original engine > used > > > flat > > > >> >>> column-oriented storage. Now we have several ways to executing, > > > >> >>> including generating java code. > > > >> >>> > > > >> >>> Adding a mode to disable flattening might make sense. > > > >> >>> On Tue, Sep 4, 2018 at 12:52 PM Rui Wang > > > > > > > > >> >>> wrote: > > > >> > > > >> Hi Community, > > > >> > > > >> While trying to support Row type in Apache Beam SQL on top of > > > >> Calcite, > > > >> >> I > > > >>
[jira] [Created] (CALCITE-3525) RexSimplify: eliminate redundant rex calls in OR
Igor Guzenko created CALCITE-3525:
-
Summary: RexSimplify: eliminate redundant rex calls in OR
Key: CALCITE-3525
URL: https://issues.apache.org/jira/browse/CALCITE-3525
Project: Calcite
Issue Type: Improvement
Reporter: Igor Guzenko
Assignee: Igor Guzenko
Sample case to reproduce in {code}RexProgramTest.simplifyOrTerms{code}:
{code:java}
// (a=1 or a=2 or (arr[1]>4 and arr[1]<3 and a=3)) => a=1 or a=2
final RelDataType intArrayType = typeFactory.createArrayType(intType, -1);
final RexInputRef ref0 = rexBuilder.makeInputRef(intType, 0);
final RexInputRef ref3 = rexBuilder.makeInputRef(intArrayType, 3);
final RexCall itm1 = (RexCall) rexBuilder.makeCall(intType,
SqlStdOperatorTable.ITEM,
ImmutableList.of(ref3, literal1));
simplify = this.simplify.withParanoid(false);
checkSimplifyFilter(
or(
eq(ref0, literal1),
eq(ref0, literal2),
and(
gt(itm1, literal4),
lt(itm1, literal3),
eq(ref0, literal3)
)
),
"OR(=($0, 1), =($0, 2))"
);
simplify = simplify.withParanoid(true);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Nested data handling in Caclite
Hello Naveen,
1. If I understand correctly, then yes you can extract nested fields from
struct type. The syntax depends on StructKind value for your data type,
for example for FULLY_QUALIFIED struct you should first
make alias for your table and then request nested field like,
table_alias.struct_column.nested_field. In rel tree such expressions are
presented as RexCall with SqlItemOperator operator.
2. Yes, this ability was implemented in CALCITE-3138 [1]. It builds call to
ROW type constructor function on top of flattened tree for necessary
columns.
3. Yes, examples of such functions are ROW(...), ANY_VALUE(...) etc.
In current implementation of flattener invocation of ROW constructor
function is done despite of null handling same issue exists for some
aggregate function flattening, like COUNT(struct_column).
Proper null handling is real pain for flattener, original idea was to
handle special null indicator for each flattened struct, but in practice I
recognized that it's really hard to deal with flattened fields indices when
related methods are called from very different points, so for now the
problem remains unsolved.
If you can't avoid dealing with null values in your struct columns you
could try to avoid invocation to SqlToRelConverter.flattenTypes(...) and
check whether final plan acceptable for you. As far as I know
there is no reading material for given topic, you can investigate source
code by debugging RelStructuredTypeFlattener and reading some related plans
in SqlToRelConverterTest.java and SqlToRelConverterTest.xml.
[1] https://issues.apache.org/jira/browse/CALCITE-3138
Thanks,
Igor
On Thu, Oct 24, 2019 at 12:57 PM Naveen Kumar
wrote:
> Hi,
>
> I work at Flipkart, we are using Calcite in our streaming platform. In most
> of our use cases, input data is nested. I understand Calcite flattens
> structs in scan and references fields positionally.
>
> I had a few questions on handling nested data -
>
>1. Can RelNode DAG work with nested data (instead of flattened fields)
>by referencing fields through their nested structure eg,
> data.order.orderId
>2. In the current flattened behavior, can output of a query be a struct.
>Eg if *orderId, orderData.timestamp, orderData.category* are output of
>select query, can I declaratively organise output to below json
> structure -
> 1.
>
>
>
>
>
> *{ "orderId": "order1", "orderData": { "timestamp": 1571904384814,
> "category": "shoes" } }*
> 3. Can output of a UDF be struct type
>
> Please point me to any reading material or example that would help with
> these questions.
>
> Regards,
> Naveen
>
> --
>
>
>
>
> *-*
>
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this email in error, please notify the
> system manager. This message contains confidential information and is
> intended only for the individual named. If you are not the named
> addressee,
> you should not disseminate, distribute or copy this email. Please notify
> the sender immediately by email if you have received this email by mistake
> and delete this email from your system. If you are not the intended
> recipient, you are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this information is
> strictly prohibited.*
>
>
>
> *Any views or opinions presented in this
> email are solely those of the author and do not necessarily represent
> those
> of the organization. Any information on shares, debentures or similar
> instruments, recommended product pricing, valuations and the like are for
> information purposes only. It is not meant to be an instruction or
> recommendation, as the case may be, to buy or to sell securities,
> products,
> services nor an offer to buy or sell securities, products or services
> unless specifically stated to be so on behalf of the Flipkart group.
> Employees of the Flipkart group of companies are expressly required not to
> make defamatory statements and not to infringe or authorise any
> infringement of copyright or any other legal right by email
> communications.
> Any such communication is contrary to organizational policy and outside
> the
> scope of the employment of the individual concerned. The organization will
> not accept any liability in respect of such communication, and the
> employee
> responsible will be personally liable for any damages or other liability
> arising.*
>
>
>
> *Our organization accepts no liability for the
> content of this email, or for the consequences of any actions taken on the
> basis of the information *provided,* unless that information is
> subsequently confirmed in writing. If you are not the intended recipient,
> you are notified that disclosing, copying, distributing or taking any
>
[jira] [Created] (CALCITE-3424) AssertionError thrown for user-defined table function with array argument
Igor Guzenko created CALCITE-3424:
-
Summary: AssertionError thrown for user-defined table function
with array argument
Key: CALCITE-3424
URL: https://issues.apache.org/jira/browse/CALCITE-3424
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.21.0
Reporter: Igor Guzenko
Assignee: Igor Guzenko
*Steps to reproduce:*
*1.* Add method with list parameter to Smalls.java
{code:java}
public static final Method GENERATE_STRINGS_2_METHOD =
Types.lookupMethod(Smalls.class, "generateStrings2", List.class);
public static QueryableTable generateStrings2(final List list) {
return generateStrings(list.size());
}
{code}
*2.* Add test method which uses new user-defined table function to
TableFunctionTest.java
{code:java}
@Test public void testTableFunction2() throws SQLException {
try (Connection connection = DriverManager.getConnection("jdbc:calcite:")) {
CalciteConnection calciteConnection =
connection.unwrap(CalciteConnection.class);
SchemaPlus rootSchema = calciteConnection.getRootSchema();
SchemaPlus schema = rootSchema.add("s", new AbstractSchema());
final TableFunction table =
TableFunctionImpl.create(Smalls.GENERATE_STRINGS_2_METHOD);
schema.add("GenerateStrings2", table);
final String sql = "select *\n"
+ "from table(\"s\".\"GenerateStrings2\"(5,4,3,1,2)) as t(n, c)\n"
+ "where char_length(c) > 3";
ResultSet resultSet = connection.createStatement().executeQuery(sql);
assertThat(CalciteAssert.toString(resultSet),
equalTo("N=4; C=abcd\n"));
}
}
{code}
Execution result produced by such test method is the following stack trace:
{code:none}
java.lang.AssertionError: use createArrayType() instead
at
org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:221)
at
org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:48)
at
org.apache.calcite.jdbc.JavaTypeFactoryImpl.toSql(JavaTypeFactoryImpl.java:255)
at
org.apache.calcite.prepare.CalciteCatalogReader.toSql(CalciteCatalogReader.java:381)
at
org.apache.calcite.prepare.CalciteCatalogReader.lambda$toSql$7(CalciteCatalogReader.java:370)
at
com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
at
com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
at
com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
at org.apache.calcite.sql.SqlFunction.(SqlFunction.java:123)
at
org.apache.calcite.sql.validate.SqlUserDefinedFunction.(SqlUserDefinedFunction.java:63)
at
org.apache.calcite.sql.validate.SqlUserDefinedTableFunction.(SqlUserDefinedTableFunction.java:45)
at
org.apache.calcite.prepare.CalciteCatalogReader.toOp(CalciteCatalogReader.java:338)
at
org.apache.calcite.prepare.CalciteCatalogReader.toOp(CalciteCatalogReader.java:302)
at
org.apache.calcite.prepare.CalciteCatalogReader.lambda$lookupOperatorOverloads$3(CalciteCatalogReader.java:271)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490)
at
org.apache.calcite.prepare.CalciteCatalogReader.lookupOperatorOverloads(CalciteCatalogReader.java:272)
at
org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:73)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1195)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1180)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1180)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1180)
[jira] [Created] (CALCITE-3393) RelStructuredTypeFlattener: improve support for functions with struct input
Igor Guzenko created CALCITE-3393:
-
Summary: RelStructuredTypeFlattener: improve support for functions
with struct input
Key: CALCITE-3393
URL: https://issues.apache.org/jira/browse/CALCITE-3393
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Igor Guzenko
Assignee: Igor Guzenko
1.
[RelStructuredTypeFlattener|https://github.com/apache/calcite/blob/148bfd329413c0272395cc0b7c322b3c5a34b667/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java#L376]
doesn't support aggregate functions for struct type column.
Example test case:
{code:java}
@Test
public void testAggregateFunctionWithStructInput() {
final String sql =
"select count(dn.skill) from sales.dept_nested dn";
sql(sql).ok();
}
{code}
2. For other functions, except ITEM, flattener uses first nested primitive
field from
original struct input. Example test case:
{code:java}
@Test
public void testFunctionWithStructInput() {
final String sql =
"select json_type(dn.skill) from sales.dept_nested dn";
sql(sql).ok();
}
{code}
Generated plan:
{code}
LogicalProject(EXPR$0=[JSON_TYPE($2.TYPE)])
LogicalTableScan(table=[[CATALOG, SALES, DEPT_NESTED]])
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Towards Calcite 1.21.0
Hello Julian, I'll work on the CALCITE-3297 and try to fix quickly. Thanks, Igor
Re: [DISCUSS] Towards Calcite 1.21.0
Hello everyone, It would be great to include CALCITE-3115 [1] into the release. [1] https://issues.apache.org/jira/browse/CALCITE-3115 [2] https://github.com/apache/calcite/pull/1397 Thanks, Igor On Tue, Aug 27, 2019 at 5:43 AM Danny Chan wrote: > Thanks for the review for CALCITE-2302, Julian. > > Best, > Danny Chan > 在 2019年8月26日 +0800 PM12:27，Julian Hyde ，写道： > > Sounds good. > > * I am reviewing 3122 and will commit shortly. > > * I see Danny has asked me to final review 2302. I'll do that tomorrow. > > * Who owns 1581? > > > > Julian > > > > On Sun, Aug 25, 2019 at 1:28 PM Stamatis Zampetakis > wrote: > > > > > > Hello, > > > > > > According to the discussion in JIRA the following issues seem to be > very > > > close to been resolved: > > > https://issues.apache.org/jira/browse/CALCITE-1581 > > > https://issues.apache.org/jira/browse/CALCITE-2302 > > > https://issues.apache.org/jira/browse/CALCITE-3122 > > > > > > I will wait for those to be merged and then I will start the release > > > process. I would like to kindly ask the reviewers involved to check > that > > > there is nothing more left to do. > > > > > > I will start moving the remaining issues to 1.22.0. Let me know if > there is > > > anything else that we should include in this release. > > > > > > Best, > > > Stamatis > > > > > > On Mon, Aug 19, 2019, 11:04 PM Julian Hyde wrote: > > > > > > > +1 > > > > > > > > I’ve poked Khai Tran re. 3122 (Pig). > > > > > > > > > On Aug 16, 2019, at 11:20 PM, Stamatis Zampetakis < > zabe...@gmail.com> > > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > The release is advancing quite well, yet we have 22 issues marked > for > > > > > 1.21.0 [1]. > > > > > > > > > > From those, the following 5 seem to be the most important: > > > > > > > > > > https://issues.apache.org/jira/browse/CALCITE-1581 > > > > > https://issues.apache.org/jira/browse/CALCITE-2302 > > > > > https://issues.apache.org/jira/browse/CALCITE-3122 > > > > > https://issues.apache.org/jira/browse/CALCITE-3224 > > > > > https://issues.apache.org/jira/browse/CALCITE-3228 > > > > > > > > > > I will try to have an RC for the 26th of August so let's try to > get them > > > > in. > > > > > > > > > > [1] > > > > > > > > > > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12333950 > > > > > > > > > > On Thu, Aug 1, 2019 at 8:51 AM Danny Chan > wrote: > > > > > > > > > > > Let’s get the CALCITE-2302[1]: the full implicit type cast > support into > > > > > > 1.21 ! > > > > > > > > > > > > [1] https://github.com/apache/calcite/pull/706 > > > > > > > > > > > > Best, > > > > > > Danny Chan > > > > > > 在 2019年7月21日 +0800 PM3:28，Stamatis Zampetakis >，写道： > > > > > > > Hi all, > > > > > > > > > > > > > > Approximately one month has passed from the previous release > (Calcite > > > > > > > 1.20.0) and I was thinking that it would be nice to have the > next > > > > release > > > > > > > out by the end of August. To do this I think we should try to > have an > > > > RC > > > > > > > around the 20 of August. > > > > > > > > > > > > > > The progress towards the next release can be seen in [1]. As > you can > > > > > > > observe we do not have many issues marked for fixing for > 1.21.0 but we > > > > do > > > > > > > have many pull requests. > > > > > > > > > > > > > > I would like to invite committers to go over the existing PRs > and for > > > > > > those > > > > > > > that we plan to review and finalize in the next month or so > set the fix > > > > > > > version to 1.21.0 adding an appropriate comment. > > > > > > > > > > > > > > In term of priorities, I think we should emphasize on > finalizing and > > > > > > > merging PRs that are blocking other open source projects and > commercial > > > > > > > products from upgrading to the latest release. If it is not > already > > > > done > > > > > > > please assign this issues the highest priority (blocker) and > set the > > > > fix > > > > > > > version to 1.21.0. > > > > > > > > > > > > > > Apart from very important issues it makes sense to treat PRs > in FIFO > > > > > > order. > > > > > > > Contributors who submit a PR early will certainly get > discouraged to > > > > > > > contribute again if we never merge these PRs in time. > > > > > > > > > > > > > > Don't hesitate to reply to this thread if the plan above is not > > > > > > convenient > > > > > > > for you! > > > > > > > > > > > > > > Best, > > > > > > > Stamatis > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12333950 > > > > > > > > > > > > > > >
Re: Re: [DISCUSSION] Problem caused by flattening of struct fields
Thanks for quick answer, I'll work on the CALCITE-3138 <https://issues.apache.org/jira/browse/CALCITE-3138> jira. If I'm not wrong this collection of fields back after flatten supposed to be done like rex invocation to create new row. For example, collecting projection might look like: LogicalProject(HOME_ADDRESS=[NEW(ROW($1, $2, $3, $4)):ObjectSqlType(ADDRESS) NOT NULL]). Please let me know if I'm wrong and there is more proper way to restructure. Thanks, Igor On Fri, Jun 21, 2019 at 4:21 AM Haisheng Yuan wrote: > I have created 2 JIRA tickets to track the issues: > https://issues.apache.org/jira/browse/CALCITE-3137 > https://issues.apache.org/jira/browse/CALCITE-3138 > > - Haisheng > > -- > 发件人：Haisheng Yuan > 日 期：2019年06月21日 08:54:55 > 收件人：Igor Guzenko; Apache Calcite dev list< > dev@calcite.apache.org> > 主 题：Re: [DISCUSSION] Problem caused by flattening of struct fields > > Consider creating ObjectSqlType like Fixture.addressType, which is > STRUCTURED type. > typeFactory.createStructType() actually creates a ROW type, which is not > supported to reconstruct fields at the moment, see > method RelStructuredTypeFlattener.restructureFields. > > But even with ObjectSqlType, you will still see an assert error. The > assertion should be removed. > > - Haisheng > > -- > 发件人：Igor Guzenko > 日 期：2019年06月20日 15:14:47 > 收件人： > 主 题：[DISCUSSION] Problem caused by flattening of struct fields > > Hello everyone, > > I've got issue while converting query with struct column. Struct's fields > are flattened while conversion to rel is performed. > But later they aren't collected back so query plan may produce incorrect > result. > > For example [1], consider table *str_table* with just one column *str* of > type STRUCT, > then query SELECT *str* FROM *str_table* produces plan > > LogicalProject(STR=[$0]) >LogicalProject(STR=[$0.name], STR1=[$0.age]) > LogicalTableScan(table=[[CATALOG, STRUCT, STR_TABLE]]) > > where top level project returns nested field `name` as `str` instead of > original struct column. My question is, what is the correct > way to collect back the flattened fields and produce correct result for the > query ? > > [1] - > > https://github.com/ihuzenko/calcite/commit/e24eaa22fbb5c950a0bd5290cc09ca56ea7f1e44 > > Thank you in advance, > Igor Guzenko > > >
[DISCUSSION] Problem caused by flattening of struct fields
Hello everyone, I've got issue while converting query with struct column. Struct's fields are flattened while conversion to rel is performed. But later they aren't collected back so query plan may produce incorrect result. For example [1], consider table *str_table* with just one column *str* of type STRUCT, then query SELECT *str* FROM *str_table* produces plan LogicalProject(STR=[$0]) LogicalProject(STR=[$0.name], STR1=[$0.age]) LogicalTableScan(table=[[CATALOG, STRUCT, STR_TABLE]]) where top level project returns nested field `name` as `str` instead of original struct column. My question is, what is the correct way to collect back the flattened fields and produce correct result for the query ? [1] - https://github.com/ihuzenko/calcite/commit/e24eaa22fbb5c950a0bd5290cc09ca56ea7f1e44 Thank you in advance, Igor Guzenko
[jira] [Created] (CALCITE-2639) FilterReduceExpressionsRule causes ArithmeticException at execution time
Igor Guzenko created CALCITE-2639:
-
Summary: FilterReduceExpressionsRule causes ArithmeticException at
execution time
Key: CALCITE-2639
URL: https://issues.apache.org/jira/browse/CALCITE-2639
Project: Calcite
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Igor Guzenko
Assignee: Julian Hyde
Consider logical plan generated for test case(put this in {{RelOptRuleTest}}):
{code:java}
@Test public void testOversimplifiedCaseStatement() {
HepProgram program = new HepProgramBuilder()
.addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
.build();
String sql = "select * from emp "
+ "where MGR > 0 and "
+ "case when MGR > 0 then deptno / MGR else null end > 1";
checkPlanning(program, sql);
}
{code}
Before applying ReduceExpressionsRule.FILTER_INSTANCE rule, query plan is
{code:java}
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
LogicalFilter(condition=[AND(>($3, 0), >(CASE(>($3, 0), /($7, $3), null),
1))])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
Plan after applying the rule:
{code:java}
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
LogicalFilter(condition=[AND(>($3, 0), CASE(IS NOT NULL($3), >(/($7, $3), 1),
false))])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
Here {{LogicalFilter}} has condition {{AND(>($3, 0), CASE(IS NOT NULL($3),
>(/($7, $3), 1), false))}} where {{CASE}} condition was replaced by {{IS NOT
NULL($3)}} condition. Since {{AND}} allows permutations for operands, the first
operand may become {{CASE}}, so query may fail with {{ArithmeticException: / by
zero}} error at execution stage.
The regression was caused by
[CALCITE-1413|https://github.com/apache/calcite/commit/b470a0cd4572c9f6c4c0e9b51926b97c5af58d3f].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
