[jira] [Updated] (CALCITE-4439) how to avoid query database column meta when I execute select query
[
https://issues.apache.org/jira/browse/CALCITE-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wudejin updated CALCITE-4439:
-
Issue Type: Bug (was: Improvement)
Priority: Blocker (was: Major)
> how to avoid query database column meta when I execute select query
> ---
>
> Key: CALCITE-4439
> URL: https://issues.apache.org/jira/browse/CALCITE-4439
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: wudejin
>Priority: Blocker
>
> I use calcite jdbc to query greenplum, code as follow：
> {code:java}
> Class.forName("org.apache.calcite.jdbc.Driver");
> Properties info = new Properties();
> info.setProperty("lex", "JAVA");
> Connection connection =
> DriverManager.getConnection("jdbc:calcite:", info);
> CalciteConnection calciteConnection =
> connection.unwrap(CalciteConnection.class);
> SchemaPlus rootSchema = calciteConnection.getRootSchema();
> Class.forName("com.mysql.cj.jdbc.Driver");
> BasicDataSource dataSource = new BasicDataSource();
> dataSource.setUrl("jdbc:postgresql://192.168.125.3:5432/sit_cicada_0819");
> dataSource.setUsername(USER);
> dataSource.setPassword(PASSWORD);
> Schema schema = JdbcSchema.create(rootSchema, "cicada", dataSource, null,
> null);
> rootSchema.add("cicada", schema);
> calciteConnection.setSchema("cicada");
> Statement statement = calciteConnection.createStatement();
> ResultSet resultSet = statement.executeQuery("select zqsl, y_w, sspcs from
> t_rh_aj_zq_jtj where ssfj='WQFJ' and n_w = '2020'");
> print(resultSet);
> resultSet.close();
> statement.close();
> connection.close();
> {code}
> this query cost 20 seconds, but i use navicat to run above sql only took
> 0.041 s.
>
> I use arthas to analysis the code consumed time：
> {code:java}
> [arthas@35681]$ trace --skipJDKMethod false -n 1
> org.apache.calcite.sql.validate.SqlValidatorImpl validateNamespace
> Press Q or Ctrl+C to abort.
> Affect(class count: 2 , method count: 1) cost in 471 ms, listenerId:
> 16
> `---ts=2020-12-14
> 17:33:21;thread_name=http-nio--exec-3;id=59;is_daemon=true;priority=5;TCCL=com.code.plugin.loader.PluginClassLoader@2e7620e1
> `---[14841.944034ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> +---[14841.716027ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
> | `---[14831.796817ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> | +---[14831.624599ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
> | | `---[0.458664ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> | | +---[0.176725ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
> | | `---[0.012963ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
> | +---[0.006568ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
> | +---[0.011847ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1007
> | +---[0.029899ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getType() #57
> | `---[0.04292ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:setValidatedNodeType() #57
> +---[0.011767ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
> +---[0.009111ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1007
> +---[0.030568ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:getType() #57
> `---[0.014533ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:setValidatedNodeType() #57
> [arthas@35681]$ trace --skipJDKMethod false -n 1
> org.apache.calcite.sql.validate.SqlValidatorImpl validateNamespace
> Press Q or Ctrl+C to abort.
> Affect(class count: 2 , method count: 1) cost in 471 ms, listenerId:
> 16
> `---ts=2020-12-14
> 17:33:21;thread_name=http-nio--exec-3;id=59;is_daemon=true;priority=5;TCCL=com.code.plugin.loader.PluginClassLoader@2e7620e1
> `---[14841.944034ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> +---[14841.716027ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
> | `---[14831.796817ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> | +---[14831.624599ms]
> org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
> | | `---[0.458664ms]
> org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
> | | +---[0.176725ms]
>
[jira] [Created] (CALCITE-4439) how to avoid query database column meta when I execute select query
wudejin created CALCITE-4439:
Summary: how to avoid query database column meta when I execute
select query
Key: CALCITE-4439
URL: https://issues.apache.org/jira/browse/CALCITE-4439
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.21.0
Reporter: wudejin
I use calcite jdbc to query greenplum, code as follow：
{code:java}
Class.forName("org.apache.calcite.jdbc.Driver");
Properties info = new Properties();
info.setProperty("lex", "JAVA");
Connection connection =
DriverManager.getConnection("jdbc:calcite:", info);
CalciteConnection calciteConnection =
connection.unwrap(CalciteConnection.class);
SchemaPlus rootSchema = calciteConnection.getRootSchema();
Class.forName("com.mysql.cj.jdbc.Driver");
BasicDataSource dataSource = new BasicDataSource();
dataSource.setUrl("jdbc:postgresql://192.168.125.3:5432/sit_cicada_0819");
dataSource.setUsername(USER);
dataSource.setPassword(PASSWORD);
Schema schema = JdbcSchema.create(rootSchema, "cicada", dataSource, null, null);
rootSchema.add("cicada", schema);
calciteConnection.setSchema("cicada");
Statement statement = calciteConnection.createStatement();
ResultSet resultSet = statement.executeQuery("select zqsl, y_w, sspcs from
t_rh_aj_zq_jtj where ssfj='WQFJ' and n_w = '2020'");
print(resultSet);
resultSet.close();
statement.close();
connection.close();
{code}
this query cost 20 seconds, but i use navicat to run above sql only took 0.041
s.
I use arthas to analysis the code consumed time：
{code:java}
[arthas@35681]$ trace --skipJDKMethod false -n 1
org.apache.calcite.sql.validate.SqlValidatorImpl validateNamespace
Press Q or Ctrl+C to abort.
Affect(class count: 2 , method count: 1) cost in 471 ms, listenerId: 16
`---ts=2020-12-14
17:33:21;thread_name=http-nio--exec-3;id=59;is_daemon=true;priority=5;TCCL=com.code.plugin.loader.PluginClassLoader@2e7620e1
`---[14841.944034ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
+---[14841.716027ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| `---[14831.796817ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
| +---[14831.624599ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| | `---[0.458664ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
| | +---[0.176725ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| | `---[0.012963ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
| +---[0.006568ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
| +---[0.011847ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1007
| +---[0.029899ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getType() #57
| `---[0.04292ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:setValidatedNodeType() #57
+---[0.011767ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
+---[0.009111ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1007
+---[0.030568ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getType() #57
`---[0.014533ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:setValidatedNodeType() #57
[arthas@35681]$ trace --skipJDKMethod false -n 1
org.apache.calcite.sql.validate.SqlValidatorImpl validateNamespace
Press Q or Ctrl+C to abort.
Affect(class count: 2 , method count: 1) cost in 471 ms, listenerId: 16
`---ts=2020-12-14
17:33:21;thread_name=http-nio--exec-3;id=59;is_daemon=true;priority=5;TCCL=com.code.plugin.loader.PluginClassLoader@2e7620e1
`---[14841.944034ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
+---[14841.716027ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| `---[14831.796817ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
| +---[14831.624599ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| | `---[0.458664ms]
org.apache.calcite.sql.validate.SqlValidatorImpl:validateNamespace()
| | +---[0.176725ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:validate() #1005
| | `---[0.012963ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
| +---[0.006568ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1006
| +---[0.011847ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getNode() #1007
| +---[0.029899ms]
org.apache.calcite.sql.validate.SqlValidatorNamespace:getType() #57
| `---[0.04292ms]
[jira] [Commented] (CALCITE-4438) Calcite SQLParser: Improve support for parsing Spark SQL - INSERT OVERWRITE, RLIKE,DATE grammar
[
https://issues.apache.org/jira/browse/CALCITE-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250005#comment-17250005
]
shradha commented on CALCITE-4438:
--
Sure . Thanks [~julianhyde] and [~zabetak] for your feedback. I'll
incorporate your feedback and create separate PR's
> Calcite SQLParser: Improve support for parsing Spark SQL - INSERT OVERWRITE,
> RLIKE,DATE grammar
> ---
>
> Key: CALCITE-4438
> URL: https://issues.apache.org/jira/browse/CALCITE-4438
> Project: Calcite
> Issue Type: Improvement
> Components: spark
>Reporter: shradha
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE,DAY,YEAR,MONTH. I am using 1.26.0 version of
> calcite-core and calcite-server.
> Also it throws error if year,day,month,identity,value,date is used as alias
> in sql query
> For example -
> 1) DATE QUERIES
> {color:#808080}
> {color} {color:#80}val {color}query1 = {color:#008000}"select
> DATE(gns_date) as dt"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Incorrect syntax near the keyword 'DATE' at line 1, column 10.{color}
> {color:#008000}
> {color} {color:#80}val {color}query3 = {color:#008000}"select
> date('2020-11-07') as date"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "date (" at line 1, column 8.{color}
> {color:#008000} {color:#80}val {color}query3 = "select loan_id as year,
> as_of_date as date"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "year" at line 1, column 19.{color}
>
> {color:#008000}val query4="{color}{color:#008000}select \{D'1990-01-01'} as
> day,a as month from table{color}{color:#008000}"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "day" at line 1, column 27.{color}
> {color:#008000}2) INSERT OVERWRITE{color}
> {color:#008000}INSERT OVERWRITE TABLE sbg_schema.tableA
> select distinct SURV_MARK_CUST_IDEN
> ,latest_surv_oci
> ,tran_mark_cust_iden
> ,surv_tran_mci
> ,orgz_mark_cust_iden from tableB
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "OVERWRITE" at line 2, column 8.{color}
> {color:#008000}3) RLIKE{color}
> {color:#80}val {color}query1 = {color:#008000}"select cola from tableA
> where MAX(realm_email) rlike
> '.+@.+{color}{color:#80}{color}{color:#008000}..+'"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "rlike" at line 1, column 48.{color}
>
> {color:#008000}All these queries are correctly being parsed via SPARK SQL .
> But calcite's parser grammer file doesn't suport these tokens{color}
>
> *{color:#008000}This is the code for creating sqlParser instance:{color}*
> {color:#008000}object Spark3SqlDialect{
> val DefaultContext: SqlDialect.Context = SqlDialect.EMPTY_CONTEXT
> .withDatabaseProduct(SqlDialect.DatabaseProduct.SPARK)
> .withNullCollation(NullCollation.LOW)
> .withCaseSensitive(false)
> .withConformance(SqlConformanceEnum.BABEL)
> .withIdentifierQuoteString("`")
> .withQuotedCasing(Casing.UNCHANGED)
> .withUnquotedCasing(Casing.UNCHANGED)
> val DEFAULT = new SparkSqlDialect(DefaultContext)
> }{color}
> {color:#008000}
> val sqlParser: SqlParser = SqlParser.create(sqlQuery,
> createSqlParserConfig()){color}
> {color:#008000}
> private def createSqlParserConfig() = {
> Spark3SqlDialect.DEFAULT.configureParser(SqlParser.config()
> .withParserFactory(SqlDdlParserImpl.FACTORY))
> .withConformance(SqlConformanceEnum.BABEL)
> }{color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4438) Calcite SQLParser: Improve support for parsing Spark SQL - INSERT OVERWRITE, RLIKE,DATE grammar
[
https://issues.apache.org/jira/browse/CALCITE-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shradha updated CALCITE-4438:
-
Summary: Calcite SQLParser: Improve support for parsing Spark SQL - INSERT
OVERWRITE, RLIKE,DATE grammar (was: Calcite SQLParser: Calcite throwing error
while parsing Spark SQL syntax - INSERT OVERWRITE, RLIKE,DATE )
> Calcite SQLParser: Improve support for parsing Spark SQL - INSERT OVERWRITE,
> RLIKE,DATE grammar
> ---
>
> Key: CALCITE-4438
> URL: https://issues.apache.org/jira/browse/CALCITE-4438
> Project: Calcite
> Issue Type: Improvement
> Components: spark
>Reporter: shradha
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE,DAY,YEAR,MONTH. I am using 1.26.0 version of
> calcite-core and calcite-server.
> Also it throws error if year,day,month,identity,value,date is used as alias
> in sql query
> For example -
> 1) DATE QUERIES
> {color:#808080}
> {color} {color:#80}val {color}query1 = {color:#008000}"select
> DATE(gns_date) as dt"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Incorrect syntax near the keyword 'DATE' at line 1, column 10.{color}
> {color:#008000}
> {color} {color:#80}val {color}query3 = {color:#008000}"select
> date('2020-11-07') as date"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "date (" at line 1, column 8.{color}
> {color:#008000} {color:#80}val {color}query3 = "select loan_id as year,
> as_of_date as date"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "year" at line 1, column 19.{color}
>
> {color:#008000}val query4="{color}{color:#008000}select \{D'1990-01-01'} as
> day,a as month from table{color}{color:#008000}"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "day" at line 1, column 27.{color}
> {color:#008000}2) INSERT OVERWRITE{color}
> {color:#008000}INSERT OVERWRITE TABLE sbg_schema.tableA
> select distinct SURV_MARK_CUST_IDEN
> ,latest_surv_oci
> ,tran_mark_cust_iden
> ,surv_tran_mci
> ,orgz_mark_cust_iden from tableB
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "OVERWRITE" at line 2, column 8.{color}
> {color:#008000}3) RLIKE{color}
> {color:#80}val {color}query1 = {color:#008000}"select cola from tableA
> where MAX(realm_email) rlike
> '.+@.+{color}{color:#80}{color}{color:#008000}..+'"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "rlike" at line 1, column 48.{color}
>
> {color:#008000}All these queries are correctly being parsed via SPARK SQL .
> But calcite's parser grammer file doesn't suport these tokens{color}
>
> *{color:#008000}This is the code for creating sqlParser instance:{color}*
> {color:#008000}object Spark3SqlDialect{
> val DefaultContext: SqlDialect.Context = SqlDialect.EMPTY_CONTEXT
> .withDatabaseProduct(SqlDialect.DatabaseProduct.SPARK)
> .withNullCollation(NullCollation.LOW)
> .withCaseSensitive(false)
> .withConformance(SqlConformanceEnum.BABEL)
> .withIdentifierQuoteString("`")
> .withQuotedCasing(Casing.UNCHANGED)
> .withUnquotedCasing(Casing.UNCHANGED)
> val DEFAULT = new SparkSqlDialect(DefaultContext)
> }{color}
> {color:#008000}
> val sqlParser: SqlParser = SqlParser.create(sqlQuery,
> createSqlParserConfig()){color}
> {color:#008000}
> private def createSqlParserConfig() = {
> Spark3SqlDialect.DEFAULT.configureParser(SqlParser.config()
> .withParserFactory(SqlDdlParserImpl.FACTORY))
> .withConformance(SqlConformanceEnum.BABEL)
> }{color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-207) Support recursive/reference data types
[
https://issues.apache.org/jira/browse/CALCITE-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249942#comment-17249942
]
Julian Hyde commented on CALCITE-207:
-
We don't have branches in the main repo anymore. Therefore I have deleted
branch {{origin/recursive_types}} and created a branch
[julianhyde/207-recursive-types|https://github.com/julianhyde/calcite/tree/207-recursive-types]
with the same contents.
> Support recursive/reference data types
> --
>
> Key: CALCITE-207
> URL: https://issues.apache.org/jira/browse/CALCITE-207
> Project: Calcite
> Issue Type: Bug
>Reporter: GitHub Import
>Priority: Major
> Labels: github-import
>
> For Eclipse memory analyzer plugin it is required to process references
> somehow, enable navigation (e.g. `where emp.manager.manager.name='boss'`)
> Currently `StackOverflowError` occurs when trying self-referencing data type
> (see relevant code at
> https://github.com/julianhyde/optiq/blob/450562d268d37e8b24421117d1834e5b8dffdda4/core/src/main/java/net/hydromatic/optiq/jdbc/JavaTypeFactoryImpl.java#L60)
> ```java
> class Employee {
> Employee manager;
> ...
> }
> ```
> `StackOverflowError`
> ```
> at
> net.hydromatic.optiq.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:118)
> at
> net.hydromatic.optiq.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:76)
> at
> net.hydromatic.optiq.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:118)
> at
> net.hydromatic.optiq.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:76)
> ```
> Imported from GitHub
> Url: https://github.com/julianhyde/optiq/issues/207
> Created by: [vlsi|https://github.com/vlsi]
> Labels:
> Assignee: [julianhyde|https://github.com/julianhyde]
> Created at: Tue Mar 25 15:34:25 CET 2014
> State: open
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4438) Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax - INSERT OVERWRITE, RLIKE,DATE
[
https://issues.apache.org/jira/browse/CALCITE-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249925#comment-17249925
]
Julian Hyde commented on CALCITE-4438:
--
I agree with [~zabetak]'s comments. Expanding a little:
* Definitely keep these as 3 independent commits. Smaller changes are easier to
review. I would suggest making them separate PRs on separate branches, and
creating 3 sub-cases in JIRA.
* Change the heading of this case so that it reads like a feature request not a
bug, e.g. 'Improve support for parsing Spark SQL'.
* Eliminate the spurious changes, e.g. those that Stamatis noted in
{{config.fmpp}}.
* {{RLIKE}} has operator syntax, not function syntax, so it does need parser
changes. But it should be in Babel parser, not core parser.
* It would be good if there were tests in SqlOperatorBaseTest for RLIKE showing
expressions and expected results. Then someone could go an implement it.
* I added support for the DATE function in Babel in CALCITE-3022. You should
build on that.
* I have not scrutinized the PR, but extra tests and documentation are probably
needed.
In summary: we definitely want these contributions. These will be useful to our
community. But it will take few iterations, so be patient.
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE
>
>
> Key: CALCITE-4438
> URL: https://issues.apache.org/jira/browse/CALCITE-4438
> Project: Calcite
> Issue Type: Improvement
> Components: spark
>Reporter: shradha
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE,DAY,YEAR,MONTH. I am using 1.26.0 version of
> calcite-core and calcite-server.
> Also it throws error if year,day,month,identity,value,date is used as alias
> in sql query
> For example -
> 1) DATE QUERIES
> {color:#808080}
> {color} {color:#80}val {color}query1 = {color:#008000}"select
> DATE(gns_date) as dt"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Incorrect syntax near the keyword 'DATE' at line 1, column 10.{color}
> {color:#008000}
> {color} {color:#80}val {color}query3 = {color:#008000}"select
> date('2020-11-07') as date"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "date (" at line 1, column 8.{color}
> {color:#008000} {color:#80}val {color}query3 = "select loan_id as year,
> as_of_date as date"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "year" at line 1, column 19.{color}
>
> {color:#008000}val query4="{color}{color:#008000}select \{D'1990-01-01'} as
> day,a as month from table{color}{color:#008000}"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "day" at line 1, column 27.{color}
> {color:#008000}2) INSERT OVERWRITE{color}
> {color:#008000}INSERT OVERWRITE TABLE sbg_schema.tableA
> select distinct SURV_MARK_CUST_IDEN
> ,latest_surv_oci
> ,tran_mark_cust_iden
> ,surv_tran_mci
> ,orgz_mark_cust_iden from tableB
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "OVERWRITE" at line 2, column 8.{color}
> {color:#008000}3) RLIKE{color}
> {color:#80}val {color}query1 = {color:#008000}"select cola from tableA
> where MAX(realm_email) rlike
> '.+@.+{color}{color:#80}{color}{color:#008000}..+'"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "rlike" at line 1, column 48.{color}
>
> {color:#008000}All these queries are correctly being parsed via SPARK SQL .
> But calcite's parser grammer file doesn't suport these tokens{color}
>
> *{color:#008000}This is the code for creating sqlParser instance:{color}*
> {color:#008000}object Spark3SqlDialect{
> val DefaultContext: SqlDialect.Context = SqlDialect.EMPTY_CONTEXT
> .withDatabaseProduct(SqlDialect.DatabaseProduct.SPARK)
>
[jira] [Commented] (CALCITE-4432) When the RelNode's input is the same subset as the node belonged to, not choose this node as best.
[ https://issues.apache.org/jira/browse/CALCITE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249874#comment-17249874 ] Julian Hyde commented on CALCITE-4432: -- That sounds plausible. Do you have a solution for people not using the top-down driver? Besides “use the top-down driver”. > When the RelNode's input is the same subset as the node belonged to, not > choose this node as best. > -- > > Key: CALCITE-4432 > URL: https://issues.apache.org/jira/browse/CALCITE-4432 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Ziwei Liu >Assignee: Ziwei Liu >Priority: Major > > If a subset have a cyclic node, the node's input is this subset itself. If > the beset -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-520) Support cancel in remote JDBC driver
[ https://issues.apache.org/jira/browse/CALCITE-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249871#comment-17249871 ] Julian Hyde commented on CALCITE-520: - The problem is setting CancelFlag remotely. > Support cancel in remote JDBC driver > > > Key: CALCITE-520 > URL: https://issues.apache.org/jira/browse/CALCITE-520 > Project: Calcite > Issue Type: Bug > Components: avatica >Reporter: Julian Hyde >Priority: Major > > Remote JDBC driver should support canceling running statements. > I believe we have the ingredients already. The client knows the statement id > of a statement before it executes it. And when cancel is received in the > client JDBC driver, it can send a (new) CancelStatement request on a separate > RPC call. The server can then set the cancel flag, and from that point, the > mechanism is the same as local cancel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures
[
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249864#comment-17249864
]
Josh Elser commented on CALCITE-4367:
-
Thanks for reporting, John. Will see if we can get the docs updated.
> Incorrect documentation for Avatica JSON request/response signatures
>
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Reporter: John Bodley
>Priority: Trivial
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html]
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
> and thus requires a {{response}} field. Note I'm not certain if this was
> intentional, i.e., it being a response, however it it is it seems that it
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}}
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are
> optional rather than required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4438) Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax - INSERT OVERWRITE, RLIKE,DATE
[
https://issues.apache.org/jira/browse/CALCITE-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249716#comment-17249716
]
Stamatis Zampetakis commented on CALCITE-4438:
--
Thanks for working on this [~sambekar]. I had a quick look in the PR and heres
a few general comments:
* Smaller contributions are easier to review and merge so when possible break
them down to different JIRAs/PRs.
* Non SQL standard syntax/functions, such as INSERT OVERWRITE, generally go to
the babel module and not core.
* Non standard functions such as RLIKE generally do not need to be added to the
parser (check
[SqlLibraryOperators|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java])
When the core is extended with non-standard features there should be some
justification.
Why there are so many changes in core/src/main/codegen/config.fmpp ?
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE
>
>
> Key: CALCITE-4438
> URL: https://issues.apache.org/jira/browse/CALCITE-4438
> Project: Calcite
> Issue Type: Improvement
> Components: spark
>Reporter: shradha
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Calcite SQLParser: Calcite throwing error while parsing Spark SQL syntax -
> INSERT OVERWRITE, RLIKE,DATE,DAY,YEAR,MONTH. I am using 1.26.0 version of
> calcite-core and calcite-server.
> Also it throws error if year,day,month,identity,value,date is used as alias
> in sql query
> For example -
> 1) DATE QUERIES
> {color:#808080}
> {color} {color:#80}val {color}query1 = {color:#008000}"select
> DATE(gns_date) as dt"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Incorrect syntax near the keyword 'DATE' at line 1, column 10.{color}
> {color:#008000}
> {color} {color:#80}val {color}query3 = {color:#008000}"select
> date('2020-11-07') as date"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "date (" at line 1, column 8.{color}
> {color:#008000} {color:#80}val {color}query3 = "select loan_id as year,
> as_of_date as date"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "year" at line 1, column 19.{color}
>
> {color:#008000}val query4="{color}{color:#008000}select \{D'1990-01-01'} as
> day,a as month from table{color}{color:#008000}"{color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "day" at line 1, column 27.{color}
> {color:#008000}2) INSERT OVERWRITE{color}
> {color:#008000}INSERT OVERWRITE TABLE sbg_schema.tableA
> select distinct SURV_MARK_CUST_IDEN
> ,latest_surv_oci
> ,tran_mark_cust_iden
> ,surv_tran_mci
> ,orgz_mark_cust_iden from tableB
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "OVERWRITE" at line 2, column 8.{color}
> {color:#008000}3) RLIKE{color}
> {color:#80}val {color}query1 = {color:#008000}"select cola from tableA
> where MAX(realm_email) rlike
> '.+@.+{color}{color:#80}{color}{color:#008000}..+'"
> {color}
> {color:#008000}StatementMetadataFragment(com.intuit.superglue.pipeline.parsers.CalciteStatementParser,UNKNOWN,List(),List(),List(org.apache.calcite.sql.parser.SqlParseException:
> Encountered "rlike" at line 1, column 48.{color}
>
> {color:#008000}All these queries are correctly being parsed via SPARK SQL .
> But calcite's parser grammer file doesn't suport these tokens{color}
>
> *{color:#008000}This is the code for creating sqlParser instance:{color}*
> {color:#008000}object Spark3SqlDialect{
> val DefaultContext: SqlDialect.Context = SqlDialect.EMPTY_CONTEXT
> .withDatabaseProduct(SqlDialect.DatabaseProduct.SPARK)
> .withNullCollation(NullCollation.LOW)
> .withCaseSensitive(false)
> .withConformance(SqlConformanceEnum.BABEL)
> .withIdentifierQuoteString("`")
> .withQuotedCasing(Casing.UNCHANGED)
> .withUnquotedCasing(Casing.UNCHANGED)
> val DEFAULT = new SparkSqlDialect(DefaultContext)
> }{color}
> {color:#008000}
> val sqlParser: SqlParser =
[jira] [Updated] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4437:
---
Description:
The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort with
fetch/offset being impossible to decorrelate in all cases.
CALCITE-4333 addressed this issue but only partially (when the Sort with
fetch/offset is on top on the plan). However, this solution is insufficient,
because any Sort with fetch/offset that is not inside a Correlate can be
decorrelated.
Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just with
an extra LogicalProject on top of the LogicalSort):
{code}
@Test void testProjectSortLimitWithCorrelateInput() {
final String sql = ""
+ "SELECT ename||deptno FROM\n"
+ "(SELECT deptno, ename\n"
+ "FROM\n"
+ "(SELECT DISTINCT deptno FROM emp) t1,\n"
+ " LATERAL (\n"
+ "SELECT ename, sal\n"
+ "FROM emp\n"
+ "WHERE deptno = t1.deptno)\n"
+ "ORDER BY ename DESC\n"
+ "LIMIT 3)";
sql(sql).ok();
}
{code}
The current plan is:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{0}])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5])
LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
It can actually decorrelated as:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
was:
The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort with
fetch/offset being impossible to decorrelate in all cases.
CALCITE-4333 addressed this issue but only partially (when the Sort with
fetch/offset is on top on the plan). However, this solution is insufficient,
because any Sort with fetch/offset that is not inside a Correlate can be
decorrelated.
Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just with
an extra LogicalProject on top of the LogicalSort):
{code}
@Test void testSortLimitWithCorrelateInput2() {
final String sql = ""
+ "SELECT ename||deptno FROM\n"
+ "(SELECT deptno, ename\n"
+ "FROM\n"
+ "(SELECT DISTINCT deptno FROM emp) t1,\n"
+ " LATERAL (\n"
+ "SELECT ename, sal\n"
+ "FROM emp\n"
+ "WHERE deptno = t1.deptno)\n"
+ "ORDER BY ename DESC\n"
+ "LIMIT 3)";
sql(sql).ok();
}
{code}
The current plan is:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{0}])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5])
LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
It can actually decorrelated as:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
> Fix For:
