[jira] [Comment Edited] (CALCITE-2312) Support Partition By in sql select statement
[
https://issues.apache.org/jira/browse/CALCITE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478405#comment-16478405
]
Fei Xu edited comment on CALCITE-2312 at 5/17/18 9:41 AM:
--
Thanks for the comment.
So, may be I can start from SqlCreateTable, add PARTITION BY strategy, just
like adding PRIMARY KEY. SqlCreateTable represents a relation table, maybe
SqlCreateStream is more suitable for stream.
was (Author: xu fei):
Thanks for the comment.
So, may be I can start from SqlCreateTable, add PARTITION BY strategy, just
like adding PRIMARY KEY.
> Support Partition By in sql select statement
>
>
> Key: CALCITE-2312
> URL: https://issues.apache.org/jira/browse/CALCITE-2312
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I noticed that calcite already have an Exchange RelNode represents
> distribution on RelNode's input, but no sql clause support.
>
> We are use calcite building SQL layer for out streaming platform. and in
> streaming computation, data shuffle is a very important function, not only
> for our engine, but also for our users.
> For example, In stream-join-table, the engine will load the table data to a
> cache at runtime, and stream join table is actually stream look up table.
> If the stream data could partition by hash or range, before look up table. It
> will be cache friendly cause particular stream data look up particular table
> data.
>
> So I consider do some extensions on sql clause to support Exchange, e.g.
> {code:java}
> // shuffle data using hash_distribution
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY o.productId
> // shuffle data to a singleton node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY SINGLETON
> // shuffle data to all nodes
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY BROADCAST
> // shuffle data to random node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY RANDOM
> // shuffle data using round-robin policy
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY ROUND_ROBIN
> // shuffle data using range policy
> // Current I'm not sure about the appropriate clause to represents range
> // shuffle, so it is just an demo.
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY RANGE o.productId, 0, 4096
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2312) Support Partition By in sql select statement
Fei Xu created CALCITE-2312:
---
Summary: Support Partition By in sql select statement
Key: CALCITE-2312
URL: https://issues.apache.org/jira/browse/CALCITE-2312
Project: Calcite
Issue Type: New Feature
Components: core
Reporter: Fei Xu
Assignee: Julian Hyde
I noticed that calcite already have an Exchange RelNode represents distribution
on RelNode's input, but no sql clause support.
We are use calcite building SQL layer for out streaming platform. and in
streaming computation, data shuffle is a very important function, not only for
our engine, but also for our users.
For example, In stream-join-table, the engine will load the table data to a
cache at runtime, and stream join table is actually stream look up table.
If the stream data could partition by hash or range, before look up table. It
will be cache friendly cause particular stream data look up particular table
data.
So I consider do some extensions on sql clause to support Exchange, e.g.
{code:java}
// shuffle data using hash_distribution
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY o.productId
// shuffle data to a singleton node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY SINGLETON
// shuffle data to all nodes
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY BROADCAST
// shuffle data to random node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANDOM
// shuffle data using round-robin policy
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY ROUND_ROBIN
// shuffle data using range policy
// Current I'm not sure about the appropriate clause to represents range
// shuffle, so it is just an demo.
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANGE o.productId, 0, 4096
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2312) Support Partition By in sql select statement
[
https://issues.apache.org/jira/browse/CALCITE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Xu updated CALCITE-2312:
Description:
I noticed that calcite already have an Exchange RelNode represents distribution
on RelNode's input, but no sql clause support.
We are use calcite building SQL layer for out streaming platform. and in
streaming computation, data shuffle is a very important function, not only for
our engine, but also for our users.
For example, In stream-join-table, the engine will load the table data to a
cache at runtime, and stream join table is actually stream look up table.
If the stream data could partition by hash or range, before look up table. It
will be cache friendly cause particular stream data look up particular table
data.
So I consider do some extensions on sql clause to support Exchange, e.g.
{code:java}
// shuffle data using hash_distribution
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY o.productId
// shuffle data to a singleton node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY SINGLETON
// shuffle data to all nodes
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY BROADCAST
// shuffle data to random node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANDOM
// shuffle data using round-robin policy
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY ROUND_ROBIN
// shuffle data using range policy
// Current I'm not sure about the appropriate clause to represents range
// shuffle, so it is just an demo.
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANGE o.productId, 0, 4096
{code}
was:
I noticed that calcite already have an Exchange RelNode represents distribution
on RelNode's input, but no sql clause support.
We are use calcite building SQL layer for out streaming platform. and in
streaming computation, data shuffle is a very important function, not only for
our engine, but also for our users.
For example, In stream-join-table, the engine will load the table data to a
cache at runtime, and stream join table is actually stream look up table.
If the stream data could partition by hash or range, before look up table. It
will be cache friendly cause particular stream data look up particular table
data.
So I consider do some extensions on sql clause to support Exchange, e.g.
{code:java}
// shuffle data using hash_distribution
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY o.productId
// shuffle data to a singleton node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY SINGLETON
// shuffle data to all nodes
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY BROADCAST
// shuffle data to random node
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANDOM
// shuffle data using round-robin policy
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY ROUND_ROBIN
// shuffle data using range policy
// Current I'm not sure about the appropriate clause to represents range
// shuffle, so it is just an demo.
INSERT INTO output
SELECT
*
FROM orders o
PARTITION BY RANGE o.productId, 0, 4096
{code}
> Support Partition By in sql select statement
>
>
> Key: CALCITE-2312
> URL: https://issues.apache.org/jira/browse/CALCITE-2312
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I noticed that calcite already have an Exchange RelNode represents
> distribution on RelNode's input, but no sql clause support.
>
> We are use calcite building SQL layer for out streaming platform. and in
> streaming computation, data shuffle is a very important function, not only
> for our engine, but also for our users.
> For example, In stream-join-table, the engine will load the table data to a
> cache at runtime, and stream join table is actually stream look up table.
> If the stream data could partition by hash or range, before look up table. It
> will be cache friendly cause particular stream data look up particular table
> data.
>
> So I consider do some extensions on sql clause to support Exchange, e.g.
> {code:java}
> // shuffle data using hash_distribution
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY o.productId
> // shuffle data to a singleton node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY SINGLETON
> // shuffle data to all nodes
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY BROADCAST
> // shuffle data to random node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY RANDOM
> // shuffle data using round-robin policy
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY ROUND_ROBIN
> // shuffle data using range policy
> // Current I'm not sure about the appropriate
[jira] [Commented] (CALCITE-2312) Support Partition By in sql select statement
[
https://issues.apache.org/jira/browse/CALCITE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478405#comment-16478405
]
Fei Xu commented on CALCITE-2312:
-
Thanks for the comment.
So, may be I can start from SqlCreateTable, add PARTITION BY strategy, just
like adding PRIMARY KEY.
> Support Partition By in sql select statement
>
>
> Key: CALCITE-2312
> URL: https://issues.apache.org/jira/browse/CALCITE-2312
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I noticed that calcite already have an Exchange RelNode represents
> distribution on RelNode's input, but no sql clause support.
>
> We are use calcite building SQL layer for out streaming platform. and in
> streaming computation, data shuffle is a very important function, not only
> for our engine, but also for our users.
> For example, In stream-join-table, the engine will load the table data to a
> cache at runtime, and stream join table is actually stream look up table.
> If the stream data could partition by hash or range, before look up table. It
> will be cache friendly cause particular stream data look up particular table
> data.
>
> So I consider do some extensions on sql clause to support Exchange, e.g.
> {code:java}
> // shuffle data using hash_distribution
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY o.productId
> // shuffle data to a singleton node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY SINGLETON
> // shuffle data to all nodes
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY BROADCAST
> // shuffle data to random node
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY RANDOM
> // shuffle data using round-robin policy
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY ROUND_ROBIN
> // shuffle data using range policy
> // Current I'm not sure about the appropriate clause to represents range
> // shuffle, so it is just an demo.
> INSERT INTO output
> SELECT
> *
> FROM orders o
> PARTITION BY RANGE o.productId, 0, 4096
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2310) SqlParser parse multiple sql statements split by semicolon
[
https://issues.apache.org/jira/browse/CALCITE-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476787#comment-16476787
]
Fei Xu commented on CALCITE-2310:
-
Put this extension to 'server' module is OK. But do you think we really need
add BEGIN and ENG for parse a block ? cause single statement can be a special
case of sequences statements, e.g.
{code:java}
/**
* Parses Sequences SQL statement followed by the end-of-file symbol.
*/
List seqSqlStmtEof() :
{
List list = new ArrayList();
SqlNode stmt;
}
{
stmt = SqlStmt() { list.add(stmt); }
( stmt = SqlStmt() {list.add(stmt);} ) *
[ ]
{
return list;
}
}
{code}
> SqlParser parse multiple sql statements split by semicolon
> --
>
> Key: CALCITE-2310
> URL: https://issues.apache.org/jira/browse/CALCITE-2310
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Minor
>
> Current, SqlParser only supports parse single sql statement to single
> SqlNode. But since we have server module, and support DDL syntax, It is
> common to write DDL statement and DML statement together to build a complete
> logic.
> For example:
> * Table orders is a source table;
> * Table output_console is a sink table;
> * Read data from source table, do some project and filter, then write to
> sink table.
> {code:java}
> CREATE TABLE orders (
> createTime TIMESTAMP,
> productId bigint,
> orderId bigint,
> units bigint,
> user_name VARCHAR
> );
> CREATE TABLE output_console(
> createTime TIMESTAMP,
> productId bigint,
> orderId bigint,
> units bigint,
> user_name VARCHAR
> );
> INSERT INTO output_console
> SELECT
> createTime,
> productId,
> orderId,
> units,
> user_name
> FROM orders
> WHERE (productId>3) AND (productId<8);
> {code}
> So, I think it really helps if SqlParser support parse multiple sql
> statements.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2336) SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495941#comment-16495941
]
Fei Xu commented on CALCITE-2336:
-
thanks julian，I’ll try this way
Julian Hyde (JIRA) 于2018年5月31日 周四上午6:31写道：
> SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2336
> URL: https://issues.apache.org/jira/browse/CALCITE-2336
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I register a table "users" with a single column, age. And try to insert two
> columns into the "users".
> {code:java}
> rootSchema = Frameworks.createRootSchema(true);
> rootSchema.add("USERS", new AbstractTable() {
> @Override public RelDataType getRowType(final RelDataTypeFactory
> typeFactory) {
> RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
> builder.add("age", new BasicSqlType(new RelDataTypeSystemImpl() {},
> SqlTypeName.CHAR));
> return builder.build();
> }
> });
> final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.Config.DEFAULT)
> .defaultSchema(rootSchema)
> .build();
> planner = Frameworks.getPlanner(config);
> dataContext = new MyDataContext(planner);
> SqlNode parse =
> planner.parse("insert into users select y, x\n"
> + "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
> + "where x > 1");
> SqlNode validate = planner.validate(parse);
> {code}
> Apparently, I want to see some error message like:
> {code:java}
> Number of INSERT target columns (1) does not equal number of source items (2)
> {code}
> But actually, I got message:
> {code:java}
> org.apache.calcite.tools.ValidationException:
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> {code}
> which was confused.
> Then I debug the code in SqlValidatorImpl#validateSelectList method.
>
> {code:java}
> protected RelDataType validateSelectList(
> final SqlNodeList selectItems,
> SqlSelect select,
> RelDataType targetRowType) {
> // First pass, ensure that aliases are unique. "*" and "TABLE.*" items
> // are ignored.
> // Validate SELECT list. Expand terms of the form "*" or "TABLE.*".
> final SqlValidatorScope selectScope = getSelectScope(select);
> final List expandedSelectItems = new ArrayList<>();
> final Set aliases = Sets.newHashSet();
> final List> fieldList = new ArrayList<>();
> for (int i = 0; i < selectItems.size(); i++) {
> SqlNode selectItem = selectItems.get(i);
> if (selectItem instanceof SqlSelect) {
> handleScalarSubQuery(
> select,
> (SqlSelect) selectItem,
> expandedSelectItems,
> aliases,
> fieldList);
> } else {
> expandSelectItem(
> selectItem,
> select,
> targetRowType.isStruct()
> && targetRowType.getFieldCount() >= i
> ? targetRowType.getFieldList().get(i).getType()
> : unknownType,
> expandedSelectItems,
> aliases,
> fieldList,
> false);
> }
> }
> {code}
> See the exception is throw from here, if selectItems's size more than
> targetRowType's field count
> {code:java}
> && targetRowType.getFieldCount() >= i
> ? targetRowType.getFieldList().get(i).getType(){code}
> When I change it to this, I get what I need.
> {code:java}
> && targetRowType.getFieldCount() - 1 >= i
> ? targetRowType.getFieldList().get(i).getType(){code}
> So is this a Bug ? Do we need fix it ?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2336) SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496013#comment-16496013
]
Fei Xu commented on CALCITE-2336:
-
Hi Julian, I change the code, e.g. but it seems doesn't help.
{code:java}
rootSchema = Frameworks.createRootSchema(true);
rootSchema.add("USERS", new AbstractTable() {
@Override public RelDataType getRowType(final RelDataTypeFactory
typeFactory) {
RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
//builder.add("name", new BasicSqlType(new RelDataTypeSystemImpl() {},
SqlTypeName.BIGINT));
builder.add("name", typeFactory.createSqlType(SqlTypeName.BIGINT));
return builder.build();
}
});
final FrameworkConfig config = Frameworks.newConfigBuilder()
.parserConfig(SqlParser.Config.DEFAULT)
.defaultSchema(rootSchema)
.build();
planner = Frameworks.getPlanner(config);
dataContext = new MyDataContext(planner);
SqlNode parse =
planner.parse("insert into users select y, x\n"
+ "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
+ "where x > 1");
SqlNode validate = planner.validate(parse);
RelNode convert = planner.rel(validate).rel;
{code}
I still get the message
{code:java}
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than
size (1)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
at
com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:41)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:3993)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3251)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateInsert(SqlValidatorImpl.java:4165)
at org.apache.calcite.sql.SqlInsert.validate(SqlInsert.java:148)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
at org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:194)
... 25 more
{code}
> SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2336
> URL: https://issues.apache.org/jira/browse/CALCITE-2336
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I register a table "users" with a single column, age. And try to insert two
> columns into the "users".
> {code:java}
> rootSchema = Frameworks.createRootSchema(true);
> rootSchema.add("USERS", new AbstractTable() {
> @Override public RelDataType getRowType(final RelDataTypeFactory
> typeFactory) {
> RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
> builder.add("age", new BasicSqlType(new RelDataTypeSystemImpl() {},
> SqlTypeName.CHAR));
> return builder.build();
> }
> });
> final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.Config.DEFAULT)
> .defaultSchema(rootSchema)
> .build();
> planner = Frameworks.getPlanner(config);
> dataContext = new MyDataContext(planner);
> SqlNode parse =
> planner.parse("insert into users select y, x\n"
> + "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
> + "where x > 1");
> SqlNode validate = planner.validate(parse);
> {code}
> Apparently, I want to see some error message like:
> {code:java}
> Number of INSERT target columns (1) does not equal number of source items (2)
> {code}
> But actually, I got message:
> {code:java}
> org.apache.calcite.tools.ValidationException:
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> {code}
> which was confused.
> Then I debug the code in SqlValidatorImpl#validateSelectList method.
>
> {code:java}
> protected RelDataType validateSelectList(
> final SqlNodeList selectItems,
> SqlSelect select,
> RelDataType targetRowType) {
> // First pass, ensure that aliases are unique. "*" and "TABLE.*" items
> // are ignored.
> // Validate SELECT list. Expand terms of the form "*" or "TABLE.*".
> final SqlValidatorScope selectScope = getSelectScope(select);
> final List expandedSelectItems = new ArrayList<>();
> final Set aliases = Sets.newHashSet();
> final List> fieldList = new ArrayList<>();
> for (int i = 0; i < selectItems.size(); i++) {
> SqlNode selectItem = selectItems.get(i);
> if (selectItem instanceof SqlSelect) {
>
[jira] [Commented] (CALCITE-2336) SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496143#comment-16496143
]
Fei Xu commented on CALCITE-2336:
-
Well, would you please execute the following unit test in
org.apache.calcite.test.InterpreterTest to verify ?, I found the problem when I
use calcite in my own application. So I write a temporary test case.
{code:java}
@Test public void testIndexOutOfBoundsException() throws Exception {
rootSchema = Frameworks.createRootSchema(true);
rootSchema.add("USERS", new AbstractTable() {
@Override public RelDataType getRowType(final RelDataTypeFactory
typeFactory) {
RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
//builder.add("name", new BasicSqlType(new RelDataTypeSystemImpl() {},
SqlTypeName.BIGINT));
builder.add("name", typeFactory.createSqlType(SqlTypeName.INTEGER));
return builder.build();
}
});
final FrameworkConfig config = Frameworks.newConfigBuilder()
.parserConfig(SqlParser.Config.DEFAULT)
.defaultSchema(rootSchema)
.build();
planner = Frameworks.getPlanner(config);
dataContext = new MyDataContext(planner);
SqlNode parse =
planner.parse("insert into users select y, x\n"
+ "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
+ "where x > 1");
SqlNode validate = planner.validate(parse);
//RelNode convert = planner.rel(validate).rel;
//final Interpreter interpreter = new Interpreter(dataContext, convert);
//assertRows(interpreter, "[0]", "[10]", "[20]", "[30]");
}
{code}
> SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2336
> URL: https://issues.apache.org/jira/browse/CALCITE-2336
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
>
> I register a table "users" with a single column, age. And try to insert two
> columns into the "users".
> {code:java}
> rootSchema = Frameworks.createRootSchema(true);
> rootSchema.add("USERS", new AbstractTable() {
> @Override public RelDataType getRowType(final RelDataTypeFactory
> typeFactory) {
> RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
> builder.add("age", new BasicSqlType(new RelDataTypeSystemImpl() {},
> SqlTypeName.CHAR));
> return builder.build();
> }
> });
> final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.Config.DEFAULT)
> .defaultSchema(rootSchema)
> .build();
> planner = Frameworks.getPlanner(config);
> dataContext = new MyDataContext(planner);
> SqlNode parse =
> planner.parse("insert into users select y, x\n"
> + "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
> + "where x > 1");
> SqlNode validate = planner.validate(parse);
> {code}
> Apparently, I want to see some error message like:
> {code:java}
> Number of INSERT target columns (1) does not equal number of source items (2)
> {code}
> But actually, I got message:
> {code:java}
> org.apache.calcite.tools.ValidationException:
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> {code}
> which was confused.
> Then I debug the code in SqlValidatorImpl#validateSelectList method.
>
> {code:java}
> protected RelDataType validateSelectList(
> final SqlNodeList selectItems,
> SqlSelect select,
> RelDataType targetRowType) {
> // First pass, ensure that aliases are unique. "*" and "TABLE.*" items
> // are ignored.
> // Validate SELECT list. Expand terms of the form "*" or "TABLE.*".
> final SqlValidatorScope selectScope = getSelectScope(select);
> final List expandedSelectItems = new ArrayList<>();
> final Set aliases = Sets.newHashSet();
> final List> fieldList = new ArrayList<>();
> for (int i = 0; i < selectItems.size(); i++) {
> SqlNode selectItem = selectItems.get(i);
> if (selectItem instanceof SqlSelect) {
> handleScalarSubQuery(
> select,
> (SqlSelect) selectItem,
> expandedSelectItems,
> aliases,
> fieldList);
> } else {
> expandSelectItem(
> selectItem,
> select,
> targetRowType.isStruct()
> && targetRowType.getFieldCount() >= i
> ? targetRowType.getFieldList().get(i).getType()
> : unknownType,
> expandedSelectItems,
> aliases,
> fieldList,
> false);
> }
> }
> {code}
> See the exception is throw from here, if selectItems's size more than
> targetRowType's field count
> {code:java}
> &&
[jira] [Created] (CALCITE-2336) SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
Fei Xu created CALCITE-2336:
---
Summary: SqlValidatorImpl throws
java.lang.IndexOutOfBoundsException
Key: CALCITE-2336
URL: https://issues.apache.org/jira/browse/CALCITE-2336
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Fei Xu
Assignee: Julian Hyde
I register a table "users" with a single column, age. And try to insert two
columns into the "users".
{code:java}
rootSchema = Frameworks.createRootSchema(true);
rootSchema.add("USERS", new AbstractTable() {
@Override public RelDataType getRowType(final RelDataTypeFactory
typeFactory) {
RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
builder.add("age", new BasicSqlType(new RelDataTypeSystemImpl() {},
SqlTypeName.CHAR));
return builder.build();
}
});
final FrameworkConfig config = Frameworks.newConfigBuilder()
.parserConfig(SqlParser.Config.DEFAULT)
.defaultSchema(rootSchema)
.build();
planner = Frameworks.getPlanner(config);
dataContext = new MyDataContext(planner);
SqlNode parse =
planner.parse("insert into users select y, x\n"
+ "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
+ "where x > 1");
SqlNode validate = planner.validate(parse);
{code}
Apparently, I want to see some error message like:
{code:java}
Number of INSERT target columns (1) does not equal number of source items (2)
{code}
But actually, I got message:
{code:java}
org.apache.calcite.tools.ValidationException:
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
{code}
which was confused.
Then I debug the code in SqlValidatorImpl#validateSelectList method.
{code:java}
protected RelDataType validateSelectList(
final SqlNodeList selectItems,
SqlSelect select,
RelDataType targetRowType) {
// First pass, ensure that aliases are unique. "*" and "TABLE.*" items
// are ignored.
// Validate SELECT list. Expand terms of the form "*" or "TABLE.*".
final SqlValidatorScope selectScope = getSelectScope(select);
final List expandedSelectItems = new ArrayList<>();
final Set aliases = Sets.newHashSet();
final List> fieldList = new ArrayList<>();
for (int i = 0; i < selectItems.size(); i++) {
SqlNode selectItem = selectItems.get(i);
if (selectItem instanceof SqlSelect) {
handleScalarSubQuery(
select,
(SqlSelect) selectItem,
expandedSelectItems,
aliases,
fieldList);
} else {
expandSelectItem(
selectItem,
select,
targetRowType.isStruct()
&& targetRowType.getFieldCount() >= i
? targetRowType.getFieldList().get(i).getType()
: unknownType,
expandedSelectItems,
aliases,
fieldList,
false);
}
}
{code}
See the exception is throw from here, if selectItems's size more than
targetRowType's field count
{code:java}
&& targetRowType.getFieldCount() >= i
? targetRowType.getFieldList().get(i).getType(){code}
When I change it to this, I get what I need.
{code:java}
&& targetRowType.getFieldCount() - 1 >= i
? targetRowType.getFieldList().get(i).getType(){code}
So is this a Bug ? Do we need fix it ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2336) SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497486#comment-16497486
]
Fei Xu commented on CALCITE-2336:
-
Thanks, It's more clear
> SqlValidatorImpl throws java.lang.IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2336
> URL: https://issues.apache.org/jira/browse/CALCITE-2336
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Fei Xu
>Assignee: Julian Hyde
>Priority: Major
> Attachments: 1.patch
>
>
> I register a table "users" with a single column, age. And try to insert two
> columns into the "users".
> {code:java}
> rootSchema = Frameworks.createRootSchema(true);
> rootSchema.add("USERS", new AbstractTable() {
> @Override public RelDataType getRowType(final RelDataTypeFactory
> typeFactory) {
> RelDataTypeFactory.FieldInfoBuilder builder = typeFactory.builder();
> builder.add("age", new BasicSqlType(new RelDataTypeSystemImpl() {},
> SqlTypeName.CHAR));
> return builder.build();
> }
> });
> final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.Config.DEFAULT)
> .defaultSchema(rootSchema)
> .build();
> planner = Frameworks.getPlanner(config);
> dataContext = new MyDataContext(planner);
> SqlNode parse =
> planner.parse("insert into users select y, x\n"
> + "from (values (1, 'a'), (2, 'b'), (3, 'c')) as t(x, y)\n"
> + "where x > 1");
> SqlNode validate = planner.validate(parse);
> {code}
> Apparently, I want to see some error message like:
> {code:java}
> Number of INSERT target columns (1) does not equal number of source items (2)
> {code}
> But actually, I got message:
> {code:java}
> org.apache.calcite.tools.ValidationException:
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> {code}
> which was confused.
> Then I debug the code in SqlValidatorImpl#validateSelectList method.
>
> {code:java}
> protected RelDataType validateSelectList(
> final SqlNodeList selectItems,
> SqlSelect select,
> RelDataType targetRowType) {
> // First pass, ensure that aliases are unique. "*" and "TABLE.*" items
> // are ignored.
> // Validate SELECT list. Expand terms of the form "*" or "TABLE.*".
> final SqlValidatorScope selectScope = getSelectScope(select);
> final List expandedSelectItems = new ArrayList<>();
> final Set aliases = Sets.newHashSet();
> final List> fieldList = new ArrayList<>();
> for (int i = 0; i < selectItems.size(); i++) {
> SqlNode selectItem = selectItems.get(i);
> if (selectItem instanceof SqlSelect) {
> handleScalarSubQuery(
> select,
> (SqlSelect) selectItem,
> expandedSelectItems,
> aliases,
> fieldList);
> } else {
> expandSelectItem(
> selectItem,
> select,
> targetRowType.isStruct()
> && targetRowType.getFieldCount() >= i
> ? targetRowType.getFieldList().get(i).getType()
> : unknownType,
> expandedSelectItems,
> aliases,
> fieldList,
> false);
> }
> }
> {code}
> See the exception is throw from here, if selectItems's size more than
> targetRowType's field count
> {code:java}
> && targetRowType.getFieldCount() >= i
> ? targetRowType.getFieldList().get(i).getType(){code}
> When I change it to this, I get what I need.
> {code:java}
> && targetRowType.getFieldCount() - 1 >= i
> ? targetRowType.getFieldList().get(i).getType(){code}
> So is this a Bug ? Do we need fix it ?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
