[jira] [Created] (FLINK-35406) Use inner serializer when casting RAW type to BINARY or STRING in cast rules
Zhenghua Gao created FLINK-35406: Summary: Use inner serializer when casting RAW type to BINARY or STRING in cast rules Key: FLINK-35406 URL: https://issues.apache.org/jira/browse/FLINK-35406 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.19.0 Reporter: Zhenghua Gao The generated code in RawToStringCastRule and RawToBinaryCastRule use BinaryRawValueData::toBytes and BinaryRawValueData::toObject to convert RawValueData(to java object or byte array), which should use inner serializer instead of RawValueDataSerializer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-21026) Align column list specification with Hive in INSERT statement
Zhenghua Gao created FLINK-21026:
Summary: Align column list specification with Hive in INSERT
statement
Key: FLINK-21026
URL: https://issues.apache.org/jira/browse/FLINK-21026
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Zhenghua Gao
[HIVE-9481|https://issues.apache.org/jira/browse/HIVE-9481] allows column list
specification in INSERT statement. The syntax is:
{code:java}
INSERT INTO TABLE table_name
[PARTITION (partcol1[=val1], partcol2[=val2] ...)]
[(column list)]
select_statement FROM from_statement
{code}
In the MeanWhile, flink introduces PARTITION syntax that the PARTITION clause
appears after the COLUMN LIST clause. It looks weird and luckily we don't
support COLUMN LIST clause
now[FLINK-18726|https://issues.apache.org/jira/browse/FLINK-18726]. I think
it'a good change to align this with Hive now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17113) Refactor view support in SQL Client
Zhenghua Gao created FLINK-17113: Summary: Refactor view support in SQL Client Key: FLINK-17113 URL: https://issues.apache.org/jira/browse/FLINK-17113 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17112) Support DESCRIBE VIEW view_name in Flink SQL
Zhenghua Gao created FLINK-17112: Summary: Support DESCRIBE VIEW view_name in Flink SQL Key: FLINK-17112 URL: https://issues.apache.org/jira/browse/FLINK-17112 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17111) Support SHOW VIEWS in Flink SQL
Zhenghua Gao created FLINK-17111: Summary: Support SHOW VIEWS in Flink SQL Key: FLINK-17111 URL: https://issues.apache.org/jira/browse/FLINK-17111 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17106) Support create/drop view in Flink SQL
Zhenghua Gao created FLINK-17106: Summary: Support create/drop view in Flink SQL Key: FLINK-17106 URL: https://issues.apache.org/jira/browse/FLINK-17106 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17105) FLIP-71: E2E viewsupport
Zhenghua Gao created FLINK-17105: Summary: FLIP-71: E2E viewsupport Key: FLINK-17105 URL: https://issues.apache.org/jira/browse/FLINK-17105 Project: Flink Issue Type: New Feature Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17098) CatalogManager#dropTemporaryTable and dropTemporaryView should use ObjectIdentifier as its argument
Zhenghua Gao created FLINK-17098: Summary: CatalogManager#dropTemporaryTable and dropTemporaryView should use ObjectIdentifier as its argument Key: FLINK-17098 URL: https://issues.apache.org/jira/browse/FLINK-17098 Project: Flink Issue Type: Improvement Components: Table SQL / API Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Since CatalogManager#createTable, createTemporaryTable and dropTable use the given fully qualified ObjectIdentifier to create or drop tables/temporary tables, we should also use ObjectIdentifier (instead of UnresolvedIdentifier) in dropTemporaryTable and dropTemporaryView. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17067) CatalogManager#createTable and createTemporaryTable should provide consistent semantics
Zhenghua Gao created FLINK-17067: Summary: CatalogManager#createTable and createTemporaryTable should provide consistent semantics Key: FLINK-17067 URL: https://issues.apache.org/jira/browse/FLINK-17067 Project: Flink Issue Type: Improvement Components: Table SQL / API Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Currently CatalogManager#createTable provides [IF NOT EXISTS] semantic and CatalogManager#createTemporaryTable provides [OR REPLACE] semantic. IMO they should provide consistent semantics: [IF NOT EXISTS] or [OR REPLACE] or BOTH. I prefer [IF NOT EXISTS] since we didn't support [OR REPLACE] in table DDL(and view DDL) currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16800) TypeMappingUtils#checkPhysicalLogicalTypeCompatible didn't deal with nested types
Zhenghua Gao created FLINK-16800:
Summary: TypeMappingUtils#checkPhysicalLogicalTypeCompatible
didn't deal with nested types
Key: FLINK-16800
URL: https://issues.apache.org/jira/browse/FLINK-16800
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.11.0
the planner will use TypeMappingUtils#checkPhysicalLogicalTypeCompatible to
validate logical schema and physical schema are compatible when translate
CatalogSinkModifyOperation to Calcite relational expression. The validation
didn't deal with nested types well, which could expose the following
ValidationException:
{code:java}
Exception in thread "main" org.apache.flink.table.api.ValidationException:
Type ARRAY> of table field 'old'
does not match with the physical type ARRAY> of the 'old' field of the TableSource return
type.
at
org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:164)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:277)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:254)
at
org.apache.flink.table.types.logical.utils.LogicalTypeDefaultVisitor.visit(LogicalTypeDefaultVisitor.java:157)
at org.apache.flink.table.types.logical.ArrayType.accept(ArrayType.java:110)
at
org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappingUtils.java:254)
at
org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:160)
at
org.apache.flink.table.utils.TypeMappingUtils.lambda$computeInCompositeType$8(TypeMappingUtils.java:232)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.flink.table.utils.TypeMappingUtils.computeInCompositeType(TypeMappingUtils.java:214)
at
org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndices(TypeMappingUtils.java:192)
at
org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndicesOrTimeAttributeMarkers(TypeMappingUtils.java:112)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.computeIndexMapping(StreamExecTableSourceScan.scala:212)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:107)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:62)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlan(StreamExecTableSourceScan.scala:62)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:84)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:44)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlan(StreamExecExchange.scala:44)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:161)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:51)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlan(StreamExecLimit.scala:51)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToTransformation(StreamExecSink.scala:184)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:153)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:48)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlan(StreamExecSink.scala:48)
at
[jira] [Created] (FLINK-16327) Add TableEnvironment.fromElements interfaces for usability
Zhenghua Gao created FLINK-16327:
Summary: Add TableEnvironment.fromElements interfaces for usability
Key: FLINK-16327
URL: https://issues.apache.org/jira/browse/FLINK-16327
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Zhenghua Gao
h1. Interface
{code:java}
/**
* Creates a table from a group of objects (known as its elements). The
schema of the table
* would be inferred from the type of elements.
*
* @param data a group of objects.
*/
Table fromElements(Collection data);
/**
* Creates a table from a group of objects (known as its elements). The
schema of the table
* would be inferred from the passed in data type.
*
* @param data a group of objects
* @param dataType the data type of the data
*/
Table fromElements(Collection data, DataType dataType);
{code}
h1. Use Case
* One potential use case for Table API
{code:java}
@Test
def testUnregisteredCollectionSource1(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
tEnv.fromElements(data.asJava)
.as('first, 'id, 'score, 'last)
.where('id > 4)
.select('last, 'score * 2)
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
@Test
def testUnregisteredCollectionSource2(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val dataType = DataTypes.ROW(
DataTypes.FIELD("first", DataTypes.STRING()),
DataTypes.FIELD("id", DataTypes.INT()),
DataTypes.FIELD("score", DataTypes.DOUBLE()),
DataTypes.FIELD("last", DataTypes.STRING()))
tEnv.fromElements(data.asJava, dataType)
.where('id > 4)
.select('last, 'score * 2)
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
{code}
* One potential use case for SQL
{code:java}
@Test
def testUnregisteredCollectionSource1(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val table = tEnv.fromElements(data.asJava).as('first, 'id, 'score, 'last)
tEnv.createTemporaryView("T", table)
tEnv.sqlQuery("SELECT last, score * 2 FROM T WHERE id > 4")
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
@Test
def testUnregisteredCollectionSource2(): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
StreamITCase.testResults = mutable.MutableList()
val data = Seq(
Row.of("Mike", new JInt(5), new JDouble(12.3), "Smith"))
val dataType = DataTypes.ROW(
DataTypes.FIELD("first", DataTypes.STRING()),
DataTypes.FIELD("id", DataTypes.INT()),
DataTypes.FIELD("score", DataTypes.DOUBLE()),
DataTypes.FIELD("last", DataTypes.STRING()))
val table = tEnv.fromElements(data.asJava, dataType)
tEnv.createTemporaryView("T", table)
tEnv.sqlQuery("SELECT last, score * 2 FROM T WHERE id > 4")
.toAppendStream[Row]
.addSink(new StreamITCase.StringSink[Row])
env.execute()
}
{code}
h1. The proposal
* data type inference
We need to infer the data type from the data for the first interface. A
potential tool is the DataTypeExtractor, but it doesn't support scala.tuple,
Row, etc. For the most popular in our test cases Row or scala.tuple type, we
could enumerate and use a recursive traversal method to get all available types
of underlying objects. This can solve most of the cases and improve usability.
* proposed changes
** A CollectionQueryOperation which implements QueryOperation to describe the
relational operation
** The logical and physical RelNode for legacy planner. In the physical node,
we can translate the data to DataStream
** The logical and physical RelNode for blink planner. In the physical node,
we can translate the data to Transformation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16160) Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path
Zhenghua Gao created FLINK-16160: Summary: Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path Key: FLINK-16160 URL: https://issues.apache.org/jira/browse/FLINK-16160 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao In ConnectTableDescriptor#createTemporaryTable, the proctime/rowtime properties are ignored so the generated catalog table is not correct. We should fix this to let TableEnvironment#connect() support watermark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16117) Avoid register source in TableTestBase#addTableSource
Zhenghua Gao created FLINK-16117: Summary: Avoid register source in TableTestBase#addTableSource Key: FLINK-16117 URL: https://issues.apache.org/jira/browse/FLINK-16117 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao This affects thousands of unit tests: 1) explainSourceAsString of CatalogSourceTable changes 2)JoinTest#testUDFInJoinCondition: SQL keywords must be escaped 3) GroupWindowTest#testTimestampEventTimeTumblingGroupWindowWithProperties: Reference to a rowtime or proctime window required 4) SetOperatorsTest#testInWithProject: legacy type vs new type -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16029) Remove register source and sink in test cases of planner
Zhenghua Gao created FLINK-16029: Summary: Remove register source and sink in test cases of planner Key: FLINK-16029 URL: https://issues.apache.org/jira/browse/FLINK-16029 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao Many test cases of planner use TableEnvironement.registerTableSource() and registerTableSink() which should be avoid。We want to refactor these cases via TableEnvironment.connect(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15968) LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO
Zhenghua Gao created FLINK-15968: Summary: LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO Key: FLINK-15968 URL: https://issues.apache.org/jira/browse/FLINK-15968 Project: Flink Issue Type: Bug Components: Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.11.0 Reporter: Zhenghua Gao Currently LegacyTypeInfoDataTypeConverter only support conversion between DataTypes.BYTES and BYTE_PRIMITIVE_ARRAY_TYPE_INFO. When we update connectors to new type system, we need to convert BINARY(n) or VARBINARY(n) to BYTE_PRIMITIVE_ARRAY_TYPE_INFO. The Hive connector achieve this via depending blink planner‘s conversion logic, which is odd because a planner dependency won't be necessary for connectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15525) HBase connector should use new type system to suppport precision/scale
Zhenghua Gao created FLINK-15525: Summary: HBase connector should use new type system to suppport precision/scale Key: FLINK-15525 URL: https://issues.apache.org/jira/browse/FLINK-15525 Project: Flink Issue Type: Bug Components: Connectors / HBase Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 Currently *HBaseTableSchema* use TypeInformation to specify an HBase table's schema, which would cause precision/scale loss for several data types. Meanwhile *HBaseTypeUtils* serialize a java.sql.Timestamp to long in bytes, which would cause precision loss for TIMESTAMP types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15469) UpsertStreamTableSink should support new type system
Zhenghua Gao created FLINK-15469:
Summary: UpsertStreamTableSink should support new type system
Key: FLINK-15469
URL: https://issues.apache.org/jira/browse/FLINK-15469
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.11.0
Currently *UpsertStreamTableSink* can only returns TypeInformation of the
requested record, which can't support types with precision and scale, e.g.
TIMESTAMP(p), DECIMAL(p,s).
A proposal is deprecating the *getRecordType* API and adding a
*getRecordDataType* API instead to return the data type of the requested record.
{code:java}
/**
* Returns the requested record type.
*
* @Deprecated This method will be removed in future versions. It's recommended
to use {@link #getRecordDataType()} instead.
*/
@Deprecated
TypeInformation getRecordType();
/*
* Returns the requested record data type.
*/
DataType getRecordDataType();
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15460) planner dependencies won't be necessary for JDBC connector
Zhenghua Gao created FLINK-15460: Summary: planner dependencies won't be necessary for JDBC connector Key: FLINK-15460 URL: https://issues.apache.org/jira/browse/FLINK-15460 Project: Flink Issue Type: Improvement Components: Connectors / HBase, Connectors / JDBC Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15445) JDBC Table Source didn't work for Types with precision (or/and scale)
Zhenghua Gao created FLINK-15445: Summary: JDBC Table Source didn't work for Types with precision (or/and scale) Key: FLINK-15445 URL: https://issues.apache.org/jira/browse/FLINK-15445 Project: Flink Issue Type: Bug Components: Connectors / JDBC Affects Versions: 1.10.0 Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15231) Wrong HeapVector in AbstractHeapVector.createHeapColumn
Zhenghua Gao created FLINK-15231: Summary: Wrong HeapVector in AbstractHeapVector.createHeapColumn Key: FLINK-15231 URL: https://issues.apache.org/jira/browse/FLINK-15231 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 For TIMESTAMP WITHOUT TIME ZONE/TIMESTAMP WITH LOCAL TIME ZONE/DECIMAL types, AbstractHeapVector.createHeapColumn generates wrong HeapVectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15213) The conversion between java.sql.Timestamp and long is not asymmetric
Zhenghua Gao created FLINK-15213:
Summary: The conversion between java.sql.Timestamp and long is not
asymmetric
Key: FLINK-15213
URL: https://issues.apache.org/jira/browse/FLINK-15213
Project: Flink
Issue Type: Bug
Reporter: Zhenghua Gao
In Calcite, we use SqlFunctions.toLong(Timestamp) and
SqlFunctions.internalToTimestamp(long) to convert java.sql.Timestmap to
internal long and vice versa. The main logical inside is +/- local time zone
offset.
But in the comments of TimeZone.getOffset(long date), the parameter represents
in milliseconds since January 1, 1970 00:00:00 GMT. It means that there will
one conversion above doesn't satisfy this hypothesis.
This causes many surprise to users:
(1) some Daylight Saving Time changes:
{code:java}
@Test public void testDayLightingSaving() {
TimeZone tz = TimeZone.getDefault();
TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"));
java.sql.Timestamp dst2018Begin = java.sql.Timestamp.valueOf("2018-03-11
03:00:00");
assertThat(dst2018Begin, is(internalToTimestamp(toLong(dst2018Begin;
TimeZone.setDefault(tz);
}{code}
fails with:
{code:java}
java.lang.AssertionError:
Expected: is <2018-03-11 04:00:00.0>
but: was <2018-03-11 03:00:00.0>
Expected :is <2018-03-11 04:00:00.0>
Actual :<2018-03-11 03:00:00.0>{code}
(2) "1900-01-01 00:00:00" Changes in some TimeZone
{code:java}
@Test public void test() {
TimeZone tz = TimeZone.getDefault();
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"));
java.sql.Timestamp ts = java.sql.Timestamp.valueOf("1900-01-01 00:00:00");
assertThat(ts, is(internalToTimestamp(toLong(ts;
TimeZone.setDefault(tz);
}{code}
fails with
{code:java}
java.lang.AssertionError:
Expected: is <1899-12-31 23:54:17.0>
but: was <1900-01-01 00:00:00.0>
Expected :is <1899-12-31 23:54:17.0>
Actual :<1900-01-01 00:00:00.0>
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15151) Use new type system in TableSourceUtil.computeIndexMapping of blink planner
Zhenghua Gao created FLINK-15151: Summary: Use new type system in TableSourceUtil.computeIndexMapping of blink planner Key: FLINK-15151 URL: https://issues.apache.org/jira/browse/FLINK-15151 Project: Flink Issue Type: Bug Components: Table SQL / Planner Reporter: Zhenghua Gao We should use new type system instead of TypeInformation in TableSourceUtil.computeIndexMapping in blink planner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14959) Support precision of LocalZonedTimestampType in blink planner
Zhenghua Gao created FLINK-14959: Summary: Support precision of LocalZonedTimestampType in blink planner Key: FLINK-14959 URL: https://issues.apache.org/jira/browse/FLINK-14959 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14927) Remove LegacyTimestampTypeInfo and LegacyLocalDateTimeTypeInfo when the conversion is not needed
Zhenghua Gao created FLINK-14927: Summary: Remove LegacyTimestampTypeInfo and LegacyLocalDateTimeTypeInfo when the conversion is not needed Key: FLINK-14927 URL: https://issues.apache.org/jira/browse/FLINK-14927 Project: Flink Issue Type: Improvement Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14925) the return type of TO_TIMESTAMP should be Timestamp(9) instead of Timestamp(3)
Zhenghua Gao created FLINK-14925: Summary: the return type of TO_TIMESTAMP should be Timestamp(9) instead of Timestamp(3) Key: FLINK-14925 URL: https://issues.apache.org/jira/browse/FLINK-14925 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14889) TIMESTAMPADD/TIMESTAMPDIFF with microsecond/nanosecond unit lost precision
Zhenghua Gao created FLINK-14889: Summary: TIMESTAMPADD/TIMESTAMPDIFF with microsecond/nanosecond unit lost precision Key: FLINK-14889 URL: https://issues.apache.org/jira/browse/FLINK-14889 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao Since the TimestampAddConvertlet and TimestampDiffConvertlet tread TIMESTAMP as long (with millisecond precision), they lost precision even if the downstream can support microsecond or nanosecond. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14848) BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part
Zhenghua Gao created FLINK-14848: Summary: BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part Key: FLINK-14848 URL: https://issues.apache.org/jira/browse/FLINK-14848 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao For the fixed-length objects, the writer calls setNullAt() to update fixed-length part（which set null bits and initialize fixed-length part with 0; For the variable-length objects, the writer calls setNullAt to update fixed-length part and need to assign & initialize variable-length part -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14810) It's weird that copy julianDateFloor from DateTimeUtils and change the implementation
Zhenghua Gao created FLINK-14810: Summary: It's weird that copy julianDateFloor from DateTimeUtils and change the implementation Key: FLINK-14810 URL: https://issues.apache.org/jira/browse/FLINK-14810 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Zhenghua Gao In SqlDateTimeUtils we copied *julianToLocalDate* from DateTimeUtils and changed the implementation. It's weird to use an *entirely* *different* implementation to do the same thing. One possible improvement of the new one is support TimeUnitRange.QUARTER. But the logic to calculate year/month/day is totally different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14806) Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter
Zhenghua Gao created FLINK-14806: Summary: Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter Key: FLINK-14806 URL: https://issues.apache.org/jira/browse/FLINK-14806 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 Since FLINK-14080 introduce a new representation of TimestampType, the binary format should add timestamp related interface to set/get TimestampType objects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14805) Remove unixDateCeil related code
Zhenghua Gao created FLINK-14805: Summary: Remove unixDateCeil related code Key: FLINK-14805 URL: https://issues.apache.org/jira/browse/FLINK-14805 Project: Flink Issue Type: Improvement Components: Table SQL / Legacy Planner, Table SQL / Planner Reporter: Zhenghua Gao FLINK-11935 removed DateTimeUtils and add unixDateCeil related code to fix CEIL(date) (CALCITE-3199). CALCITE-3199 has merged to avatica-1.16.0 and we can remove the copied code when we upgraded to avatica-1.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14764) The final Explicit/Implicit conversion matrix we should support in our planner
Zhenghua Gao created FLINK-14764: Summary: The final Explicit/Implicit conversion matrix we should support in our planner Key: FLINK-14764 URL: https://issues.apache.org/jira/browse/FLINK-14764 Project: Flink Issue Type: Improvement Components: Table SQL / Legacy Planner, Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 The SQL standard defines the cast specification with an explicit conversion matrix (SQL 2011 Part 2 Section 6.13 Syntax Rules 6)). But neither Legacy planner nor blink planner would follow that. IMO we should determine a final Explicit/Implicit conversion matrix before 1.10 (at least in blink planner). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14737) HiveTypeUtil.toFlinkPrimitiveType returns wrong Timestamp(6) instead of Timestamp(9)
Zhenghua Gao created FLINK-14737: Summary: HiveTypeUtil.toFlinkPrimitiveType returns wrong Timestamp(6) instead of Timestamp(9) Key: FLINK-14737 URL: https://issues.apache.org/jira/browse/FLINK-14737 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Zhenghua Gao Hive's Timestamp type holds nanosecond's precision, and when transfer to Flink typesystem, it's should be Timestamp(9). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14696) Support precision of TimestampType in built-in SQL functions and operators
Zhenghua Gao created FLINK-14696: Summary: Support precision of TimestampType in built-in SQL functions and operators Key: FLINK-14696 URL: https://issues.apache.org/jira/browse/FLINK-14696 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Zhenghua Gao Many built-in SQL functions and operators use long as internal representation of Timestamp type and only support millisecond precision. This ticket will check fix it and let them support nanosecond precision. The related SQL functions and operators are: (To Be Confirmed) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14650) Thread safety issue in the piece of code example of dev/stream/testing document
Zhenghua Gao created FLINK-14650: Summary: Thread safety issue in the piece of code example of dev/stream/testing document Key: FLINK-14650 URL: https://issues.apache.org/jira/browse/FLINK-14650 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Zhenghua Gao As mentioned by Gilles in user ML[1], the piece of code example has thread safety issue. One possibility is use Collections.synchronizedList() to create a thread-safety list and remove the synchronized keyword. [1] [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Documentation-issue-maybe-td30929.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14599) Support precision of TimestampType
Zhenghua Gao created FLINK-14599: Summary: Support precision of TimestampType Key: FLINK-14599 URL: https://issues.apache.org/jira/browse/FLINK-14599 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Zhenghua Gao Since FLINK-14080 introduced an internal representation(SqlTimestamp) of TimestampType with precision. This subtask will replace current long with SqlTimestamp, and let blink planner support precision of TimestampType. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14535) Fix distinct key type for DecimalType in DistinctInfo
Zhenghua Gao created FLINK-14535: Summary: Fix distinct key type for DecimalType in DistinctInfo Key: FLINK-14535 URL: https://issues.apache.org/jira/browse/FLINK-14535 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.1 Reporter: Zhenghua Gao Fix For: 1.10.0 DecimalType in DistinctInfo bridged to wrong external BigDecimal type, which causes failures count distinct on decimal type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-13665) decimal(p, s) where p is less than s should be illegal
Zhenghua Gao created FLINK-13665:
Summary: decimal(p, s) where p is less than s should be illegal
Key: FLINK-13665
URL: https://issues.apache.org/jira/browse/FLINK-13665
Project: Flink
Issue Type: Bug
Components: Table SQL / Legacy Planner
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhenghua Gao
Fix For: 1.10.0
e.g.: {{cast(42.345 as decimal(2, 3)) should get a ValidationException}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (FLINK-13664) "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING
Zhenghua Gao created FLINK-13664: Summary: "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING Key: FLINK-13664 URL: https://issues.apache.org/jira/browse/FLINK-13664 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao Both planners do not support MD5(binary), this should fast fail at validate phrase and give users more meaningful messages. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13651) table api not support cast to decimal with precision and scale
Zhenghua Gao created FLINK-13651: Summary: table api not support cast to decimal with precision and scale Key: FLINK-13651 URL: https://issues.apache.org/jira/browse/FLINK-13651 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao could reproduce in ScalarFunctionsTest: `testAllApis( 'f31.cast(DataTypes.DECIMAL(38, 18)).truncate(2), "f31.cast(DECIMAL(10, 10)).truncate(2)", "truncate(cast(f31 as decimal(38, 18)), 2)", "-0.12")` A possible reason is LookupCallResolver treat decimal(38, 18) as a function call. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13547) Verify and correct string function's semantic for Blink planner
Zhenghua Gao created FLINK-13547: Summary: Verify and correct string function's semantic for Blink planner Key: FLINK-13547 URL: https://issues.apache.org/jira/browse/FLINK-13547 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13523) Verify and correct arithmetic function's semantic for Blink planner
Zhenghua Gao created FLINK-13523: Summary: Verify and correct arithmetic function's semantic for Blink planner Key: FLINK-13523 URL: https://issues.apache.org/jira/browse/FLINK-13523 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13522) Verify and correct builtin function's semantic for Blink planner
Zhenghua Gao created FLINK-13522: Summary: Verify and correct builtin function's semantic for Blink planner Key: FLINK-13522 URL: https://issues.apache.org/jira/browse/FLINK-13522 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13494) Blink planner changes source parallelism which causes stream SQL e2e test fails
Zhenghua Gao created FLINK-13494: Summary: Blink planner changes source parallelism which causes stream SQL e2e test fails Key: FLINK-13494 URL: https://issues.apache.org/jira/browse/FLINK-13494 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13394) secure MapR repo URL is not work in E2E crontab builds
Zhenghua Gao created FLINK-13394: Summary: secure MapR repo URL is not work in E2E crontab builds Key: FLINK-13394 URL: https://issues.apache.org/jira/browse/FLINK-13394 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 [FLINK-12578|https://issues.apache.org/jira/browse/FLINK-12578] [FLINK-12578|http://example.com/] intros https URL for MapR, but this causes fails on Travis for some reason. travis_watchdog.sh and travis_controller.sh are fixed by unsafe-mapr-repo profile, but nightly.sh is not fixed. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13377) Streaming SQL e2e test failed on travis
Zhenghua Gao created FLINK-13377: Summary: Streaming SQL e2e test failed on travis Key: FLINK-13377 URL: https://issues.apache.org/jira/browse/FLINK-13377 Project: Flink Issue Type: Bug Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 This is an instance: [https://api.travis-ci.org/v3/job/562011491/log.txt] == Running 'Streaming SQL end-to-end test' == TEST_DATA_DIR: /home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314 Flink dist directory: /home/travis/build/apache/flink/flink-dist/target/flink-1.9-SNAPSHOT-bin/flink-1.9-SNAPSHOT Starting cluster. Starting standalonesession daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... Dispatcher REST endpoint is up. [INFO] 1 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 2 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 3 instance(s) of taskexecutor are already running on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting execution of program Program execution finished Job with JobID 7c7b66dd4e8dc17e229700b1c746aba6 has finished. Job Runtime: 77371 ms cat: '/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/.part-*': No such file or directory cat: '/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/part-*': No such file or directory FAIL StreamSQL: Output hash mismatch. Got d41d8cd98f00b204e9800998ecf8427e, expected b29f14ed221a936211202ff65b51ee26. head hexdump of actual: Stopping taskexecutor daemon (pid: 9983) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping standalonesession daemon (pid: 8088) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 21571), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 22154), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 22595), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 30622), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 3850), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 4405), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon (pid: 4839), because it is not running anymore on travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 8531) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 9077) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon (pid: 9518) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [FAIL] Test script contains errors. Checking of logs skipped. [FAIL] 'Streaming SQL end-to-end test' failed after 1 minutes and 51 seconds! Test exited with exit code 1 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13374) flink-table-planner-blink compile failed for scala-2.12 on travis
Zhenghua Gao created FLINK-13374: Summary: flink-table-planner-blink compile failed for scala-2.12 on travis Key: FLINK-13374 URL: https://issues.apache.org/jira/browse/FLINK-13374 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: Zhenghua Gao Fix For: 1.10.0 Here is a instance: [https://api.travis-ci.org/v3/job/562043336/log.txt] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13302) DateTimeUtils.unixDateCeil returns the same value as unixDateFloor does
Zhenghua Gao created FLINK-13302: Summary: DateTimeUtils.unixDateCeil returns the same value as unixDateFloor does Key: FLINK-13302 URL: https://issues.apache.org/jira/browse/FLINK-13302 Project: Flink Issue Type: Bug Components: Table SQL / Legacy Planner Affects Versions: 1.9.0, 1.10.0 Reporter: Zhenghua Gao Assignee: Zhenghua Gao Fix For: 1.9.0, 1.10.0 Internally, unixDateCeil & unixDateFloor call julianDateFloor and pass a boolean parameter to differentiate them. But unixDateCeil passes wrong parameter value and returns the same value as unixDateFloor does. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13280) Revert blink changes in DateTimeUtils, and keep it same as flink version.
Zhenghua Gao created FLINK-13280: Summary: Revert blink changes in DateTimeUtils, and keep it same as flink version. Key: FLINK-13280 URL: https://issues.apache.org/jira/browse/FLINK-13280 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao Fix For: 1.9.0, 1.10.0 This class have some diff between flink/blink planner: * Blink intros some constants (e.g., MICROS_PER_DAY, SECONDS_PER_DAY), inner use, it does not matter. * Blink intros a function unixDateTimeToString (new) * Blink changes the behavior of some function * dateStringToUnixDate: only used in test & codegen now, can be moved into another util class * timeStringToUnixDate: only used in test & codegen now, can be moved into another util class * Blink intros USER TimeZone, but now it’s always UTC TimeZone, so it does not matter -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13105) Add documentation for blink planner's built-in functions
Zhenghua Gao created FLINK-13105: Summary: Add documentation for blink planner's built-in functions Key: FLINK-13105 URL: https://issues.apache.org/jira/browse/FLINK-13105 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Zhenghua Gao Assignee: Zhenghua Gao Fix For: 1.9.0 Blink planner intros some built-in functions which need to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12999) Can't generate valid execution plan for "SELECT uuid() FROM VALUES(1) T(a)"
Zhenghua Gao created FLINK-12999:
Summary: Can't generate valid execution plan for "SELECT uuid()
FROM VALUES(1) T(a)"
Key: FLINK-12999
URL: https://issues.apache.org/jira/browse/FLINK-12999
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Zhenghua Gao
The ERROR message is:
org.apache.flink.table.api.TableException: Cannot generate a valid execution
plan for the given query:
LogicalSink(fields=[EXPR$0])
+- LogicalProject(EXPR$0=[UUID()])
+- LogicalValues(tuples=[[\{ 1, 2, 3 }]])
This exception indicates that the query uses an unsupported SQL feature.
Please check the documentation for the set of currently supported SQL features.
at
org.apache.flink.table.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:72)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
at
org.apache.flink.table.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.optimizeTree(BatchCommonSubGraphBasedOptimizer.scala:82)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.org$apache$flink$table$plan$optimize$BatchCommonSubGraphBasedOptimizer$$optimizeBlock(BatchCommonSubGraphBasedOptimizer.scala:51)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:39)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:39)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.flink.table.plan.optimize.BatchCommonSubGraphBasedOptimizer.doOptimize(BatchCommonSubGraphBasedOptimizer.scala:39)
at
org.apache.flink.table.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:65)
at
org.apache.flink.table.api.TableEnvironment.optimize(TableEnvironment.scala:251)
at
org.apache.flink.table.api.TableEnvironment.compileToExecNodePlan(TableEnvironment.scala:200)
at
org.apache.flink.table.api.TableEnvironment.compile(TableEnvironment.scala:184)
at
org.apache.flink.table.api.TableEnvironment.generateStreamGraph(TableEnvironment.scala:155)
at
org.apache.flink.table.api.BatchTableEnvironment.execute(BatchTableEnvironment.scala:93)
at
org.apache.flink.table.api.TableEnvironment.execute(TableEnvironment.scala:136)
at
org.apache.flink.table.runtime.utils.BatchTableEnvUtil$.collect(BatchTableEnvUtil.scala:55)
at
org.apache.flink.table.runtime.utils.TableUtil$.collectSink(TableUtil.scala:60)
at org.apache.flink.table.runtime.utils.TableUtil$.collect(TableUtil.scala:41)
at
org.apache.flink.table.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:308)
at
org.apache.flink.table.runtime.utils.BatchTestBase.check(BatchTestBase.scala:164)
at
org.apache.flink.table.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:103)
at
org.apache.flink.table.runtime.batch.sql.ValuesITCase.test(ValuesITCase.scala:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
[jira] [Created] (FLINK-12845) Execute multiple statements in command line or sql script file
Zhenghua Gao created FLINK-12845: Summary: Execute multiple statements in command line or sql script file Key: FLINK-12845 URL: https://issues.apache.org/jira/browse/FLINK-12845 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao User may copy multiple statements and paste them on command line GUI of SQL Client, or User may pass a script file(using SOURCE command or -f option), we should parse and execute them one by one(like other sql cli applications) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12828) Support -f option with a sql script file as input
Zhenghua Gao created FLINK-12828: Summary: Support -f option with a sql script file as input Key: FLINK-12828 URL: https://issues.apache.org/jira/browse/FLINK-12828 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao We expect user to run a script file directly on the command line. Something like: sql-client embedded -f myscript.sql, which will execute the given file without entering interactive mode -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12819) Reuse TableEnvironment between different SQL statements
Zhenghua Gao created FLINK-12819: Summary: Reuse TableEnvironment between different SQL statements Key: FLINK-12819 URL: https://issues.apache.org/jira/browse/FLINK-12819 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Zhenghua Gao We have introduced catalogs to store catalog object(tables, views etc). And the catalogs are tied to TableEnvironment, So we need to reuse TableEnvironment so the previously registered tables and views are available(suppose we use an InMemory catalog). BTW, reuse TableEnvironment is more resource and time saving. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12814) Support a traditional and scrolling view of result (non-interactive)
Zhenghua Gao created FLINK-12814: Summary: Support a traditional and scrolling view of result (non-interactive) Key: FLINK-12814 URL: https://issues.apache.org/jira/browse/FLINK-12814 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Affects Versions: 1.8.0 Reporter: Zhenghua Gao Assignee: Zhenghua Gao Attachments: image-2019-06-12-16-11-06-070.png In table mode, we want to introduce a non-interactive view (so-called FinalizedResult), which submit SQL statements(DQLs) in attach mode with a user defined timeout, fetch results until the job finished/failed/timeout or interrupted by user(Ctrl+C), and output them in a non-interactive way (the behavior in change-log mode is under discussion) !image-2019-06-12-16-11-06-070.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8045) Add Internal DATE/TIME/TIMESTAMP as internal representation of DATE/TIME/TIMESTAMP
Zhenghua Gao created FLINK-8045: --- Summary: Add Internal DATE/TIME/TIMESTAMP as internal representation of DATE/TIME/TIMESTAMP Key: FLINK-8045 URL: https://issues.apache.org/jira/browse/FLINK-8045 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Zhenghua Gao Assignee: Zhenghua Gao Currently DATE/TIME/TIMESTAMP have internal representation. Such as Date is represented as Int internal. This feature may improve performance processing DATE/TIME/TIMESTAMP data. But I found there is a LIMITATION: internal representation exists only within one operator. We transfer DATE/TIME/TIMESTAMP objects between operators. I think we could treat DATE/TIME/TIMESTAMP as internal representation in the whole job, and cast them to java.sql.* as needed(UDF/UDTF/OUTPUT) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6173) flink-table not pack-in com.fasterxml.jackson.* in after #FLINK-5414
Zhenghua Gao created FLINK-6173:
---
Summary: flink-table not pack-in com.fasterxml.jackson.* in after
#FLINK-5414
Key: FLINK-6173
URL: https://issues.apache.org/jira/browse/FLINK-6173
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Reporter: Zhenghua Gao
Currently, flink-table will pack-in com.fasterxml.jackson.* and rename them to
org.apache.flink.shaded.calcite.com.fasterxml.jackson.*
If a project depends on flink-table, and uses fasterxml as follows(function
explain uses fasterxml indirectly):
```
object WordCountWithTable {
def main(args: Array[String]): Unit = {
// set up execution environment
val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = TableEnvironment.getTableEnvironment(env)
val input = env.fromElements(WC("hello", 1), WC("hello", 1), WC("ciao", 1))
val expr = input.toTable(tEnv)
val result = expr
.groupBy('word)
.select('word, 'frequency.sum as 'frequency)
.filter('frequency === 2)
println(tEnv.explain(result))
result.toDataSet[WC].print()
}
case class WC(word: String, frequency: Long)
}
```
It actually uses org.apache.flink.shaded.calcite.com.fasterxml.jackson.*
I found after FLINK-5414, flink-table didn't pack-in com.fasterxml.jackson.*
and the project would throw class not found exception.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/flink/shaded/calcite/com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.flink.table.explain.PlanJsonParser.getSqlExecutionPlan(PlanJsonParser.java:32)
at
org.apache.flink.table.api.BatchTableEnvironment.explain(BatchTableEnvironment.scala:143)
at
org.apache.flink.table.api.BatchTableEnvironment.explain(BatchTableEnvironment.scala:164)
at
org.apache.flink.quickstart.WordCountWithTable$.main(WordCountWithTable.scala:34)
at
org.apache.flink.quickstart.WordCountWithTable.main(WordCountWithTable.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.shaded.calcite.com.fasterxml.jackson.databind.ObjectMapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6124) support max/min aggregations for string type
Zhenghua Gao created FLINK-6124: --- Summary: support max/min aggregations for string type Key: FLINK-6124 URL: https://issues.apache.org/jira/browse/FLINK-6124 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Zhenghua Gao Assignee: Zhenghua Gao Recently when I port some query to Flink SQL, I found currently min/max aggregations on string type is not supported and should be added. When min/max aggregations are used on string column, return min/max value by lexicographically order. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5545) remove FlinkAggregateExpandDistinctAggregatesRule when we upgrade calcite
Zhenghua Gao created FLINK-5545: --- Summary: remove FlinkAggregateExpandDistinctAggregatesRule when we upgrade calcite Key: FLINK-5545 URL: https://issues.apache.org/jira/browse/FLINK-5545 Project: Flink Issue Type: Bug Reporter: Zhenghua Gao Assignee: Zhenghua Gao Priority: Minor We copy calcite's AggregateExpandDistinctAggregatesRule to Flink project, and do a quick fix to avoid some bad case mentioned in CALCITE-1558. Should drop it and use calcite's AggregateExpandDistinctAggregatesRule when we upgrade to calcite 1.12(above) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
