[jira] [Closed] (FLINK-15231) Wrong HeapVector in AbstractHeapVector.createHeapColumn
[ https://issues.apache.org/jira/browse/FLINK-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15231. -- Fix Version/s: (was: 1.11.0) 1.10.0 Resolution: Fixed master: 9dc252849966dd21279572afff55dcbdd3f77f35 1.10.0: 62f0303f07120c51317aff171f105ecb0c65a2be > Wrong HeapVector in AbstractHeapVector.createHeapColumn > --- > > Key: FLINK-15231 > URL: https://issues.apache.org/jira/browse/FLINK-15231 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > For TIMESTAMP WITHOUT TIME ZONE/TIMESTAMP WITH LOCAL TIME ZONE/DECIMAL types, > AbstractHeapVector.createHeapColumn generates wrong HeapVectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15440) Enable savepoint support for Table & SQL program
Kurt Young created FLINK-15440: -- Summary: Enable savepoint support for Table & SQL program Key: FLINK-15440 URL: https://issues.apache.org/jira/browse/FLINK-15440 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Kurt Young -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15397) Streaming and batch has different value in the case of count function
[
https://issues.apache.org/jira/browse/FLINK-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15397:
--
Assignee: Jingsong Lee
> Streaming and batch has different value in the case of count function
> -
>
> Key: FLINK-15397
> URL: https://issues.apache.org/jira/browse/FLINK-15397
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> *The sql is:*
> CREATE TABLE `testdata` (
> a INT,
> b INT
> ) WITH (
>
> 'connector.path'='/defender_test_data/daily_regression_batch_spark_1.10/test_group_agg/sources/testdata.csv',
> 'format.empty-column-as-null'='true',
> 'format.field-delimiter'='|',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
> 'format.type'='csv'
> );
> SELECT COUNT(1) FROM testdata WHERE false;
> If the configuration's type is batch ,the result will be 0, but if the
> configuration is streaming, there will be no value;
> *The configuration is:*
> execution:
> planner: blink
> type: streaming
> *The input data is:*
> {code:java}
> 1|1
> 1|2
> 2|1
> 2|2
> 3|1
> 3|2
> |1
> 3|
> |
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15290) Need a way to turn off vectorized orc reader for SQL CLI
[ https://issues.apache.org/jira/browse/FLINK-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-15290: -- Assignee: Rui Li > Need a way to turn off vectorized orc reader for SQL CLI > > > Key: FLINK-15290 > URL: https://issues.apache.org/jira/browse/FLINK-15290 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-15411: -- Assignee: Jingsong Lee (was: Rui Li) > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > Hive should work after planner fixed due to: > [https://github.com/apache/flink/pull/10690#issuecomment-569021089] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003924#comment-17003924 ] Kurt Young edited comment on FLINK-15411 at 12/27/19 6:32 AM: -- Yeah, that might also be true. Even if users want to partition the data by date, they could still choose STRING as field type. was (Author: ykt836): Yeah, that might also be true. Even if the want to partition the data by date, they could still choose STRING as field type. > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > > Hive should work after planner fixed due to: > [https://github.com/apache/flink/pull/10690#issuecomment-569021089] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003924#comment-17003924 ] Kurt Young commented on FLINK-15411: Yeah, that might also be true. Even if the want to partition the data by date, they could still choose STRING as field type. > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > > Hive should work after planner fixed due to: > [https://github.com/apache/flink/pull/10690#issuecomment-569021089] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15412) LocalExecutorITCase#testParameterizedTypes failed in travis
[
https://issues.apache.org/jira/browse/FLINK-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003909#comment-17003909
]
Kurt Young commented on FLINK-15412:
cc [~lirui]
> LocalExecutorITCase#testParameterizedTypes failed in travis
> ---
>
> Key: FLINK-15412
> URL: https://issues.apache.org/jira/browse/FLINK-15412
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Dian Fu
>Priority: Major
> Fix For: 1.10.0
>
>
> The travis of release-1.9 failed with the following error:
> {code:java}
> 14:43:17.916 [INFO] Running
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase
> 14:44:47.388 [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 1, Time
> elapsed: 89.468 s <<< FAILURE! - in
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase
> 14:44:47.388 [ERROR] testParameterizedTypes[Planner:
> blink](org.apache.flink.table.client.gateway.local.LocalExecutorITCase) Time
> elapsed: 7.88 s <<< ERROR!
> org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL
> statement at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.ValidationException: SQL validation
> failed. findAndCreateTableSource failed
> at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.TableException:
> findAndCreateTableSource failed
> at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException:
> Could not find a suitable table factory for
> 'org.apache.flink.table.factories.TableSourceFactory' in
> the classpath.
> Reason: No context matches.
> {code}
> instance: [https://api.travis-ci.org/v3/job/629636106/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15412) LocalExecutorITCase#testParameterizedTypes failed in travis
[
https://issues.apache.org/jira/browse/FLINK-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15412:
---
Fix Version/s: 1.10.0
> LocalExecutorITCase#testParameterizedTypes failed in travis
> ---
>
> Key: FLINK-15412
> URL: https://issues.apache.org/jira/browse/FLINK-15412
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Dian Fu
>Priority: Major
> Fix For: 1.10.0
>
>
> The travis of release-1.9 failed with the following error:
> {code:java}
> 14:43:17.916 [INFO] Running
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase
> 14:44:47.388 [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 1, Time
> elapsed: 89.468 s <<< FAILURE! - in
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase
> 14:44:47.388 [ERROR] testParameterizedTypes[Planner:
> blink](org.apache.flink.table.client.gateway.local.LocalExecutorITCase) Time
> elapsed: 7.88 s <<< ERROR!
> org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL
> statement at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.ValidationException: SQL validation
> failed. findAndCreateTableSource failed
> at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.TableException:
> findAndCreateTableSource failed
> at
> org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testParameterizedTypes(LocalExecutorITCase.java:557)
> Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException:
> Could not find a suitable table factory for
> 'org.apache.flink.table.factories.TableSourceFactory' in
> the classpath.
> Reason: No context matches.
> {code}
> instance: [https://api.travis-ci.org/v3/job/629636106/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15409) Add semicolon to WindowJoinUtil#generateJoinFunction '$collectorTerm.collect($joinedRow)' statement
[
https://issues.apache.org/jira/browse/FLINK-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003907#comment-17003907
]
Kurt Young commented on FLINK-15409:
cc [~jark] [~jinyu.zj]
> Add semicolon to WindowJoinUtil#generateJoinFunction
> '$collectorTerm.collect($joinedRow)' statement
> ---
>
> Key: FLINK-15409
> URL: https://issues.apache.org/jira/browse/FLINK-15409
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.1
>Reporter: hailong wang
>Priority: Major
> Fix For: 1.9.2, 1.10.0
>
>
> In WindowJoinUtil#generateJoinFunction, When otherCondition is none, it will
> go into statement:
> {code:java}
> case None =>
> s"""
> |$buildJoinedRow
> |$collectorTerm.collect($joinedRow)
> |""".stripMargin
> {code}
> And it miss a semicolon after collet($joinedRow). This will cause compile
> fail:
> {code:java}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.Caused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue. at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:65)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 26 moreCaused by: org.codehaus.commons.compiler.CompileException: Line
> 28, Column 21: Expression "c.collect(joinedRow)" is not a type
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15409) Add semicolon to WindowJoinUtil#generateJoinFunction '$collectorTerm.collect($joinedRow)' statement
[
https://issues.apache.org/jira/browse/FLINK-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15409:
---
Affects Version/s: (was: 1.10.0)
1.9.1
> Add semicolon to WindowJoinUtil#generateJoinFunction
> '$collectorTerm.collect($joinedRow)' statement
> ---
>
> Key: FLINK-15409
> URL: https://issues.apache.org/jira/browse/FLINK-15409
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.1
>Reporter: hailong wang
>Priority: Major
> Fix For: 1.9.2, 1.10.0
>
>
> In WindowJoinUtil#generateJoinFunction, When otherCondition is none, it will
> go into statement:
> {code:java}
> case None =>
> s"""
> |$buildJoinedRow
> |$collectorTerm.collect($joinedRow)
> |""".stripMargin
> {code}
> And it miss a semicolon after collet($joinedRow). This will cause compile
> fail:
> {code:java}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.Caused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue. at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:65)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 26 moreCaused by: org.codehaus.commons.compiler.CompileException: Line
> 28, Column 21: Expression "c.collect(joinedRow)" is not a type
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15409) Add semicolon to WindowJoinUtil#generateJoinFunction '$collectorTerm.collect($joinedRow)' statement
[
https://issues.apache.org/jira/browse/FLINK-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15409:
---
Fix Version/s: (was: 1.11.0)
1.10.0
1.9.2
> Add semicolon to WindowJoinUtil#generateJoinFunction
> '$collectorTerm.collect($joinedRow)' statement
> ---
>
> Key: FLINK-15409
> URL: https://issues.apache.org/jira/browse/FLINK-15409
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0
>Reporter: hailong wang
>Priority: Major
> Fix For: 1.9.2, 1.10.0
>
>
> In WindowJoinUtil#generateJoinFunction, When otherCondition is none, it will
> go into statement:
> {code:java}
> case None =>
> s"""
> |$buildJoinedRow
> |$collectorTerm.collect($joinedRow)
> |""".stripMargin
> {code}
> And it miss a semicolon after collet($joinedRow). This will cause compile
> fail:
> {code:java}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.Caused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue. at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:65)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 26 moreCaused by: org.codehaus.commons.compiler.CompileException: Line
> 28, Column 21: Expression "c.collect(joinedRow)" is not a type
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15411: --- Affects Version/s: (was: 1.11.0) > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15411: --- Fix Version/s: (was: 1.11.0) > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0, 1.11.0 >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15411) Planner can't prune partition on DATE/TIMESTAMP columns
[ https://issues.apache.org/jira/browse/FLINK-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003905#comment-17003905 ] Kurt Young commented on FLINK-15411: IMO partition with DATE field maybe the most widely used in hive data warehouse. > Planner can't prune partition on DATE/TIMESTAMP columns > --- > > Key: FLINK-15411 > URL: https://issues.apache.org/jira/browse/FLINK-15411 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Affects Versions: 1.10.0, 1.11.0 >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0, 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15413) ScalarOperatorsTest failed in travis
[
https://issues.apache.org/jira/browse/FLINK-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15413:
---
Fix Version/s: 1.9.2
> ScalarOperatorsTest failed in travis
>
>
> Key: FLINK-15413
> URL: https://issues.apache.org/jira/browse/FLINK-15413
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Dian Fu
>Priority: Major
> Fix For: 1.9.2
>
>
> The travis of release-1.9 failed with the following error:
> {code:java}
> 14:50:19.796 [ERROR] ScalarOperatorsTest>ExpressionTestBase.evaluateExprs:161
> Wrong result for: [CASE WHEN (CASE WHEN f2 = 1 THEN CAST('' as INT) ELSE 0
> END) is null THEN 'null' ELSE 'not null' END] optimized to: [_UTF-16LE'not
> null':VARCHAR(8) CHARACTER SET "UTF-16LE"] expected: but was: n]ull>
> {code}
> instance: [https://api.travis-ci.org/v3/job/629636107/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15391) DATE and TIMESTAMP partition columns don't work
[ https://issues.apache.org/jira/browse/FLINK-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15391: --- Fix Version/s: (was: 1.11.0) > DATE and TIMESTAMP partition columns don't work > --- > > Key: FLINK-15391 > URL: https://issues.apache.org/jira/browse/FLINK-15391 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15404) How to insert hive table for different catalog
[
https://issues.apache.org/jira/browse/FLINK-15404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003597#comment-17003597
]
Kurt Young commented on FLINK-15404:
try
{noformat}
insert into myhive.default.hive_table select * from
default_catalog.default_database.source_table{noformat}
> How to insert hive table for different catalog
> ---
>
> Key: FLINK-15404
> URL: https://issues.apache.org/jira/browse/FLINK-15404
> Project: Flink
> Issue Type: Wish
> Components: Table SQL / Planner
>Reporter: hehuiyuan
>Priority: Major
>
> I have a hive catalog :
>
> {code:java}
> catalog name : myhive
> database : default
> {code}
>
> and the flink has a default catalog :
>
> {code:java}
> catalog name : default_catalog
> database : default_database
> {code}
>
> For example :
> I have a source table 'source_table' that's from kafka which is register to
> default_catalog,
> I want to insert hive table 'hive_table' that is from myhive catalog.
> SQL:
> insert into hive_table select * from source_table;
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15395) No API for execute insert into statement
[ https://issues.apache.org/jira/browse/FLINK-15395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003502#comment-17003502 ] Kurt Young commented on FLINK-15395: [~zjffdu] There is no need to follow sql client, normally, tableEnv.sqlUpdate + tableEnv.execute should be enough. > No API for execute insert into statement > - > > Key: FLINK-15395 > URL: https://issues.apache.org/jira/browse/FLINK-15395 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Jeff Zhang >Priority: Major > > IIUC, TableEnv#sqlUpdate is used for DDL & DML while TableEnv#sqlQuery is > used for select statement. Unfortunately, it seems `insert into` is a special > case that no simple api can be used for it. > The code of implementing `insert into` in sql-client is pretty complex, it > would be nice to have one simple api for executing `insert into` > https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java#L572 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15395) No API for execute insert into statement
[ https://issues.apache.org/jira/browse/FLINK-15395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003499#comment-17003499 ] Kurt Young commented on FLINK-15395: Why? Can you provide more details? > No API for execute insert into statement > - > > Key: FLINK-15395 > URL: https://issues.apache.org/jira/browse/FLINK-15395 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Jeff Zhang >Priority: Major > > IIUC, TableEnv#sqlUpdate is used for DDL & DML while TableEnv#sqlQuery is > used for select statement. Unfortunately, it seems `insert into` is a special > case that no simple api can be used for it. > The code of implementing `insert into` in sql-client is pretty complex, it > would be nice to have one simple api for executing `insert into` > https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java#L572 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15381) INSERT INTO VALUES statement fails if a cast project is applied
[
https://issues.apache.org/jira/browse/FLINK-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003496#comment-17003496
]
Kurt Young commented on FLINK-15381:
Just curious, who generated collation trait in the plan? It seems no operator
involved has such information.
> INSERT INTO VALUES statement fails if a cast project is applied
> ---
>
> Key: FLINK-15381
> URL: https://issues.apache.org/jira/browse/FLINK-15381
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: godfrey he
>Priority: Blocker
> Fix For: 1.10.0
>
> Attachments: image-2019-12-26-14-56-00-634.png
>
>
> The following query will fail:
> {code:scala}
> @Test
> def test(): Unit = {
> val sinkDDL =
> """
> |create table t2(
> | a int,
> | b string
> |) with (
> | 'connector' = 'COLLECTION'
> |)
> """.stripMargin
> val query =
> """
> |insert into t2 select cast(a as int), cast(b as varchar) from
> (values (3, 'c')) T(a,b)
> """.stripMargin
> tableEnv.sqlUpdate(sinkDDL)
> tableEnv.sqlUpdate(query)
> execJob("testJob")
> }
> {code}
> exception:
> {code}
> org.apache.flink.table.api.TableException: Cannot generate a valid execution
> plan for the given query:
> LogicalSink(name=[`default_catalog`.`default_database`.`t2`], fields=[a, b])
> +- LogicalProject(EXPR$0=[$0], EXPR$1=[CAST($1):VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL])
>+- LogicalValues(type=[RecordType(INTEGER a, CHAR(1) b)], tuples=[[{ 3,
> _UTF-16LE'c' }]])
> This exception indicates that the query uses an unsupported SQL feature.
> Please check the documentation for the set of currently supported SQL
> features.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15395) No API for execute insert into statement
[ https://issues.apache.org/jira/browse/FLINK-15395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003477#comment-17003477 ] Kurt Young commented on FLINK-15395: sqlUpdate is supposed to execute `insert into`. > No API for execute insert into statement > - > > Key: FLINK-15395 > URL: https://issues.apache.org/jira/browse/FLINK-15395 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Jeff Zhang >Priority: Major > > IIUC, TableEnv#sqlUpdate is used for DDL while TableEnv#sqlQuery is used for > DML. Unfortunately, it seems `insert into` is a special case that no simple > api can be used for it. > The code of implementing `insert into` in sql-client is pretty complex, it > would be nice to have one simple api for executing `insert into` > https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java#L572 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15384) Allow to pass ExecutionEnvironment/StreamExecutionEnvironment to TableEnvironment
[
https://issues.apache.org/jira/browse/FLINK-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003045#comment-17003045
]
Kurt Young commented on FLINK-15384:
Can't you use `StreamTableEnvironment.create(StreamExecutionEnvironment)`?
> Allow to pass ExecutionEnvironment/StreamExecutionEnvironment to
> TableEnvironment
> --
>
> Key: FLINK-15384
> URL: https://issues.apache.org/jira/browse/FLINK-15384
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> In the new approach of creating TableEnvironement, it is not possible to pass
> ExecutionEnvironment/StreamExecutionEnvironment to TableEnvironement. This
> cause the all the features in ExecutionEnvironment/StreamExecutionEnvironment
> is not available in table api. Such as add JobListener, execute Job async. So
> I suggest to allow to pass ExecutionEnvironment/StreamExecutionEnvironment to
> TableEnvironment
> {code}
> EnvironmentSettings bbSettings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
> TableEnvironment bbTableEnv = TableEnvironment.create(bbSettings);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15066:
--
Assignee: Jingsong Lee (was: Danny Chen)
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15175) syntax not supported in SQLClient for TPCDS queries
[
https://issues.apache.org/jira/browse/FLINK-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-15175.
--
Fix Version/s: 1.10.0
Resolution: Fixed
master: fc4927e41989be75866218c2a60aa914e1eedcd3
1.10.0: 3af8e1ef31aa61cf08ed910df32a1a26dd26f892
> syntax not supported in SQLClient for TPCDS queries
>
>
> Key: FLINK-15175
> URL: https://issues.apache.org/jira/browse/FLINK-15175
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: liupengcheng
>Assignee: liupengcheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> Flink SQL> WITH customer_total_return AS
> > ( SELECT
> > sr_customer_sk AS ctr_customer_sk,
> > sr_store_sk AS ctr_store_sk,
> > sum(sr_return_amt) AS ctr_total_return
> > FROM store_returns, date_dim
> > WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
> > GROUP BY sr_customer_sk, sr_store_sk)
> > SELECT c_customer_id
> > FROM customer_total_return ctr1, store, customer
> > WHERE ctr1.ctr_total_return >
> > (SELECT avg(ctr_total_return) * 1.2
> > FROM customer_total_return ctr2
> > WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> > AND s_store_sk = ctr1.ctr_store_sk
> > AND s_state = 'TN'
> > AND ctr1.ctr_customer_sk = c_customer_sk
> > ORDER BY c_customer_id
> > LIMIT 100;
> [ERROR] Unknown or invalid SQL statement.
> {code}
> It seems that the newest branch already support all TPCDS queries, but
> currently the sql client parser has not supported yet.
> Anyone already working on this? If not I can try it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15331) Create a table from a changelog
[ https://issues.apache.org/jira/browse/FLINK-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000543#comment-17000543 ] Kurt Young commented on FLINK-15331: +10086 > Create a table from a changelog > --- > > Key: FLINK-15331 > URL: https://issues.apache.org/jira/browse/FLINK-15331 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > > This an umbrella issue for the feature of interpreting a changelog as a > table. It can be used to collect initial ideas and use cases until a FLIP is > proposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15290) Need a way to turn off vectorized orc reader for SQL CLI
[
https://issues.apache.org/jira/browse/FLINK-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997990#comment-16997990
]
Kurt Young commented on FLINK-15290:
We already have this in `HiveOptions`, the config name is
{noformat}
table.exec.hive.fallback-mapred-reader = true
{noformat}
Could you check whether this can work?
> Need a way to turn off vectorized orc reader for SQL CLI
>
>
> Key: FLINK-15290
> URL: https://issues.apache.org/jira/browse/FLINK-15290
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Rui Li
>Priority: Major
> Fix For: 1.10.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15266) NPE in blink planner code gen
[
https://issues.apache.org/jira/browse/FLINK-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15266:
---
Fix Version/s: 1.9.2
> NPE in blink planner code gen
> -
>
> Key: FLINK-15266
> URL: https://issues.apache.org/jira/browse/FLINK-15266
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.1
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.2, 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> `cast` function in blink planner and old planner are different:
> in legacy planner:
> cast('' as int) -> throw NumberFormatException
> cast(null as int) -> throw NullPointerException
> cast('abc' as int) -> throw NumberFormatException
> but in blink planner:
> cast('' as int) -> return null
> cast(null as int) -> return null
> cast('abc' as int) -> return null
> A step forward:
> ```
> create table source {
> age int,
> id varchar
> };
> select case when age < 20 then cast(id as bigint) else 0 end from source;
> ```
> queries like above will throw NPE because we will try assign a `null` to a
> `long` field when the input satisfy `age < 20`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15245) Flink running in one cluster cannot write data to Hive tables in another cluster
[ https://issues.apache.org/jira/browse/FLINK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15245. -- Assignee: Rui Li Resolution: Fixed master: b0783b50cb1610611f9442d456501160322b5028 1.10.0: 34093227ed681f1b0c842c50eb39bc790ee10090 > Flink running in one cluster cannot write data to Hive tables in another > cluster > > > Key: FLINK-15245 > URL: https://issues.apache.org/jira/browse/FLINK-15245 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Launch Flink cluster and write some data to a Hive table in another cluster. > The job finishes successfully but data is not really written. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15287) Vectorized orc reader fails with Hive 2.0.1
[
https://issues.apache.org/jira/browse/FLINK-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15287:
--
Assignee: Jingsong Lee
> Vectorized orc reader fails with Hive 2.0.1
> ---
>
> Key: FLINK-15287
> URL: https://issues.apache.org/jira/browse/FLINK-15287
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Rui Li
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> Reading ORC table from Hive 2.0.1 fails with:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.orc.OrcFile.createReader(Lorg/apache/hadoop/fs/Path;Lorg/apache/orc/OrcFile$ReaderOptions;)Lorg/apache/orc/Reader;
> at org.apache.flink.orc.OrcSplitReader.(OrcSplitReader.java:78)
> at
> org.apache.flink.orc.OrcColumnarRowSplitReader.(OrcColumnarRowSplitReader.java:53)
> at
> org.apache.flink.orc.OrcSplitReaderUtil.genPartColumnarRowReader(OrcSplitReaderUtil.java:93)
> at
> org.apache.flink.connectors.hive.read.HiveVectorizedOrcSplitReader.(HiveVectorizedOrcSplitReader.java:64)
> at
> org.apache.flink.connectors.hive.read.HiveTableInputFormat.open(HiveTableInputFormat.java:117)
> at
> org.apache.flink.connectors.hive.read.HiveTableInputFormat.open(HiveTableInputFormat.java:57)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:85)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:196)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15287) Vectorized orc reader fails with Hive 2.0.1
[
https://issues.apache.org/jira/browse/FLINK-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15287:
---
Affects Version/s: 1.10.0
> Vectorized orc reader fails with Hive 2.0.1
> ---
>
> Key: FLINK-15287
> URL: https://issues.apache.org/jira/browse/FLINK-15287
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> Reading ORC table from Hive 2.0.1 fails with:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.orc.OrcFile.createReader(Lorg/apache/hadoop/fs/Path;Lorg/apache/orc/OrcFile$ReaderOptions;)Lorg/apache/orc/Reader;
> at org.apache.flink.orc.OrcSplitReader.(OrcSplitReader.java:78)
> at
> org.apache.flink.orc.OrcColumnarRowSplitReader.(OrcColumnarRowSplitReader.java:53)
> at
> org.apache.flink.orc.OrcSplitReaderUtil.genPartColumnarRowReader(OrcSplitReaderUtil.java:93)
> at
> org.apache.flink.connectors.hive.read.HiveVectorizedOrcSplitReader.(HiveVectorizedOrcSplitReader.java:64)
> at
> org.apache.flink.connectors.hive.read.HiveTableInputFormat.open(HiveTableInputFormat.java:117)
> at
> org.apache.flink.connectors.hive.read.HiveTableInputFormat.open(HiveTableInputFormat.java:57)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:85)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:196)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15284) Sql error (Failed to push project into table source!)
[
https://issues.apache.org/jira/browse/FLINK-15284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-15284.
--
Resolution: Duplicate
> Sql error (Failed to push project into table source!)
> -
>
> Key: FLINK-15284
> URL: https://issues.apache.org/jira/browse/FLINK-15284
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> *The sql is:*
> CREATE TABLE `t` (
> x INT
> ) WITH (
> 'format.field-delimiter'=',',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
>
> 'connector.path'='/defender_test_data/daily_regression_batch_spark_1.10/test_binary_comparison_coercion/sources/t.csv',
> 'format.type'='csv'
> );
> SELECT cast(' ' as BINARY(2)) = X'0020' FROM t;
> *The exception is:*
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.TableException: Failed to push project into table
> source! table source with pushdown capability must override and change
> explainSource() API to explain the pushdown applied!
>
>
> *The whole exception is:*
> Caused by: org.apache.flink.table.api.TableException: Sql optimization:
> Cannot generate a valid execution plan for the given query:Caused by:
> org.apache.flink.table.api.TableException: Sql optimization: Cannot generate
> a valid execution plan for the given query:
>
> LogicalSink(name=[`default_catalog`.`default_database`.`_tmp_table_2136189659`],
> fields=[EXPR$0])+- LogicalProject(EXPR$0=[false])+ -
> LogicalTableScan(table=[[default_catalog, default_database, t, source:
> [CsvTableSource(read fields: x)]]])
> Failed to push project into table source! table source with pushdown
> capability must override and change explainSource() API to explain the
> pushdown applied!Please check the documentation for the set of currently
> supported SQL features. at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:86)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157) at
> scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104) at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
> at
> org.apache.flink.table.planner.plan.optimize.BatchCommonSubGraphBasedOptimizer.optimizeTree(BatchCommonSubGraphBasedOptimizer.scala:83)
> at
> org.apache.flink.table.planner.plan.optimize.BatchCommonSubGraphBasedOptimizer.org$apache$flink$table$planner$plan$optimize$BatchCommonSubGraphBasedOptimizer$$optimizeBlock(BatchCommonSubGraphBasedOptimizer.scala:56)
> at
> org.apache.flink.table.planner.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:44)
> at
> org.apache.flink.table.planner.plan.optimize.BatchCommonSubGraphBasedOptimizer$$anonfun$doOptimize$1.apply(BatchCommonSubGraphBasedOptimizer.scala:44)
> at scala.collection.immutable.List.foreach(List.scala:392) at
> org.apache.flink.table.planner.plan.optimize.BatchCommonSubGraphBasedOptimizer.doOptimize(BatchCommonSubGraphBasedOptimizer.scala:44)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:77)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:223)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:150)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:680)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.insertIntoInternal(TableEnvironmentImpl.java:353)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.insertInto(TableEnvironmentImpl.java:341)
> at
> org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:428)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeQueryAndPersistInternal$14(LocalExecutor.java:701)
>
[jira] [Updated] (FLINK-15246) Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] not equal to TableSink schema: [EXPR$0: TIMESTAMP(3)]
[
https://issues.apache.org/jira/browse/FLINK-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15246:
---
Component/s: (was: Table SQL / Client)
Table SQL / Planner
> Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] not equal to TableSink
> schema:[EXPR$0: TIMESTAMP(3)]
> ---
>
> Key: FLINK-15246
> URL: https://issues.apache.org/jira/browse/FLINK-15246
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
> Fix For: 1.10.0
>
>
> When I excute the sql below and check the result of it, the "Query result
> schema" is not equal to the "TableSink schema";
>
>
> The sql is:
> CREATE TABLE `t` (
> x INT
> ) WITH (
> 'format.field-delimiter'=',',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
>
> 'connector.path'='/defender_test_data/daily_regression_batch_spark_1.10/test_case_when_coercion/sources/t.csv',
> 'format.type'='csv'
> );
> SELECT CASE WHEN true THEN cast('2017-12-12 09:30:00.0' as timestamp) ELSE
> cast(2 as tinyint) END FROM t;
>
> The exception is:
> org.apache.flink.table.api.ValidationException: Field types of query result
> and registered TableSink
> `default_catalog`.`default_database`.`_tmp_table_443938765` do not match.
> Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] TableSink schema:
> [EXPR$0: TIMESTAMP(3)]
>
> The input data is:
> 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15246) Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] not equal to TableSink schema: [EXPR$0: TIMESTAMP(3)]
[
https://issues.apache.org/jira/browse/FLINK-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15246:
---
Fix Version/s: 1.10.0
> Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] not equal to TableSink
> schema:[EXPR$0: TIMESTAMP(3)]
> ---
>
> Key: FLINK-15246
> URL: https://issues.apache.org/jira/browse/FLINK-15246
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
> Fix For: 1.10.0
>
>
> When I excute the sql below and check the result of it, the "Query result
> schema" is not equal to the "TableSink schema";
>
>
> The sql is:
> CREATE TABLE `t` (
> x INT
> ) WITH (
> 'format.field-delimiter'=',',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
>
> 'connector.path'='/defender_test_data/daily_regression_batch_spark_1.10/test_case_when_coercion/sources/t.csv',
> 'format.type'='csv'
> );
> SELECT CASE WHEN true THEN cast('2017-12-12 09:30:00.0' as timestamp) ELSE
> cast(2 as tinyint) END FROM t;
>
> The exception is:
> org.apache.flink.table.api.ValidationException: Field types of query result
> and registered TableSink
> `default_catalog`.`default_database`.`_tmp_table_443938765` do not match.
> Query result schema: [EXPR$0: TIMESTAMP(6) NOT NULL] TableSink schema:
> [EXPR$0: TIMESTAMP(3)]
>
> The input data is:
> 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15167) SQL CLI library option doesn't work for Hive connector
[
https://issues.apache.org/jira/browse/FLINK-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-15167.
--
Resolution: Fixed
master: cd4c3ddf
1.10.0: 5a43907e
> SQL CLI library option doesn't work for Hive connector
> --
>
> Key: FLINK-15167
> URL: https://issues.apache.org/jira/browse/FLINK-15167
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Launch a standalone cluster. Put all Hive connector dependency jars in a
> folder and start sql cli like: {{sql-client.sh embedded -l }}. Hit
> the following exception:
> {noformat}
> at
> org.apache.flink.table.catalog.hive.HiveCatalog.open(HiveCatalog.java:186)
> at
> org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:102)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.registerCatalog(TableEnvironmentImpl.java:233)
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:519)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:463)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:156)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:115)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:724)
> ... 3 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.table.catalog.hive.client.HiveShimV230.getHiveMetastoreClient(HiveShimV230.java:55)
> ... 15 more
> Caused by:
> MetaException(message:org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> class not found)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1676)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89)
> ... 20 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15238) A sql can't generate a valid execution plan
[ https://issues.apache.org/jira/browse/FLINK-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15238: --- Component/s: (was: Table SQL / Client) Table SQL / Planner > A sql can't generate a valid execution plan > --- > > Key: FLINK-15238 > URL: https://issues.apache.org/jira/browse/FLINK-15238 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > *The table and the query is like this:* > CREATE TABLE `scott_emp` ( > > empno INT, > > ename VARCHAR, > > job VARCHAR, > > mgr INT, > > hiredate DATE, > > sal DOUBLE, > > comm DOUBLE, > > deptno INT > > ) WITH ( > > 'format.field-delimiter'='|', > > 'connector.type'='filesystem', > > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_blink_sql_1.10/test_scalar/sources/scott_emp.csv', > > 'format.type'='csv' > > ); > CREATE TABLE `scott_dept` ( > > deptno INT, > > dname VARCHAR, > > loc VARCHAR > > ) WITH ( > > 'format.field-delimiter'='|', > > 'connector.type'='filesystem', > > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_blink_sql_1.10/test_scalar/sources/scott_dept.csv', > > 'format.type'='csv' > > ); > select deptno, (select empno from scott_emp order by empno limit 1) as x from > scott_dept; > > > *After execution the sql, the exception will appear:* > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.TableException: Cannot generate a valid execution > plan for the given query: > LogicalProject(deptno=[$0], x=[$3]) > LogicalJoin(condition=[true], joinType=[left]) > LogicalTableScan(table=[[default_catalog, default_database, scott_dept]]) > LogicalSort(sort0=[$0], dir0=[ASC], fetch=[1]) > LogicalProject(empno=[$0]) > LogicalTableScan(table=[[default_catalog, default_database, scott_emp]]) > This exception indicates that the query uses an unsupported SQL feature. > Please check the documentation for the set of currently supported SQL > features. > > > *The whole exception is:* > Caused by: org.apache.flink.table.api.TableException: Cannot generate a valid > execution plan for the given query:Caused by: > org.apache.flink.table.api.TableException: Cannot generate a valid execution > plan for the given query: > LogicalProject(deptno=[$0], x=[$3]) LogicalJoin(condition=[true], > joinType=[left]) LogicalTableScan(table=[[default_catalog, > default_database, scott_dept]]) LogicalSort(sort0=[$0], dir0=[ASC], > fetch=[1]) LogicalProject(empno=[$0]) > LogicalTableScan(table=[[default_catalog, default_database, scott_emp]]) > This exception indicates that the query uses an unsupported SQL > feature.Please check the documentation for the set of currently supported SQL > features. at > org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:284) > at > org.apache.flink.table.plan.Optimizer.optimizeLogicalPlan(Optimizer.scala:199) > at > org.apache.flink.table.plan.StreamOptimizer.optimize(StreamOptimizer.scala:66) > at > org.apache.flink.table.planner.StreamPlanner.translateToType(StreamPlanner.scala:389) > at > org.apache.flink.table.planner.StreamPlanner.writeToRetractSink(StreamPlanner.scala:308) > at > org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$writeToSink(StreamPlanner.scala:272) > at > org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:166) > at > org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:145) > at scala.Option.map(Option.scala:146) at > org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$translate(StreamPlanner.scala:145) > at > org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117) > at > org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) at > scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.AbstractTraversable.map(Traversable.scala:104) at > org.apache.flink.table.planner.StreamPlanner.translate(StreamPlanner.scala:117) > at >
[jira] [Commented] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995376#comment-16995376
]
Kurt Young commented on FLINK-15066:
[~danny0405] I'm not sure if this issue relates to INSERT OVERWRITE, quite the
opposite, I didn't use INSERT OVERWRITE at all. What I did is type
`INSERT INTO csvTable VALUES(...)` twice.
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Kurt Young
>Assignee: Danny Chen
>Priority: Major
> Fix For: 1.10.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15190) add documentation for DDL in FLIP-69
[ https://issues.apache.org/jira/browse/FLINK-15190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15190: --- Priority: Blocker (was: Major) > add documentation for DDL in FLIP-69 > > > Key: FLINK-15190 > URL: https://issues.apache.org/jira/browse/FLINK-15190 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Bowen Li >Assignee: Terry Wang >Priority: Blocker > Fix For: 1.10.0 > > > in > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sql.html#ddl -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15189) add documentation for catalog view and hive view
[ https://issues.apache.org/jira/browse/FLINK-15189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15189: --- Priority: Blocker (was: Major) > add documentation for catalog view and hive view > > > Key: FLINK-15189 > URL: https://issues.apache.org/jira/browse/FLINK-15189 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Bowen Li >Assignee: Rui Li >Priority: Blocker > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15223) Csv connector should unescape delimiter parameter characters
[
https://issues.apache.org/jira/browse/FLINK-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15223:
---
Priority: Blocker (was: Major)
> Csv connector should unescape delimiter parameter characters
>
>
> Key: FLINK-15223
> URL: https://issues.apache.org/jira/browse/FLINK-15223
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Reporter: Jark Wu
>Priority: Blocker
> Fix For: 1.10.0
>
>
> As described in documentation[1], a csv format can use
> {{'format.line-delimiter' = '\n'}} to specify line delimiter. However, the
> property value is parsed into two characters "\n" , this result to reading
> failed. There is no workaround for now, unless fix it. The delimiter should
> be unescaped, e.g. using {{StringEscapeUtils.unescapeJava}}. Note that both
> old csv and new csv have the same problem, and both {{field-delimiter}} and
> {{line-delimiter}}.
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connect.html#old-csv-format
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15001) The digest of sub-plan reuse should contain retraction traits for stream physical nodes
[ https://issues.apache.org/jira/browse/FLINK-15001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15001: --- Fix Version/s: (was: 1.11.0) > The digest of sub-plan reuse should contain retraction traits for stream > physical nodes > --- > > Key: FLINK-15001 > URL: https://issues.apache.org/jira/browse/FLINK-15001 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.9.1 >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2, 1.10.0 > > Attachments: image-2019-12-02-10-49-46-916.png, > image-2019-12-02-10-52-01-399.png > > Time Spent: 20m > Remaining Estimate: 0h > > This bug is found in [FLINK-14946| > https://issues.apache.org/jira/browse/FLINK-14946]: > The plan for the given sql in [FLINK-14946| > https://issues.apache.org/jira/browse/FLINK-14946] is > !image-2019-12-02-10-49-46-916.png! > however, the plan after sub-plan reuse is: > !image-2019-12-02-10-52-01-399.png! > in the first picture, we could find that the accMode of two joins are > different, but the two joins are reused in the second picture. > The reason is the digest of sub-plan reuse does not contain retraction traits > for stream physical nodes now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15185) Hive sink can not run in standalone mode
[ https://issues.apache.org/jira/browse/FLINK-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15185. -- Resolution: Fixed master: 3a30e29043351e749df928b2bf86c578bb0f1bdb 1.10: cebe6e954917c0ec74a0310ef70bb0578bd8aa80 > Hive sink can not run in standalone mode > > > Key: FLINK-15185 > URL: https://issues.apache.org/jira/browse/FLINK-15185 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now in hive HadoopFileSystemFactory, we use > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem to get FileSystem. > But it should not work after we setting default child first class loader. > Because in standalone mode, the cluster has no hadoop dependency. So the > solution is: > # Add `flink-hadoop-fs` dependency to hive module, not work, because classes > with "org.apache.flink" prefix will always be loaded by parent class loader :( > # User add hadoop dependency to standalone cluster, it breaks out-of-the-box. > # Shade hadoop FileSystem in hive module, not complex, good. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15185) Hive sink can not run in standalone mode
[ https://issues.apache.org/jira/browse/FLINK-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-15185: -- Assignee: Jingsong Lee > Hive sink can not run in standalone mode > > > Key: FLINK-15185 > URL: https://issues.apache.org/jira/browse/FLINK-15185 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Now in hive HadoopFileSystemFactory, we use > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem to get FileSystem. > But it should not work after we setting default child first class loader. > Because in standalone mode, the cluster has no hadoop dependency. So the > solution is: > # Add `flink-hadoop-fs` dependency to hive module, not work, because classes > with "org.apache.flink" prefix will always be loaded by parent class loader :( > # User add hadoop dependency to standalone cluster, it breaks out-of-the-box. > # Shade hadoop FileSystem in hive module, not complex, good. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15073) sql client fails to run same query multiple times
[ https://issues.apache.org/jira/browse/FLINK-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15073. -- Resolution: Fixed 1.10.0: 4a5a720024992c12bbfd4fb316d04f24d23a109e > sql client fails to run same query multiple times > - > > Key: FLINK-15073 > URL: https://issues.apache.org/jira/browse/FLINK-15073 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Kurt Young >Assignee: Danny Chen >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Flink SQL> select abs(-1); > [INFO] Result retrieval cancelled. > Flink SQL> select abs(-1); > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.ValidationException: Table 'default: select > abs(-1)' already exists. Please choose a different name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15212) PROCTIME attribute causes problems with timestamp times before 1900 ?
[
https://issues.apache.org/jira/browse/FLINK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15212:
---
Component/s: (was: Table SQL / API)
Table SQL / Legacy Planner
> PROCTIME attribute causes problems with timestamp times before 1900 ?
> --
>
> Key: FLINK-15212
> URL: https://issues.apache.org/jira/browse/FLINK-15212
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.9.1
> Environment: flink 1.9.1
> jdk1.8.0_211
> idea2019.3
>Reporter: Rockey Cui
>Priority: Major
> Labels: easyfix
>
> A simple DataStreamSource with timestamp registered as a table.
>
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource = env.fromElements(
> "1001,1002,adc0,1900-01-01 00:00:00.0",
> "1002,1003,adc1,1910-01-01 00:00:00.0",
> "1003,1004,adc2,1920-01-01 00:00:00.0",
> "1004,1005,adc3,1930-01-01 00:00:00.0",
> "1005,1006,adc4,1970-01-01 00:00:00.0",
> ",,adc5,1971-01-01 00:00:00.0"
> );
> TypeInformation[] fieldTypes = new TypeInformation[]{Types.LONG,
> Types.LONG, Types.STRING, Types.SQL_TIM
> String[] fieldNames = new String[]{"id", "cityId", "url", "clickTime"};
> RowTypeInfo rowTypeInfo = new RowTypeInfo(fieldTypes, fieldNames);
> DataStream stream = stringDataStreamSource.map((MapFunction Row>) s -> {
> String[] split = s.split(",");
> Row row = new Row(split.length);
> for (int i = 0; i < split.length; i++) {
> Object value = null;
> if (fieldTypes[i].equals(Types.STRING)) {
> value = split[i];
> }
> if (fieldTypes[i].equals(Types.LONG)) {
> value = Long.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.INT)) {
> value = Integer.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.DOUBLE)) {
> value = Double.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.SQL_TIMESTAMP)) {
> value = Timestamp.valueOf(split[i]);
> }
> row.setField(i, value);
> }
> //System.out.println(row.toString());
> return row;
> }).returns(rowTypeInfo);
> tableEnv.registerDataStream("user_click_info", stream, String.join(",",
> fieldNames) + ",www.proctime");
> String sql = "select * from user_click_info";
> Table table = tableEnv.sqlQuery(sql);
> DataStream result = tableEnv.toAppendStream(table, Row.class);
> result.print();
> table.printSchema();
> tableEnv.execute("Test");
> {code}
> result ==>
>
> root
> |-- id: BIGINT
> |-- cityId: BIGINT
> |-- url: STRING
> |-- clickTime: TIMESTAMP(3)
> |-- www: TIMESTAMP(3) *PROCTIME*
>
> 1001,1002,adc0,{color:#FF}1899-12-31 23:54:17.0{color},2019-12-12
> 03:37:18.036
> 1002,1003,adc1,1910-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1003,1004,adc2,1920-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1004,1005,adc3,1930-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1005,1006,adc4,1970-01-01 00:00:00.0,2019-12-12 03:37:18.196
> ,,adc5,1971-01-01 00:00:00.0,2019-12-12 03:37:18.196
>
> without PROCTIME attribute is OK ==>
>
> root
> |-- id: BIGINT
> |-- cityId: BIGINT
> |-- url: STRING
> |-- clickTime: TIMESTAMP(3)
>
> 1001,1002,adc0,1900-01-01 00:00:00.0
> 1002,1003,adc1,1910-01-01 00:00:00.0
> 1003,1004,adc2,1920-01-01 00:00:00.0
> 1004,1005,adc3,1930-01-01 00:00:00.0
> 1005,1006,adc4,1970-01-01 00:00:00.0
> ,,adc5,1971-01-01 00:00:00.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15212) PROCTIME attribute causes problems with timestamp times before 1900 ?
[
https://issues.apache.org/jira/browse/FLINK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994282#comment-16994282
]
Kurt Young commented on FLINK-15212:
[~docete] Do we have a chance to first fix it in Flink?
> PROCTIME attribute causes problems with timestamp times before 1900 ?
> --
>
> Key: FLINK-15212
> URL: https://issues.apache.org/jira/browse/FLINK-15212
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.1
> Environment: flink 1.9.1
> jdk1.8.0_211
> idea2019.3
>Reporter: Rockey Cui
>Priority: Major
> Labels: easyfix
>
> A simple DataStreamSource with timestamp registered as a table.
>
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource = env.fromElements(
> "1001,1002,adc0,1900-01-01 00:00:00.0",
> "1002,1003,adc1,1910-01-01 00:00:00.0",
> "1003,1004,adc2,1920-01-01 00:00:00.0",
> "1004,1005,adc3,1930-01-01 00:00:00.0",
> "1005,1006,adc4,1970-01-01 00:00:00.0",
> ",,adc5,1971-01-01 00:00:00.0"
> );
> TypeInformation[] fieldTypes = new TypeInformation[]{Types.LONG,
> Types.LONG, Types.STRING, Types.SQL_TIM
> String[] fieldNames = new String[]{"id", "cityId", "url", "clickTime"};
> RowTypeInfo rowTypeInfo = new RowTypeInfo(fieldTypes, fieldNames);
> DataStream stream = stringDataStreamSource.map((MapFunction Row>) s -> {
> String[] split = s.split(",");
> Row row = new Row(split.length);
> for (int i = 0; i < split.length; i++) {
> Object value = null;
> if (fieldTypes[i].equals(Types.STRING)) {
> value = split[i];
> }
> if (fieldTypes[i].equals(Types.LONG)) {
> value = Long.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.INT)) {
> value = Integer.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.DOUBLE)) {
> value = Double.valueOf(split[i]);
> }
> if (fieldTypes[i].equals(Types.SQL_TIMESTAMP)) {
> value = Timestamp.valueOf(split[i]);
> }
> row.setField(i, value);
> }
> //System.out.println(row.toString());
> return row;
> }).returns(rowTypeInfo);
> tableEnv.registerDataStream("user_click_info", stream, String.join(",",
> fieldNames) + ",www.proctime");
> String sql = "select * from user_click_info";
> Table table = tableEnv.sqlQuery(sql);
> DataStream result = tableEnv.toAppendStream(table, Row.class);
> result.print();
> table.printSchema();
> tableEnv.execute("Test");
> {code}
> result ==>
>
> root
> |-- id: BIGINT
> |-- cityId: BIGINT
> |-- url: STRING
> |-- clickTime: TIMESTAMP(3)
> |-- www: TIMESTAMP(3) *PROCTIME*
>
> 1001,1002,adc0,{color:#FF}1899-12-31 23:54:17.0{color},2019-12-12
> 03:37:18.036
> 1002,1003,adc1,1910-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1003,1004,adc2,1920-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1004,1005,adc3,1930-01-01 00:00:00.0,2019-12-12 03:37:18.196
> 1005,1006,adc4,1970-01-01 00:00:00.0,2019-12-12 03:37:18.196
> ,,adc5,1971-01-01 00:00:00.0,2019-12-12 03:37:18.196
>
> without PROCTIME attribute is OK ==>
>
> root
> |-- id: BIGINT
> |-- cityId: BIGINT
> |-- url: STRING
> |-- clickTime: TIMESTAMP(3)
>
> 1001,1002,adc0,1900-01-01 00:00:00.0
> 1002,1003,adc1,1910-01-01 00:00:00.0
> 1003,1004,adc2,1920-01-01 00:00:00.0
> 1004,1005,adc3,1930-01-01 00:00:00.0
> 1005,1006,adc4,1970-01-01 00:00:00.0
> ,,adc5,1971-01-01 00:00:00.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15019) publish blog and document Flink-Hive benchmark v.s. Hive on MR/Tez
[ https://issues.apache.org/jira/browse/FLINK-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993588#comment-16993588 ] Kurt Young commented on FLINK-15019: Sounds great, thanks! [~phoenixjiangnan] > publish blog and document Flink-Hive benchmark v.s. Hive on MR/Tez > -- > > Key: FLINK-15019 > URL: https://issues.apache.org/jira/browse/FLINK-15019 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Bowen Li >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > need some PR work, like blog and documentation, to drive users adoptions of > Flink-Hive integration and Flink batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15175) syntax not supported in SQLClient for TPCDS queries
[
https://issues.apache.org/jira/browse/FLINK-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993584#comment-16993584
]
Kurt Young edited comment on FLINK-15175 at 12/11/19 2:18 PM:
--
The final plan would be FLINK-14671, we could do a temporary fix for this
version. Feel free to open a PR [~liupengcheng]
was (Author: ykt836):
The final plan would be FLINK-14671
> syntax not supported in SQLClient for TPCDS queries
>
>
> Key: FLINK-15175
> URL: https://issues.apache.org/jira/browse/FLINK-15175
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: liupengcheng
>Priority: Major
>
> {code:java}
> Flink SQL> WITH customer_total_return AS
> > ( SELECT
> > sr_customer_sk AS ctr_customer_sk,
> > sr_store_sk AS ctr_store_sk,
> > sum(sr_return_amt) AS ctr_total_return
> > FROM store_returns, date_dim
> > WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
> > GROUP BY sr_customer_sk, sr_store_sk)
> > SELECT c_customer_id
> > FROM customer_total_return ctr1, store, customer
> > WHERE ctr1.ctr_total_return >
> > (SELECT avg(ctr_total_return) * 1.2
> > FROM customer_total_return ctr2
> > WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> > AND s_store_sk = ctr1.ctr_store_sk
> > AND s_state = 'TN'
> > AND ctr1.ctr_customer_sk = c_customer_sk
> > ORDER BY c_customer_id
> > LIMIT 100;
> [ERROR] Unknown or invalid SQL statement.
> {code}
> It seems that the newest branch already support all TPCDS queries, but
> currently the sql client parser has not supported yet.
> Anyone already working on this? If not I can try it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15175) syntax not supported in SQLClient for TPCDS queries
[
https://issues.apache.org/jira/browse/FLINK-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15175:
--
Assignee: liupengcheng
> syntax not supported in SQLClient for TPCDS queries
>
>
> Key: FLINK-15175
> URL: https://issues.apache.org/jira/browse/FLINK-15175
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: liupengcheng
>Assignee: liupengcheng
>Priority: Major
>
> {code:java}
> Flink SQL> WITH customer_total_return AS
> > ( SELECT
> > sr_customer_sk AS ctr_customer_sk,
> > sr_store_sk AS ctr_store_sk,
> > sum(sr_return_amt) AS ctr_total_return
> > FROM store_returns, date_dim
> > WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
> > GROUP BY sr_customer_sk, sr_store_sk)
> > SELECT c_customer_id
> > FROM customer_total_return ctr1, store, customer
> > WHERE ctr1.ctr_total_return >
> > (SELECT avg(ctr_total_return) * 1.2
> > FROM customer_total_return ctr2
> > WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> > AND s_store_sk = ctr1.ctr_store_sk
> > AND s_state = 'TN'
> > AND ctr1.ctr_customer_sk = c_customer_sk
> > ORDER BY c_customer_id
> > LIMIT 100;
> [ERROR] Unknown or invalid SQL statement.
> {code}
> It seems that the newest branch already support all TPCDS queries, but
> currently the sql client parser has not supported yet.
> Anyone already working on this? If not I can try it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15175) syntax not supported in SQLClient for TPCDS queries
[
https://issues.apache.org/jira/browse/FLINK-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993584#comment-16993584
]
Kurt Young commented on FLINK-15175:
The final plan would be FLINK-14671
> syntax not supported in SQLClient for TPCDS queries
>
>
> Key: FLINK-15175
> URL: https://issues.apache.org/jira/browse/FLINK-15175
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: liupengcheng
>Priority: Major
>
> {code:java}
> Flink SQL> WITH customer_total_return AS
> > ( SELECT
> > sr_customer_sk AS ctr_customer_sk,
> > sr_store_sk AS ctr_store_sk,
> > sum(sr_return_amt) AS ctr_total_return
> > FROM store_returns, date_dim
> > WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
> > GROUP BY sr_customer_sk, sr_store_sk)
> > SELECT c_customer_id
> > FROM customer_total_return ctr1, store, customer
> > WHERE ctr1.ctr_total_return >
> > (SELECT avg(ctr_total_return) * 1.2
> > FROM customer_total_return ctr2
> > WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> > AND s_store_sk = ctr1.ctr_store_sk
> > AND s_state = 'TN'
> > AND ctr1.ctr_customer_sk = c_customer_sk
> > ORDER BY c_customer_id
> > LIMIT 100;
> [ERROR] Unknown or invalid SQL statement.
> {code}
> It seems that the newest branch already support all TPCDS queries, but
> currently the sql client parser has not supported yet.
> Anyone already working on this? If not I can try it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15175) syntax not supported in SQLClient for TPCDS queries
[
https://issues.apache.org/jira/browse/FLINK-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992490#comment-16992490
]
Kurt Young commented on FLINK-15175:
Hi [~liupengcheng], thanks for reporting this. We will look into this, and my
gut feeling is there maybe some disconnection between SQL CLI and Flink SQL.
> syntax not supported in SQLClient for TPCDS queries
>
>
> Key: FLINK-15175
> URL: https://issues.apache.org/jira/browse/FLINK-15175
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: liupengcheng
>Priority: Major
>
> {code:java}
> Flink SQL> WITH customer_total_return AS
> > ( SELECT
> > sr_customer_sk AS ctr_customer_sk,
> > sr_store_sk AS ctr_store_sk,
> > sum(sr_return_amt) AS ctr_total_return
> > FROM store_returns, date_dim
> > WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
> > GROUP BY sr_customer_sk, sr_store_sk)
> > SELECT c_customer_id
> > FROM customer_total_return ctr1, store, customer
> > WHERE ctr1.ctr_total_return >
> > (SELECT avg(ctr_total_return) * 1.2
> > FROM customer_total_return ctr2
> > WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> > AND s_store_sk = ctr1.ctr_store_sk
> > AND s_state = 'TN'
> > AND ctr1.ctr_customer_sk = c_customer_sk
> > ORDER BY c_customer_id
> > LIMIT 100;
> [ERROR] Unknown or invalid SQL statement.
> {code}
> It seems that the newest branch already support all TPCDS queries, but
> currently the sql client parser has not supported yet.
> Anyone already working on this? If not I can try it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14953) Parquet table source should use schema type to build FilterPredicate
[ https://issues.apache.org/jira/browse/FLINK-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14953. -- Resolution: Fixed 1.10.0: b6801d50ade388b28d349711fdf75e9f75b562da 1.9.2: f65be9b37f0405b1605ecec533614ff2aab3cd52 > Parquet table source should use schema type to build FilterPredicate > > > Key: FLINK-14953 > URL: https://issues.apache.org/jira/browse/FLINK-14953 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0, 1.8.2, 1.9.0, 1.9.1 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > The issue happens when the data type of value in predicate inferred from SQL > doesn't match the parquet schema. For example, foo is a long type, foo < 1 is > the predicate. Literal will be recognized as an integration. It causes the > parquet FilterPredicate is mistakenly created for the column of Integer type. > Then, the exception comes. > java.lang.UnsupportedOperationException > at > org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate$ValueInspector.update(IncrementallyUpdatedFilterPredicate.java:71) > at > org.apache.parquet.filter2.recordlevel.FilteringPrimitiveConverter.addLong(FilteringPrimitiveConverter.java:105) > at > org.apache.parquet.column.impl.ColumnReaderImpl$2$4.writeValue(ColumnReaderImpl.java:268) > at > org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367) > at > org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.readNextRecord(ParquetRecordReader.java:235) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.reachEnd(ParquetRecordReader.java:207) > at > org.apache.flink.formats.parquet.ParquetInputFormat.reachedEnd(ParquetInputFormat.java:233) > at > org.apache.flink.api.common.operators.GenericDataSourceBase.executeOnCollections(GenericDataSourceBase.java:231) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSource(CollectionExecutor.java:219) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:155) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeUnaryOperator(CollectionExecutor.java:229) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:149) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSink(CollectionExecutor.java:182) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:158) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:115) > at > org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:38) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:52) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:47) > at org.apache.flink.api.java.DataSet.collect(DataSet.java:413) > at > org.apache.flink.formats.parquet.ParquetTableSourceITCase.testScanWithProjectionAndFilter(ParquetTableSourceITCase.java:91) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at
[jira] [Assigned] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-13437: -- Assignee: Terry Wang (was: Jingsong Lee) > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Terry Wang >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992208#comment-16992208 ] Kurt Young commented on FLINK-13437: [~liyu] We will try to do this in 1.10 > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-13437: -- Assignee: Jingsong Lee > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13437: --- Fix Version/s: 1.10.0 > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992202#comment-16992202 ] Kurt Young commented on FLINK-15017: cc [~godfreyhe] [~docete] > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13437: --- Fix Version/s: (was: 1.10.0) > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Priority: Major > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-11899: -- Assignee: Zhenqiu Huang (was: Jingsong Lee) > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Zhenqiu Huang >Priority: Major > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992136#comment-16992136 ] Kurt Young commented on FLINK-11899: Thanks [~ZhenqiuHuang], will assign to you. BTW we have already implemented vectorized orc input format, you can take as an example if needed. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992126#comment-16992126 ] Kurt Young edited comment on FLINK-15017 at 12/10/19 2:31 AM: -- I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we start doing this. was (Author: ykt836): I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we starting to do this. > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992126#comment-16992126 ] Kurt Young commented on FLINK-15017: I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we starting to do this. > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14649) Flatten all the connector properties keys to make it easy to configure in DDL
[
https://issues.apache.org/jira/browse/FLINK-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-14649:
---
Release Note: {{PlaceHolder}}
> Flatten all the connector properties keys to make it easy to configure in DDL
> -
>
> Key: FLINK-14649
> URL: https://issues.apache.org/jira/browse/FLINK-14649
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are some connector properties are lists. For example, Kafka connector
> specific properties have to been set in the following way:
> {code}
> 'connector.properties.0.key' = 'zookeeper.connect',
> 'connector.properties.0.value' = 'localhost:2181',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'localhost:9092',
> 'connector.properties.2.key' = 'group.id',
> 'connector.properties.2.value' = 'testGroup',
> {code}
> It is complex and not intuitive to define in this way. In order to cooperate
> with DDL better, we propose to flatten all the property keys.
> It has some disadvantage to define in this way.
> - Users need to keep track of the indices
> - The key space is not constant. Validation of keys would require prefix
> magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
> - It is complex and not intuitive to define and document.
> See FLIP-86 for the proposed new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-13438) Support date type in Hive
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-13438. -- Resolution: Fixed master: 2994f0e44b53c85535f2f29fb43d320ace91f6f8 > Support date type in Hive > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Caizhi Weng >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Attachments: 0001-hive.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14649) Flatten all the connector properties keys to make it easy to configure in DDL
[
https://issues.apache.org/jira/browse/FLINK-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991469#comment-16991469
]
Kurt Young commented on FLINK-14649:
Needs to attach some release notes to this jira
> Flatten all the connector properties keys to make it easy to configure in DDL
> -
>
> Key: FLINK-14649
> URL: https://issues.apache.org/jira/browse/FLINK-14649
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are some connector properties are lists. For example, Kafka connector
> specific properties have to been set in the following way:
> {code}
> 'connector.properties.0.key' = 'zookeeper.connect',
> 'connector.properties.0.value' = 'localhost:2181',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'localhost:9092',
> 'connector.properties.2.key' = 'group.id',
> 'connector.properties.2.value' = 'testGroup',
> {code}
> It is complex and not intuitive to define in this way. In order to cooperate
> with DDL better, we propose to flatten all the property keys.
> It has some disadvantage to define in this way.
> - Users need to keep track of the indices
> - The key space is not constant. Validation of keys would require prefix
> magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
> - It is complex and not intuitive to define and document.
> See FLIP-86 for the proposed new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15147) Add doc for alter table set properties and rename table ddl
[ https://issues.apache.org/jira/browse/FLINK-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15147: --- Fix Version/s: 1.10.0 > Add doc for alter table set properties and rename table ddl > --- > > Key: FLINK-15147 > URL: https://issues.apache.org/jira/browse/FLINK-15147 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Terry Wang >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15147) Add doc for alter table set properties and rename table ddl
[ https://issues.apache.org/jira/browse/FLINK-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15147: --- Priority: Blocker (was: Major) > Add doc for alter table set properties and rename table ddl > --- > > Key: FLINK-15147 > URL: https://issues.apache.org/jira/browse/FLINK-15147 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Terry Wang >Priority: Blocker > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15148) Add doc for create/drop/alter database ddl
[ https://issues.apache.org/jira/browse/FLINK-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15148: --- Fix Version/s: 1.10.0 > Add doc for create/drop/alter database ddl > -- > > Key: FLINK-15148 > URL: https://issues.apache.org/jira/browse/FLINK-15148 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Terry Wang >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15148) Add doc for create/drop/alter database ddl
[ https://issues.apache.org/jira/browse/FLINK-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15148: --- Priority: Blocker (was: Major) > Add doc for create/drop/alter database ddl > -- > > Key: FLINK-15148 > URL: https://issues.apache.org/jira/browse/FLINK-15148 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Terry Wang >Priority: Blocker > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14513) Implement listPartitionsByFilter to HiveCatalog
[ https://issues.apache.org/jira/browse/FLINK-14513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14513. -- Assignee: Rui Li Resolution: Fixed master: 47f29391cf4c0c856e2401289c06658471508654 > Implement listPartitionsByFilter to HiveCatalog > --- > > Key: FLINK-14513 > URL: https://issues.apache.org/jira/browse/FLINK-14513 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Jingsong Lee >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15004:
---
Affects Version/s: (was: 1.9.0)
1.10.0
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.10.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14133) Improve batch sql and hive integrate performance milestone-1
[ https://issues.apache.org/jira/browse/FLINK-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14133. -- Assignee: Jingsong Lee Resolution: Fixed > Improve batch sql and hive integrate performance milestone-1 > > > Key: FLINK-14133 > URL: https://issues.apache.org/jira/browse/FLINK-14133 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > Now we have basically merged the batch function of blink planner and > basically integrated hive. But there are still many problems with > performance, and we need to ensure basic performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15004:
---
Parent: (was: FLINK-14133)
Issue Type: Improvement (was: Sub-task)
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15141) Using decimal type in a sink table, the result returns a not match ValidationException
[ https://issues.apache.org/jira/browse/FLINK-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991228#comment-16991228 ] Kurt Young commented on FLINK-15141: I think it's highly possible the same issue with FLINK-15124, after fix FLINK-15124, we can double check this issue again > Using decimal type in a sink table, the result returns a not match > ValidationException > --- > > Key: FLINK-15141 > URL: https://issues.apache.org/jira/browse/FLINK-15141 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > The planner what I used is blink. > *The source table is:* > CREATE TABLE `aggtest` ( > a smallint, > b float > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/sources/aggtest.csv', > 'format.type'='csv' > ); > > *The sink table is:* > CREATE TABLE `agg_decimal_res` ( > avg_107_943 DECIMAL(10, 3) > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/test_aggregates__test_avg_cast_batch/results/agg_decimal_res.csv', > 'format.type'='csv' > ); > > *The sql is:* > INSERT INTO agg_decimal_res SELECT CAST(avg(b) AS numeric(10,3)) AS > avg_107_943 FROM aggtest; > > After execute the sql, there will be a exception appear, just like this: > [INFO] Submitting SQL update statement to the cluster... > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.ValidationException: Field types of query result > and registered TableSink > `default_catalog`.`default_database`.`agg_decimal_res1` do not match. > Query result schema: [avg_107_943: DECIMAL(10, 3)] > TableSink schema: [avg_107_943: DECIMAL(38, 18)] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-13438) Hive source/sink/udx should respect the conversion class of DataType
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-13438: -- Assignee: Rui Li (was: Caizhi Weng) > Hive source/sink/udx should respect the conversion class of DataType > > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Caizhi Weng >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Attachments: 0001-hive.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15093) StreamExecutionEnvironment does not clear transformations when executing
[
https://issues.apache.org/jira/browse/FLINK-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15093:
--
Assignee: Danny Chen
> StreamExecutionEnvironment does not clear transformations when executing
>
>
> Key: FLINK-15093
> URL: https://issues.apache.org/jira/browse/FLINK-15093
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0, 1.9.1
>Reporter: Jeff Zhang
>Assignee: Danny Chen
>Priority: Critical
> Fix For: 1.10.0
>
> Attachments: screenshot-1.png
>
>
> Use the following code in scala shell to reproduce this issue.
> {code}
> val data = senv.fromElements("hello world", "hello flink", "hello hadoop")
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15093) StreamExecutionEnvironment does not clear transformations when executing
[
https://issues.apache.org/jira/browse/FLINK-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15093:
---
Priority: Blocker (was: Critical)
> StreamExecutionEnvironment does not clear transformations when executing
>
>
> Key: FLINK-15093
> URL: https://issues.apache.org/jira/browse/FLINK-15093
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0, 1.9.1
>Reporter: Jeff Zhang
>Assignee: Danny Chen
>Priority: Blocker
> Fix For: 1.10.0
>
> Attachments: screenshot-1.png
>
>
> Use the following code in scala shell to reproduce this issue.
> {code}
> val data = senv.fromElements("hello world", "hello flink", "hello hadoop")
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15093) StreamExecutionEnvironment does not clear transformations when executing
[
https://issues.apache.org/jira/browse/FLINK-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15093:
---
Affects Version/s: (was: 1.9.1)
> StreamExecutionEnvironment does not clear transformations when executing
>
>
> Key: FLINK-15093
> URL: https://issues.apache.org/jira/browse/FLINK-15093
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Assignee: Danny Chen
>Priority: Blocker
> Fix For: 1.10.0
>
> Attachments: screenshot-1.png
>
>
> Use the following code in scala shell to reproduce this issue.
> {code}
> val data = senv.fromElements("hello world", "hello flink", "hello hadoop")
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> data.flatMap(line => line.split("\\s")).
> map(w => (w, 1)).
> keyBy(0).
> sum(1).
> print
> senv.execute()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15128) Document support for Hive timestamp type
[ https://issues.apache.org/jira/browse/FLINK-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15128: --- Priority: Blocker (was: Major) > Document support for Hive timestamp type > > > Key: FLINK-15128 > URL: https://issues.apache.org/jira/browse/FLINK-15128 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15128) Document support for Hive timestamp type
[ https://issues.apache.org/jira/browse/FLINK-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15128: --- Fix Version/s: 1.10.0 > Document support for Hive timestamp type > > > Key: FLINK-15128 > URL: https://issues.apache.org/jira/browse/FLINK-15128 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Priority: Blocker > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15020) Support timestamp type in hive
[ https://issues.apache.org/jira/browse/FLINK-15020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15020. -- Assignee: Rui Li Resolution: Fixed master: 57bb952e3a1732caa5acb640a1712bcdd55d813d > Support timestamp type in hive > -- > > Key: FLINK-15020 > URL: https://issues.apache.org/jira/browse/FLINK-15020 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Jingsong Lee >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now, FLINK-14599 is finished, blink-planner have the ability to support > timestamp type with precision 9. So we can support timestamp type in hive now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15095) bridge table schema's primary key to metadata handler in blink planner
[ https://issues.apache.org/jira/browse/FLINK-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15095. -- Resolution: Fixed master: d4bd2d6ee64ba83c42981e4e62374fb962bb71bf > bridge table schema's primary key to metadata handler in blink planner > -- > > Key: FLINK-15095 > URL: https://issues.apache.org/jira/browse/FLINK-15095 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14970) Doomed test for equality to NaN
[
https://issues.apache.org/jira/browse/FLINK-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-14970.
--
Fix Version/s: 1.10.0
Resolution: Fixed
master: 3e28aedbaa85710aab34c0abb3a3973d73f7f805
> Doomed test for equality to NaN
> ---
>
> Key: FLINK-14970
> URL: https://issues.apache.org/jira/browse/FLINK-14970
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Legacy Planner, Table SQL / Planner
>Affects Versions: 1.8.2, 1.9.0, 1.9.1
>Reporter: Dezhi Cai
>Assignee: Dezhi Cai
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Attachments: image-2019-11-27-16-51-00-150.png,
> image-2019-11-27-16-51-06-801.png, image-2019-12-04-13-39-38-549.png,
> image-2019-12-04-13-40-19-441.png, image-2019-12-04-13-55-14-117.png,
> image-2019-12-04-13-56-14-815.png, image-2019-12-04-13-58-14-131.png,
> image-2019-12-04-14-01-27-420.png, image-2019-12-06-17-31-13-627.png,
> image-2019-12-06-17-31-22-230.png, image-2019-12-06-17-32-37-195.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> scala pattern matching can't handle "NaN".
> in GenerateUtils, CodeGenerator, some code logic fall into this issue(as
> there is no way to get {{NaN}} out of {{BigDecimal,}}
> {{so the code does not cause any impact}}), please refer to
> the screenshot for details.
> {code:java}
> def main(args: Array[String]): Unit = {
> val floatVaue = Float.NaN
> floatVaue match {
> case Float.NaN => println("Float.NaN")
> case _ => println("not match")
> }
> }
> will output: not match
> {code}
> {code:java}
> // this one works
> def main(args: Array[String]): Unit = {
> val floatVaue = Float.NaN
> floatVaue match {
> case value if value.isNaN => println("Float.NaN")
> case _ => println("not match")
> }
> }
> will output: Float.NaN
> {code}
>
>
> org.apache.flink.table.planner.codegen.GenerateUtils
> !image-2019-11-27-16-51-06-801.png|width=723,height=257!
> org.apache.flink.table.codegen.CodeGenerator
> !image-2019-11-27-16-51-00-150.png|width=727,height=158!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15124) types with precision can't be executed in sql client with blink planner
Kurt Young created FLINK-15124:
--
Summary: types with precision can't be executed in sql client with
blink planner
Key: FLINK-15124
URL: https://issues.apache.org/jira/browse/FLINK-15124
Project: Flink
Issue Type: Bug
Components: Table SQL / Client, Table SQL / Planner
Affects Versions: 1.10.0
Reporter: Kurt Young
I created a table in sql client with blink planner:
{noformat}
create table t (
a int,
b varchar,
c decimal(10, 5))
with (
'connector.type' = 'filesystem',
'format.type' = 'csv',
'format.derive-schema' = 'true',
'connector.path' = 'xxx'
);
{noformat}
The table description looks good:
{noformat}
Flink SQL> describe t;
root
|-- a: INT
|-- b: STRING
|-- c: DECIMAL(10, 5){noformat}
But the select query failed:
{noformat}
Flink SQL> select * from t;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.planner.codegen.CodeGenException: Incompatible types of
expression and result type.
Expression[GeneratedExpression(field$3,isNull$3,,DECIMAL(38, 18),None)] type is
[DECIMAL(38, 18)], result type is [DECIMAL(10, 5)]
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15124) types with precision can't be executed in sql client with blink planner
[
https://issues.apache.org/jira/browse/FLINK-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15124:
---
Fix Version/s: 1.10.0
> types with precision can't be executed in sql client with blink planner
> ---
>
> Key: FLINK-15124
> URL: https://issues.apache.org/jira/browse/FLINK-15124
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client, Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Kurt Young
>Priority: Major
> Fix For: 1.10.0
>
>
> I created a table in sql client with blink planner:
> {noformat}
> create table t (
> a int,
> b varchar,
> c decimal(10, 5))
> with (
> 'connector.type' = 'filesystem',
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true',
> 'connector.path' = 'xxx'
> );
> {noformat}
> The table description looks good:
> {noformat}
> Flink SQL> describe t;
> root
> |-- a: INT
> |-- b: STRING
> |-- c: DECIMAL(10, 5){noformat}
> But the select query failed:
> {noformat}
> Flink SQL> select * from t;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.planner.codegen.CodeGenException: Incompatible types
> of expression and result type.
> Expression[GeneratedExpression(field$3,isNull$3,,DECIMAL(38, 18),None)] type
> is [DECIMAL(38, 18)], result type is [DECIMAL(10, 5)]
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15107) SQL-CLI can not execute insert into statement with lowercase "INSERT INTO" keyword
[ https://issues.apache.org/jira/browse/FLINK-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15107. -- Assignee: Terry Wang Resolution: Fixed master: 6b389ebef0751d54f53f5fa59a359283e29586ae > SQL-CLI can not execute insert into statement with lowercase "INSERT INTO" > keyword > -- > > Key: FLINK-15107 > URL: https://issues.apache.org/jira/browse/FLINK-15107 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Danny Chen >Assignee: Terry Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This was introduced by FLINK-15026 which has a always uppercase case "INSERT > INTO" pattern matcher check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15035) Introduce unknown memory setting to table in blink planner
[ https://issues.apache.org/jira/browse/FLINK-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15035. -- Resolution: Fixed master: d5853467fd70f22a119ed00166e4523a4907bfeb > Introduce unknown memory setting to table in blink planner > -- > > Key: FLINK-15035 > URL: https://issues.apache.org/jira/browse/FLINK-15035 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > After https://jira.apache.org/jira/browse/FLINK-14566 > We can just set unknown resources with setting whether managed memory is used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-13195) Add create table support for SqlClient
[ https://issues.apache.org/jira/browse/FLINK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-13195. -- Fix Version/s: 1.10.0 Resolution: Fixed master: caff2ed21d548996c89843a23ee216689616c961 > Add create table support for SqlClient > -- > > Key: FLINK-13195 > URL: https://issues.apache.org/jira/browse/FLINK-13195 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Danny Chen >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15117) JoinITCase.testFullJoinWithLeftLocalPred fails
[
https://issues.apache.org/jira/browse/FLINK-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-15117.
--
Resolution: Duplicate
> JoinITCase.testFullJoinWithLeftLocalPred fails
> --
>
> Key: FLINK-15117
> URL: https://issues.apache.org/jira/browse/FLINK-15117
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0
>Reporter: Bowen Li
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> {code:java}
> 11:27:57.153 [ERROR] Tests run: 180, Failures: 0, Errors: 1, Skipped: 68,
> Time elapsed: 31.709 s <<< FAILURE! - in
> org.apache.flink.table.planner.runtime.batch.sql.join.JoinITCase
> 11:27:57.153 [ERROR]
> testFullJoinWithLeftLocalPred[SortMergeJoin](org.apache.flink.table.planner.runtime.batch.sql.join.JoinITCase)
> Time elapsed: 0.34 s <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.table.planner.runtime.batch.sql.join.JoinITCase.testFullJoinWithLeftLocalPred(JoinITCase.scala:404)
> Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 32 pages. Only 0 pages are remaining.
> {code}
> https://travis-ci.com/flink-ci/flink/jobs/263716503
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15114) Add execute result info for alter/create/drop database in sql client.
[ https://issues.apache.org/jira/browse/FLINK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15114: --- Issue Type: Improvement (was: Bug) > Add execute result info for alter/create/drop database in sql client. > - > > Key: FLINK-15114 > URL: https://issues.apache.org/jira/browse/FLINK-15114 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Reporter: Terry Wang >Assignee: Terry Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add execute result info for alter/create/drop database in sql-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15114) Add execute result info for alter/create/drop database in sql client.
[ https://issues.apache.org/jira/browse/FLINK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15114. -- Resolution: Fixed master: 26431ec71bb65bd9a80d7674ec573aa0a71e901d > Add execute result info for alter/create/drop database in sql client. > - > > Key: FLINK-15114 > URL: https://issues.apache.org/jira/browse/FLINK-15114 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Reporter: Terry Wang >Assignee: Terry Wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add execute result info for alter/create/drop database in sql-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15114) Add execute result info for alter/create/drop database in sql client.
[ https://issues.apache.org/jira/browse/FLINK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15114: --- Priority: Minor (was: Major) > Add execute result info for alter/create/drop database in sql client. > - > > Key: FLINK-15114 > URL: https://issues.apache.org/jira/browse/FLINK-15114 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Reporter: Terry Wang >Assignee: Terry Wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add execute result info for alter/create/drop database in sql-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15027) Support alter table DDLs in SQL CLI
[
https://issues.apache.org/jira/browse/FLINK-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-15027.
--
Resolution: Fixed
master: 7af523dd130756b0bc1da38246d2daba44880a05
> Support alter table DDLs in SQL CLI
> ---
>
> Key: FLINK-15027
> URL: https://issues.apache.org/jira/browse/FLINK-15027
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Client
>Affects Versions: 1.9.1
>Reporter: Danny Chen
>Assignee: Terry Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Support syntax as following:
> {code:sql}
> ALTER TABLE [[catalogName.] dataBasesName].tableName RENAME TO newTableName
> ALTER TABLE [[catalogName.] dataBasesName].tableName SET ( name=value [,
> name=value]*)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15114) Add execute result info for alter/create/drop database in sql client.
[ https://issues.apache.org/jira/browse/FLINK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-15114: -- Assignee: Terry Wang > Add execute result info for alter/create/drop database in sql client. > - > > Key: FLINK-15114 > URL: https://issues.apache.org/jira/browse/FLINK-15114 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Terry Wang >Assignee: Terry Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Add execute result info for alter/create/drop database in sql-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14959) Support precision of LocalZonedTimestampType in blink planner
[ https://issues.apache.org/jira/browse/FLINK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14959. -- Resolution: Fixed master: fc2ef35b5fff94e822d4219e5d2022c4b5d55a03 6fdc0e67e563df65dab9c206e524b61795910ecf > Support precision of LocalZonedTimestampType in blink planner > - > > Key: FLINK-14959 > URL: https://issues.apache.org/jira/browse/FLINK-14959 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14512) Introduce listPartitionsByFilter to Catalog
[ https://issues.apache.org/jira/browse/FLINK-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14512. -- Resolution: Fixed master: 393a34fd760dbdba567dc98c8e52802d1dcbbcd5 > Introduce listPartitionsByFilter to Catalog > --- > > Key: FLINK-14512 > URL: https://issues.apache.org/jira/browse/FLINK-14512 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > planner will use catalog to do partition pruning. > So catalog need provide listPartitionsByFilter to partition pruning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15108) SQL-CLI can not execute same query multiple times
[ https://issues.apache.org/jira/browse/FLINK-15108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15108. -- Resolution: Duplicate > SQL-CLI can not execute same query multiple times > - > > Key: FLINK-15108 > URL: https://issues.apache.org/jira/browse/FLINK-15108 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Danny Chen >Priority: Major > Fix For: 1.10.0 > > > After we change the SQL-CLI to stateful in FLINK-14672, each query's temporal > table was left out so we can not re-registered the same object(from the same > query). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14847) Support retrieving Hive PK constraints
[ https://issues.apache.org/jira/browse/FLINK-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-14847. -- Resolution: Fixed master: 4cfe3ddd1ca1ad4b6d9158356d343fc47dbdb23d > Support retrieving Hive PK constraints > -- > > Key: FLINK-14847 > URL: https://issues.apache.org/jira/browse/FLINK-14847 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15098) CsvTableSink has no properties of a OVERWRITE mode to write, but the exception hint the user use it when a file or directory already exists
[ https://issues.apache.org/jira/browse/FLINK-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989757#comment-16989757 ] Kurt Young commented on FLINK-15098: I think this is the same issue which I was encountered: FLINK-15066 > CsvTableSink has no properties of a OVERWRITE mode to write, but the > exception hint the user use it when a file or directory already exists > --- > > Key: FLINK-15098 > URL: https://issues.apache.org/jira/browse/FLINK-15098 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > When using sql-client to execute a sql , there will be a IOException just > like the content below if the file or directory already exists. But there is > no place for the user to set the overwrite mode. In another word, the hint > tell users to use overwrite mode, but flink has not support it now. > > > The exception: > Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot > initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, > count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: > CsvTableSink(avg_1)': File or directory already exists. Existing files and > directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to > overwrite existing files and directories.Caused by: > org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task > 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) > -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or > directory already exists. Existing files and directories are not overwritten > in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and > directories. at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:243) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:215) > at > org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:203) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:119) > at > org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:104) > at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) > at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) > at > org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146) > ... 10 moreCaused by: java.io.IOException: File or directory already exists. > Existing files and directories are not overwritten in NO_OVERWRITE mode. Use > OVERWRITE mode to overwrite existing files and directories. at > org.apache.flink.core.fs.FileSystem.initOutPathDistFS(FileSystem.java:937) at > org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:286) > at > org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:100) > at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:212) > ... 20 more > > > > The properties that a csvTableSink support now: > connector.type, connector.property-version, connector.path, format.type, > format.property-version, format.field-delimiter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15098) CsvTableSink has no properties of a OVERWRITE mode to write, but the exception hint the user use it when a file or directory already exists
[ https://issues.apache.org/jira/browse/FLINK-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-15098. -- Resolution: Duplicate > CsvTableSink has no properties of a OVERWRITE mode to write, but the > exception hint the user use it when a file or directory already exists > --- > > Key: FLINK-15098 > URL: https://issues.apache.org/jira/browse/FLINK-15098 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > When using sql-client to execute a sql , there will be a IOException just > like the content below if the file or directory already exists. But there is > no place for the user to set the overwrite mode. In another word, the hint > tell users to use overwrite mode, but flink has not support it now. > > > The exception: > Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot > initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, > count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: > CsvTableSink(avg_1)': File or directory already exists. Existing files and > directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to > overwrite existing files and directories.Caused by: > org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task > 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) > -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or > directory already exists. Existing files and directories are not overwritten > in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and > directories. at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:243) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:215) > at > org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:203) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:119) > at > org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:104) > at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) > at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) > at > org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146) > ... 10 moreCaused by: java.io.IOException: File or directory already exists. > Existing files and directories are not overwritten in NO_OVERWRITE mode. Use > OVERWRITE mode to overwrite existing files and directories. at > org.apache.flink.core.fs.FileSystem.initOutPathDistFS(FileSystem.java:937) at > org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:286) > at > org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:100) > at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:212) > ... 20 more > > > > The properties that a csvTableSink support now: > connector.type, connector.property-version, connector.path, format.type, > format.property-version, format.field-delimiter -- This message was sent by Atlassian Jira (v8.3.4#803005)
