[jira] [Created] (FLINK-20578) Cannot create empty array using ARRAY[]
Fabian Hueske created FLINK-20578: - Summary: Cannot create empty array using ARRAY[] Key: FLINK-20578 URL: https://issues.apache.org/jira/browse/FLINK-20578 Project: Flink Issue Type: Sub-task Components: Table SQL / API Affects Versions: 1.11.2 Reporter: Fabian Hueske Calling the ARRAY function without an element (`ARRAY[]`) results in an error message. Is that the expected behavior? How can users create empty arrays? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20319) Improving the visitor pattern for operations
[ https://issues.apache.org/jira/browse/FLINK-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske updated FLINK-20319: -- Component/s: Table SQL / Planner > Improving the visitor pattern for operations > > > Key: FLINK-20319 > URL: https://issues.apache.org/jira/browse/FLINK-20319 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.11.2 >Reporter: Ingo Bürk >Priority: Minor > > The *OperationVisitor interfaces (which are not public API) don't always > implement the visitor pattern correctly, and some things which would be > useful are missing. Some things I discovered: > # CatalogSinkModifyOperation doesn't accept() its child. It's likely that > others have this problem as well, but I haven't checked further. > # The base Operation interface doesn't have an accept() method at all. > Potentially intentional since this interface actually is public API? > # There's a catch-all QueryOperationVisitor#visit(QueryOperation other) that > would be nice to split up into its subtypes (PlannerQueryOperation, …) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19790) Writing MAP to Kafka with JSON format produces incorrect data.
[
https://issues.apache.org/jira/browse/FLINK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219675#comment-17219675
]
Fabian Hueske commented on FLINK-19790:
---
Thanks for the quick response and working on a fix [~libenchao]! (y)
> Writing MAP to Kafka with JSON format produces incorrect data.
> --
>
> Key: FLINK-19790
> URL: https://issues.apache.org/jira/browse/FLINK-19790
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.11.2
>Reporter: Fabian Hueske
>Assignee: Benchao Li
>Priority: Critical
>
> Running the following SQL script writes incorrect data to Kafka:
> {code:java}
> CREATE TEMPORARY TABLE tmp_1 (m MAP) WITH (
> 'connector' = 'kafka',
> 'format' = 'json',
> 'properties.bootstrap.servers' = '...',
> 'properties.group.id' = '...',
> 'topic' = 'tmp-1'
> );
> CREATE TEMPORARY TABLE gen (k STRING, v STRING) WITH (
> 'connector' = 'datagen'
> );
> CREATE TEMPORARY VIEW gen_short AS
> SELECT SUBSTR(k, 0, 4) AS k, SUBSTR(v, 0, 4) AS v FROM gen;
> INSERT INTO tmp_1
> SELECT MAP[k, v] FROM gen_short; {code}
> Printing the content of the {{tmp-1}} topics results in the following output:
> {code:java}
> $ kafka-console-consumer --bootstrap-server ... --from-beginning --topic
> tmp-1 | head -n 5
> {"m":{"8a93":"6102"}}
> {"m":{"8a93":"6102","7922":"f737"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c","222c":"f3e2"}}
> {code}
> As you can see, the map is not correctly encoded as JSON and written to Kafka.
> I've run the query with the Blink planner with object reuse and operator
> pipelining disabled.
> Writing with Avro works as expected.
> Hence I assume that the JSON encoder/serializer reuses the Map object when
> encoding the JSON.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19790) Writing MAP to Kafka with JSON format produces incorrect data.
Fabian Hueske created FLINK-19790:
-
Summary: Writing MAP to Kafka with JSON format
produces incorrect data.
Key: FLINK-19790
URL: https://issues.apache.org/jira/browse/FLINK-19790
Project: Flink
Issue Type: Bug
Components: Table SQL / Ecosystem
Affects Versions: 1.11.2
Reporter: Fabian Hueske
Running the following SQL script writes incorrect data to Kafka:
{code:java}
CREATE TEMPORARY TABLE tmp_1 (m MAP) WITH (
'connector' = 'kafka',
'format' = 'json',
'properties.bootstrap.servers' = '...',
'properties.group.id' = '...',
'topic' = 'tmp-1'
);
CREATE TEMPORARY TABLE gen (k STRING, v STRING) WITH (
'connector' = 'datagen'
);
CREATE TEMPORARY VIEW gen_short AS
SELECT SUBSTR(k, 0, 4) AS k, SUBSTR(v, 0, 4) AS v FROM gen;
INSERT INTO tmp_1
SELECT MAP[k, v] FROM gen_short; {code}
Printing the content of the {{tmp-1}} topics results in the following output:
{code:java}
$ kafka-console-consumer --bootstrap-server ... --from-beginning --topic tmp-1
| head -n 5
{"m":{"8a93":"6102"}}
{"m":{"8a93":"6102","7922":"f737"}}
{"m":{"8a93":"6102","7922":"f737","9b63":"15b0"}}
{"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c"}}
{"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c","222c":"f3e2"}}
{code}
As you can see, the map is not correctly encoded as JSON and written to Kafka.
I've run the query with the Blink planner with object reuse and operator
pipelining disabled.
Writing with Avro works as expected.
Hence I assume that the JSON encoder/serializer reuses the Map object when
encoding the JSON.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
Fabian Hueske created FLINK-19398:
-
Summary: Hive connector fails with IllegalAccessError if submitted
as usercode
Key: FLINK-19398
URL: https://issues.apache.org/jira/browse/FLINK-19398
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Affects Versions: 1.11.2, 1.12.0
Reporter: Fabian Hueske
Using Flink's Hive connector fails if the dependency is loaded with the user
code classloader with the following exception.
{code:java}
java.lang.IllegalAccessError: tried to access method
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V
from class
org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder
at
org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127)
~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81)
~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT]
{code}
The problem is the constructor of {{Buckets}} with default visibility which is
called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as both
classes are loaded with the same classloader but when they are loaded in
different classloaders, the access fails.
{{Buckets}} is loaded with the system CL because it is part of
flink-streaming-java.
To solve this issue, we should change the visibility of the {{Buckets}}
constructor to {{public}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-19321) CollectSinkFunction does not define serialVersionUID
[
https://issues.apache.org/jira/browse/FLINK-19321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske reassigned FLINK-19321:
-
Assignee: CloseRiver
> CollectSinkFunction does not define serialVersionUID
>
>
> Key: FLINK-19321
> URL: https://issues.apache.org/jira/browse/FLINK-19321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Fabian Hueske
>Assignee: CloseRiver
>Priority: Major
>
> The {{org.apache.flink.streaming.api.operators.collect.CollectSinkFunction}}
> function does not define a {{serialVersionUID}}.
> Function objects are serialized using Java Serialization and should define a
> {{serialVersionUID.}}
>
> If no o serialVersionUID is defined, Java automatically generates IDs to
> check compatibility of objects during deserialization. However, the
> generation depends on the environment (JVM, class version, etc.) and can
> hence lead to {{[java.io|http://java.io/].InvalidClassException}} even if the
> classes are compatible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19321) CollectSinkFunction does not define serialVersionUID
[
https://issues.apache.org/jira/browse/FLINK-19321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199363#comment-17199363
]
Fabian Hueske commented on FLINK-19321:
---
Hi [~ZhuShang] , Thanks for helping here!
As far as I know, most Serializable classes in Flink just define it as {{1L}}.
> CollectSinkFunction does not define serialVersionUID
>
>
> Key: FLINK-19321
> URL: https://issues.apache.org/jira/browse/FLINK-19321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Fabian Hueske
>Priority: Major
>
> The {{org.apache.flink.streaming.api.operators.collect.CollectSinkFunction}}
> function does not define a {{serialVersionUID}}.
> Function objects are serialized using Java Serialization and should define a
> {{serialVersionUID.}}
>
> If no o serialVersionUID is defined, Java automatically generates IDs to
> check compatibility of objects during deserialization. However, the
> generation depends on the environment (JVM, class version, etc.) and can
> hence lead to {{[java.io|http://java.io/].InvalidClassException}} even if the
> classes are compatible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19321) CollectSinkFunction does not define serialVersionUID
Fabian Hueske created FLINK-19321:
-
Summary: CollectSinkFunction does not define serialVersionUID
Key: FLINK-19321
URL: https://issues.apache.org/jira/browse/FLINK-19321
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.11.2, 1.12.0
Reporter: Fabian Hueske
The {{org.apache.flink.streaming.api.operators.collect.CollectSinkFunction}}
function does not define a {{serialVersionUID}}.
Function objects are serialized using Java Serialization and should define a
{{serialVersionUID.}}
If no o serialVersionUID is defined, Java automatically generates IDs to check
compatibility of objects during deserialization. However, the generation
depends on the environment (JVM, class version, etc.) and can hence lead to
{{[java.io|http://java.io/].InvalidClassException}} even if the classes are
compatible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17845) Can't remove a table connector property with ALTER TABLE
[
https://issues.apache.org/jira/browse/FLINK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185820#comment-17185820
]
Fabian Hueske commented on FLINK-17845:
---
[~lzljs3620320] I think that's a goop proposal and would address the problem.
Nonetheless, {{ALTER TABLE t REPLACE PROPERTIES ()}} would be good to have
too, IMO.
> Can't remove a table connector property with ALTER TABLE
>
>
> Key: FLINK-17845
> URL: https://issues.apache.org/jira/browse/FLINK-17845
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Fabian Hueske
>Priority: Major
>
> It is not possible to remove an existing table property from a table.
> Looking at the [source
> code|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/sqlexec/SqlToOperationConverter.java#L295]
> this seems to be the intended semantics, but it seems counter-intuitive to
> me.
> If I create a table with the following statement:
> {code}
> CREATE TABLE `testTable` (
> id INT
> )
> WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topicX' = 'test', -- Woops, I made a typo here
> [...]
> )
> {code}
> The statement will be successfully executed. However, the table cannot be
> used due to the typo.
> Fixing the typo with the following DDL is not possible:
> {code}
> ALTER TABLE `testTable` SET (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'test', -- Fixing the typo
> )
> {code}
> because the key {{connector.topicX}} is not removed.
> Right now it seems that the only way to fix a table with an invalid key is to
> DROP and CREATE it. I think that this use case should be supported by ALTER
> TABLE.
> I would even argue that the expected behavior is that previous properties are
> removed and replaced by the new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18988) Continuous query with LATERAL and LIMIT produces wrong result
[
https://issues.apache.org/jira/browse/FLINK-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17180331#comment-17180331
]
Fabian Hueske commented on FLINK-18988:
---
[~danny0405] Yes, I used the Blink planner for this. Thanks for picking up this
issue!
> Continuous query with LATERAL and LIMIT produces wrong result
> -
>
> Key: FLINK-18988
> URL: https://issues.apache.org/jira/browse/FLINK-18988
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: Fabian Hueske
>Assignee: Danny Chen
>Priority: Critical
>
> I was trying out the example queries provided in this blog post:
> [https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
> Flink supports the same and found that the queries were translated and
> executed but produced the wrong result.
> I used the SQL Client and Kafka (running at kafka:9092) to store the table
> data. I executed the following statements:
> {code:java}
> -- create cities table
> CREATE TABLE cities (
> name STRING NOT NULL,
> state STRING NOT NULL,
> pop INT NOT NULL
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'cities',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'mygroup',
> 'scan.startup.mode' = 'earliest-offset',
> 'format' = 'json'
> );
> -- fill cities table
> INSERT INTO cities VALUES
> ('Los_Angeles', 'CA', 3979576),
> ('Phoenix', 'AZ', 1680992),
> ('Houston', 'TX', 2320268),
> ('San_Diego', 'CA', 1423851),
> ('San_Francisco', 'CA', 881549),
> ('New_York', 'NY', 8336817),
> ('Dallas', 'TX', 1343573),
> ('San_Antonio', 'TX', 1547253),
> ('San_Jose', 'CA', 1021795),
> ('Chicago', 'IL', 2695598),
> ('Austin', 'TX', 978908);
> -- execute query
> SELECT state, name
> FROM
> (SELECT DISTINCT state FROM cities) states,
> LATERAL (
> SELECT name, pop
> FROM cities
> WHERE state = states.state
> ORDER BY pop
> DESC LIMIT 3
> );
> -- result
> state name
>CA Los_Angeles
>NY New_York
>IL Chicago
> -- expected result
> state | name
> --+-
> TX | Dallas
> AZ | Phoenix
> IL | Chicago
> TX | Houston
> CA | San_Jose
> NY | New_York
> CA | San_Diego
> CA | Los_Angeles
> TX | San_Antonio
> {code}
> As you can see from the query result, Flink computes the top3 cities over all
> states, not for every state individually. Hence, I assume that this is a bug
> in the query optimizer or one of the rewriting rules.
> There are two valid ways to solve this issue:
> * Fixing the rewriting rules / optimizer (obviously preferred)
> * Disabling this feature and throwing an exception
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18988) Continuous query with LATERAL and LIMIT produces wrong result
[
https://issues.apache.org/jira/browse/FLINK-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-18988:
--
Description:
I was trying out the example queries provided in this blog post:
[https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
Flink supports the same and found that the queries were translated and executed
but produced the wrong result.
I used the SQL Client and Kafka (running at kafka:9092) to store the table
data. I executed the following statements:
{code:java}
-- create cities table
CREATE TABLE cities (
name STRING NOT NULL,
state STRING NOT NULL,
pop INT NOT NULL
) WITH (
'connector' = 'kafka',
'topic' = 'cities',
'properties.bootstrap.servers' = 'kafka:9092',
'properties.group.id' = 'mygroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);
-- fill cities table
INSERT INTO cities VALUES
('Los_Angeles', 'CA', 3979576),
('Phoenix', 'AZ', 1680992),
('Houston', 'TX', 2320268),
('San_Diego', 'CA', 1423851),
('San_Francisco', 'CA', 881549),
('New_York', 'NY', 8336817),
('Dallas', 'TX', 1343573),
('San_Antonio', 'TX', 1547253),
('San_Jose', 'CA', 1021795),
('Chicago', 'IL', 2695598),
('Austin', 'TX', 978908);
-- execute query
SELECT state, name
FROM
(SELECT DISTINCT state FROM cities) states,
LATERAL (
SELECT name, pop
FROM cities
WHERE state = states.state
ORDER BY pop
DESC LIMIT 3
);
-- result
state name
CA Los_Angeles
NY New_York
IL Chicago
-- expected result
state | name
--+-
TX | Dallas
AZ | Phoenix
IL | Chicago
TX | Houston
CA | San_Jose
NY | New_York
CA | San_Diego
CA | Los_Angeles
TX | San_Antonio
{code}
As you can see from the query result, Flink computes the top3 cities over all
states, not for every state individually. Hence, I assume that this is a bug in
the query optimizer or one of the rewriting rules.
There are two valid ways to solve this issue:
* Fixing the rewriting rules / optimizer (obviously preferred)
* Disabling this feature and throwing an exception
was:
I was trying out the example queries provided in this blog post:
[https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
Flink supports the same and found that the queries were translated and executed
but produced the wrong result.
I used the SQL Client and Kafka (running at kafka:9092) to store the table
data. I executed the following statements:
{code:java}
-- create cities table
CREATE TABLE cities (
name STRING NOT NULL,
state STRING NOT NULL,
pop INT NOT NULL
) WITH (
'connector' = 'kafka',
'topic' = 'cities',
'properties.bootstrap.servers' = 'kafka:9092',
'properties.group.id' = 'mygroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);
-- fill cities table
INSERT INTO cities VALUES
('Los_Angeles', 'CA', 3979576),
('Phoenix', 'AZ', 1680992),
('Houston', 'TX', 2320268),
('San_Diego', 'CA', 1423851),
('San_Francisco', 'CA', 881549),
('New_York', 'NY', 8336817),
('Dallas', 'TX', 1343573),
('San_Antonio', 'TX', 1547253),
('San_Jose', 'CA', 1021795),
('Chicago', 'IL', 2695598),
('Austin', 'TX', 978908);
-- execute query
SELECT state, name
FROM
(SELECT DISTINCT state FROM cities) states,
LATERAL (
SELECT name, pop
FROM cities
WHERE state = states.state
ORDER BY pop
DESC LIMIT 3
);
-- result
state name
CA Los_Angeles
NY New_York
IL Chicago
-- expected result
state | name
--+-
TX | Dallas
AZ | Phoenix
IL | Chicago
TX | Houston
CA | San_Jose
NY | New_York
CA | San_Diego
CA | Los_Angeles
TX | San_Antonio
{code}
As you can see from the query result, Flink computes the top3 cities over all
states, not for every state individually. Hence, I assume that this is a bug in
the query optimizer or one of the rewriting rules.
> Continuous query with LATERAL and LIMIT produces wrong result
> -
>
> Key: FLINK-18988
> URL: https://issues.apache.org/jira/browse/FLINK-18988
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: Fabian Hueske
>Priority: Critical
>
> I was trying out the example queries provided in this blog post:
> [https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
> Flink supports the same and found that the queries were translated and
> executed but produced the wrong result.
> I used the SQL Client and Kafka (running at kafka:9092) to store the table
> data. I executed the following statements:
> {code:java}
> -- create
[jira] [Created] (FLINK-18988) Continuous query with LATERAL and LIMIT produces wrong result
Fabian Hueske created FLINK-18988:
-
Summary: Continuous query with LATERAL and LIMIT produces wrong
result
Key: FLINK-18988
URL: https://issues.apache.org/jira/browse/FLINK-18988
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.11.1
Reporter: Fabian Hueske
I was trying out the example queries provided in this blog post:
[https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
Flink supports the same and found that the queries were translated and executed
but produced the wrong result.
I used the SQL Client and Kafka (running at kafka:9092) to store the table
data. I executed the following statements:
{code:java}
-- create cities table
CREATE TABLE cities (
name STRING NOT NULL,
state STRING NOT NULL,
pop INT NOT NULL
) WITH (
'connector' = 'kafka',
'topic' = 'cities',
'properties.bootstrap.servers' = 'kafka:9092',
'properties.group.id' = 'mygroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);
-- fill cities table
INSERT INTO cities VALUES
('Los_Angeles', 'CA', 3979576),
('Phoenix', 'AZ', 1680992),
('Houston', 'TX', 2320268),
('San_Diego', 'CA', 1423851),
('San_Francisco', 'CA', 881549),
('New_York', 'NY', 8336817),
('Dallas', 'TX', 1343573),
('San_Antonio', 'TX', 1547253),
('San_Jose', 'CA', 1021795),
('Chicago', 'IL', 2695598),
('Austin', 'TX', 978908);
-- execute query
SELECT state, name
FROM
(SELECT DISTINCT state FROM cities) states,
LATERAL (
SELECT name, pop
FROM cities
WHERE state = states.state
ORDER BY pop
DESC LIMIT 3
);
-- result
state name
CA Los_Angeles
NY New_York
IL Chicago
-- expected result
state | name
--+-
TX | Dallas
AZ | Phoenix
IL | Chicago
TX | Houston
CA | San_Jose
NY | New_York
CA | San_Diego
CA | Los_Angeles
TX | San_Antonio
{code}
As you can see from the query result, Flink computes the top3 cities over all
states, not for every state individually. Hence, I assume that this is a bug in
the query optimizer or one of the rewriting rules.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18704) Support DECIMAL types in datagen Table source connector
[
https://issues.apache.org/jira/browse/FLINK-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177020#comment-17177020
]
Fabian Hueske commented on FLINK-18704:
---
Thanks for picking this up [~micg]!
Since datagen is just a connector for testing and demonstration purposes, I
wouldn't worry too much about the "correctness" of the generated data.
Of course the returned values should be in the value range of the specified
BigDecimal type.
For me it would be fine if we just generate a double value that's in the value
range of the BigDecimal type and then cast it to BigDecimal.
> Support DECIMAL types in datagen Table source connector
> ---
>
> Key: FLINK-18704
> URL: https://issues.apache.org/jira/browse/FLINK-18704
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Ecosystem
>Affects Versions: 1.11.0, 1.11.1
>Reporter: Fabian Hueske
>Priority: Major
> Labels: starter
>
> It would be great if the [{{datagen}} source
> connector|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/datagen.html]
> would support {{DECIMAL}} types.
> Since data is randomly generated and FLOAT and DOUBLE are supported, we could
> implement this feature by creatig a {{BigDecimal}} from a random float.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18704) Support DECIMAL types in datagen Table source connector
Fabian Hueske created FLINK-18704:
-
Summary: Support DECIMAL types in datagen Table source connector
Key: FLINK-18704
URL: https://issues.apache.org/jira/browse/FLINK-18704
Project: Flink
Issue Type: Improvement
Components: Table SQL / Ecosystem
Affects Versions: 1.11.1, 1.11.0
Reporter: Fabian Hueske
It would be great if the [{{datagen}} source
connector|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/datagen.html]
would support {{DECIMAL}} types.
Since data is randomly generated and FLOAT and DOUBLE are supported, we could
implement this feature by creatig a {{BigDecimal}} from a random float.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18683) Support @DataTypeHint for TableFunction output types
Fabian Hueske created FLINK-18683:
-
Summary: Support @DataTypeHint for TableFunction output types
Key: FLINK-18683
URL: https://issues.apache.org/jira/browse/FLINK-18683
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Fabian Hueske
For ScalarFunctions, the return type of an eval method can be declared with a
{{@DataTypeHint}}:
{code:java}
@DataTypeHint("INT")
public Integer eval(Integer value) {
return value * 2;
}{code}
This does not work for TableFunctions because the {{@DataTypeHint}} annotation
refers to the {{void}} return type. Hence, {{TableFunction}} {{eval()}} methods
must always be annotated with the more complex {{@FunctionHint}} method.
However, I think that context, it is clear that the {{@DataTypeHint}}
annotation refers to the actual return type of the table function (the type
parameter of {{TableFunction}}).
We could consider allowing {{@DataTypeHint}} annotations also on
{{TableFunction}} classes (defining the output type of all eval methods) and
{{eval()}} methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18673) Calling ROW() in a UDF results in UnsupportedOperationException
[
https://issues.apache.org/jira/browse/FLINK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-18673:
--
Description:
Given a UDF {{func}} that accepts a {{ROW(INT, STRING)}} as parameter, it
cannot be called like this:
{code:java}
SELECT func(ROW(a, b)) FROM t{code}
while this works
{code:java}
SELECT func(r) FROM (SELECT ROW(a, b) FROM t){code}
The exception returned is:
{quote}org.apache.flink.table.api.ValidationException: SQL validation failed.
null
{quote}
with an empty {{UnsupportedOperationException}} as cause.
was:
Given a UDF {{func}} that accepts a {{ROW(INT, STRING)}} as parameter, it
cannot be called like this:
{code:java}
SELECT func(ROW(a, b)) FROM t{code}
while this works
{code:java}
SELECT func(r) FROM (SELECT ROW(a, b) FROM t){code}
The exception returned is:
{quote}
org.apache.flink.table.api.ValidationException: SQL validation failed. null
{quote}
with an empty {{UnsupportedOperationException}} as cause.
> Calling ROW() in a UDF results in UnsupportedOperationException
> ---
>
> Key: FLINK-18673
> URL: https://issues.apache.org/jira/browse/FLINK-18673
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: Fabian Hueske
>Priority: Major
>
> Given a UDF {{func}} that accepts a {{ROW(INT, STRING)}} as parameter, it
> cannot be called like this:
> {code:java}
> SELECT func(ROW(a, b)) FROM t{code}
> while this works
> {code:java}
> SELECT func(r) FROM (SELECT ROW(a, b) FROM t){code}
>
> The exception returned is:
> {quote}org.apache.flink.table.api.ValidationException: SQL validation failed.
> null
> {quote}
> with an empty {{UnsupportedOperationException}} as cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18673) Calling ROW() in a UDF results in UnsupportedOperationException
[
https://issues.apache.org/jira/browse/FLINK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-18673:
--
Affects Version/s: 1.11.1
> Calling ROW() in a UDF results in UnsupportedOperationException
> ---
>
> Key: FLINK-18673
> URL: https://issues.apache.org/jira/browse/FLINK-18673
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: Fabian Hueske
>Priority: Major
>
> Given a UDF {{func}} that accepts a {{ROW(INT, STRING)}} as parameter, it
> cannot be called like this:
> {code:java}
> SELECT func(ROW(a, b)) FROM t{code}
> while this works
>
>
> {code:java}
> SELECT func(r) FROM (SELECT ROW(a, b) FROM t){code}
>
> The exception returned is:
> {quote}
> org.apache.flink.table.api.ValidationException: SQL validation failed. null
> {quote}
> with an empty {{UnsupportedOperationException}} as cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18673) Calling ROW() in a UDF results in UnsupportedOperationException
Fabian Hueske created FLINK-18673:
-
Summary: Calling ROW() in a UDF results in
UnsupportedOperationException
Key: FLINK-18673
URL: https://issues.apache.org/jira/browse/FLINK-18673
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Fabian Hueske
Given a UDF {{func}} that accepts a {{ROW(INT, STRING)}} as parameter, it
cannot be called like this:
{code:java}
SELECT func(ROW(a, b)) FROM t{code}
while this works
{code:java}
SELECT func(r) FROM (SELECT ROW(a, b) FROM t){code}
The exception returned is:
{quote}
org.apache.flink.table.api.ValidationException: SQL validation failed. null
{quote}
with an empty {{UnsupportedOperationException}} as cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18672) Fix Scala code examples for UDF type inference annotations
Fabian Hueske created FLINK-18672:
-
Summary: Fix Scala code examples for UDF type inference annotations
Key: FLINK-18672
URL: https://issues.apache.org/jira/browse/FLINK-18672
Project: Flink
Issue Type: Bug
Components: Documentation, Table SQL / API
Reporter: Fabian Hueske
The Scala code examples for the [UDF type inference
annotations|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/functions/udfs.html#type-inference]
are not correct.
For example: the following {{FunctionHint}} annotation
{code:scala}
@FunctionHint(
input = Array(@DataTypeHint("INT"), @DataTypeHint("INT")),
output = @DataTypeHint("INT")
)
{code}
needs to be changed to
{code:scala}
@FunctionHint(
input = Array(new DataTypeHint("INT"), new DataTypeHint("INT")),
output = new DataTypeHint("INT")
)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18657) Change semantics of DESCRIBE SELECT statement
Fabian Hueske created FLINK-18657:
-
Summary: Change semantics of DESCRIBE SELECT statement
Key: FLINK-18657
URL: https://issues.apache.org/jira/browse/FLINK-18657
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Fabian Hueske
Running a {{DESCRIBE SELECT ...}} returns the same as {{EXPLAIN PLAN FOR SELECT
...}}, i.e., an unstructured, textual description of the execution plan.
I think it would be more consistent and also more useful if {{DESCRIBE SELECT
...}} would display the result schema of the {{SELECT}} query, just like
{{DESCRIBE TABLE xxx}} shows the schema of the table {{xxx}}. This would be
useful for users who need to create a sink table to write the result schema of
a SELECT query and figure out the schema of the query.
This would be a breaking change, but {{DESCRIBE SELECT ...}} seems to be an
undocumented feature (at least it is not listed on the [DESCRIBE
docs|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/describe.html])
and the original behavior would be still available via {{EXPLAIN PLAN FOR
SELECT ...}} (which is documented).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18623) CREATE TEMPORARY TABLE not documented
[
https://issues.apache.org/jira/browse/FLINK-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske closed FLINK-18623.
-
Resolution: Duplicate
Duplicate of FLINK-18624
> CREATE TEMPORARY TABLE not documented
> -
>
> Key: FLINK-18623
> URL: https://issues.apache.org/jira/browse/FLINK-18623
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
>
> The {{CREATE TEMPORARY TABLE}} syntax that was added with FLINK-15591 is not
> included in the [{{CREATE TABLE}}
> documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/create.html#create-table]
> and therefore not visible to our users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18623) CREATE TEMPORARY TABLE not documented
Fabian Hueske created FLINK-18623:
-
Summary: CREATE TEMPORARY TABLE not documented
Key: FLINK-18623
URL: https://issues.apache.org/jira/browse/FLINK-18623
Project: Flink
Issue Type: Bug
Components: Documentation, Table SQL / API
Affects Versions: 1.11.0
Reporter: Fabian Hueske
The {{CREATE TEMPORARY TABLE}} syntax that was added with FLINK-15591 is not
included in the [{{CREATE TABLE}}
documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/create.html#create-table]
and therefore not visible to our users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-9278) Allow restore savepoint with some SQL queries added/removed
[ https://issues.apache.org/jira/browse/FLINK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147286#comment-17147286 ] Fabian Hueske commented on FLINK-9278: -- I'm not aware of any updates, but I'm also not following the development of SQL features super closely anymore. > Allow restore savepoint with some SQL queries added/removed > --- > > Key: FLINK-9278 > URL: https://issues.apache.org/jira/browse/FLINK-9278 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.4.2 >Reporter: Adrian Hains >Assignee: vinoyang >Priority: Major > > We are running a Flink job that contains multiple SQL queries. This is > configured by calling sqlQuery(String) one time for each SQL query, on a > single instance of StreamTableEnvironment. The queries are simple > aggregations with a tumble window. > Currently I can configure my environment with queries Q1, Q2, and Q3, create > a savepoint, and restart the job from that savepoint if the same set of SQL > queries are used. > If I remove some queries and add some others, Q2, Q4, and Q3, I am unable to > restart the job from the same savepoint. This behavior is expected, as the > documentation clearly describes that the operator IDs are generated if they > are not explicitly defined, and they cannot be explicitly defined when using > flink SQL. > I would like to be able to specify a scoping operator id prefix when > registering a SQL query to a StreamTableEnvironment. This can then be used to > programmatically generate unique IDs for each of the operators created to > execute the SQL queries. For example, if I specify a prefix of "ID:Q2:" for > my Q2 query, and I restart the job with an identical SQL query for this > prefix, then I would be able to restore the state for this query even in the > presence of other queries being added or removed to the job graph. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-16835:
--
Description:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
[~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
comments below) and concluded with the following design:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption LOCAL_TIME_ZONE =
key("table.local-time-zone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
*Note*: The following {{TableConfig}} options are not preserved:
* {{nullCheck}}: Flink will automatically enable null checks based on the table
schema ({{NOT NULL}} property)
* {{decimalContext}}: this configuration is only used by the legacy planner
which will be removed in one of the next releases
* {{maxIdleStateRetention}}: is automatically derived as 1.5*
{{idleStateRetention}} until StateTtlConfig is fully supported (at which point
only a single parameter is required).
was:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
[~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
comments below) and concluded with the following design:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption LOCAL_TIME_ZONE =
key("table.local-time-zone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
*Note*: The following {{TableConfig}} options are not preserved:
* {{nullCheck}}: Flink will automatically enable null checks based on the table
schema ({{NOT NULL}} property)
* {{decimalContext}}: this configuration is only used by the legacy planner
which will be removed in one of the next releases
* {{
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
> [~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
> comments below) and concluded with the following design:
> {code:java}
> public static final ConfigOption IDLE_STATE_RETENTION =
> key("table.exec.state.ttl")
> .durationType()
> .defaultValue(Duration.ofMillis(0));
> public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
> key("table.generated-code.max-length")
> .intType()
> .defaultValue(64000);
> public static final ConfigOption LOCAL_TIME_ZONE
[jira] [Updated] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-16835:
--
Description:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
After a discussion of [~jark] ,[~twalthr] and [~fhueske] we concluded the
following configuration options:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption LOCAL_TIME_ZONE =
key("table.local-time-zone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
was:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
> [~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
> comments below) and concluded with the following design:
> {code:java}
> public static final ConfigOption IDLE_STATE_RETENTION =
> key("table.exec.state.ttl")
> .durationType()
> .defaultValue(Duration.ofMillis(0));
> public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
> key("table.generated-code.max-length")
> .intType()
> .defaultValue(64000);
> public static final ConfigOption LOCAL_TIME_ZONE =
> key("table.local-time-zone")
> .stringType()
> .defaultValue(ZoneId.systemDefault().toString());
> {code}
> *Note*: The following {{TableConfig}} options are not preserved:
> * {{nullCheck}}: Flink will automatically enable null checks based on the
> table schema ({{NOT NULL}} property)
> * {{decimalContext}}: this configuration is only used by the legacy planner
> which will be removed in one of the next releases
> * {{
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-16835:
--
Description:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
[~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
comments below) and concluded with the following design:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption LOCAL_TIME_ZONE =
key("table.local-time-zone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
*Note*: The following {{TableConfig}} options are not preserved:
* {{nullCheck}}: Flink will automatically enable null checks based on the table
schema ({{NOT NULL}} property)
* {{decimalContext}}: this configuration is only used by the legacy planner
which will be removed in one of the next releases
* {{
was:
In order to allow reading and writing of configuration from a file or
string-based properties. We should consider removing {{TableConfig}} and fully
rely on a Configuration-based object with {{ConfigOptions}}.
This effort was partially already started which is why
{{TableConfig.getConfiguration}} exists.
However, we should clarify if we would like to have control and traceability
over layered configurations such as {{flink-conf,yaml <
StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
{{Configuration}} class is not the right abstraction for this.
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
> [~jark], [~twalthr], and [~fhueske] discussed the configuration options (see
> comments below) and concluded with the following design:
> {code:java}
> public static final ConfigOption IDLE_STATE_RETENTION =
> key("table.exec.state.ttl")
> .durationType()
> .defaultValue(Duration.ofMillis(0));
> public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
> key("table.generated-code.max-length")
> .intType()
> .defaultValue(64000);
> public static final ConfigOption LOCAL_TIME_ZONE =
> key("table.local-time-zone")
> .stringType()
> .defaultValue(ZoneId.systemDefault().toString());
> {code}
> *Note*: The following {{TableConfig}} options are not preserved:
> * {{nullCheck}}: Flink will automatically enable null checks based on the
> table schema ({{NOT NULL}} property)
> * {{decimalContext}}: this configuration is only used by the legacy planner
> which will be removed in one of the next releases
> * {{
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142699#comment-17142699
]
Fabian Hueske commented on FLINK-16835:
---
I think we have consensus.
I'll copy the result of our discussion to the issue description, so it's easy
to find for the contributor who works on this issue.
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142693#comment-17142693
]
Fabian Hueske commented on FLINK-16835:
---
Thanks for the feedback [~twalthr].
So the options are even more condensed to:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption LOCAL_TIME_ZONE =
key("table.local-time-zone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142134#comment-17142134
]
Fabian Hueske edited comment on FLINK-16835 at 6/22/20, 2:56 PM:
-
Thanks for the feedback. Sounds good to me.
So the config options would look like this:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption DECIMAL_CONTEXT =
key("table.decimal-context")
.stringType()
.defaultValue("DECIMAL128");
public static final ConfigOption LOCAL_TIMEZONE =
key("table.local-timezone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());{code}
Any other feedback?
was (Author: fhueske):
Thanks for the feedback. Sounds good to me.
So the config options would look like this:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption DECIMAL_CONTEXT =
key("table.decimal-context")
.stringType()
.defaultValue("DECIMAL128");
public static final ConfigOption LOCAL_TIMEZONE =
key("table.local-timezone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142134#comment-17142134
]
Fabian Hueske commented on FLINK-16835:
---
Thanks for the feedback. Sounds good to me.
So the config options would look like this:
{code:java}
public static final ConfigOption IDLE_STATE_RETENTION =
key("table.exec.state.ttl")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption DECIMAL_CONTEXT =
key("table.decimal-context")
.stringType()
.defaultValue("DECIMAL128");
public static final ConfigOption LOCAL_TIMEZONE =
key("table.local-timezone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17845) Can't remove a table connector property with ALTER TABLE
[
https://issues.apache.org/jira/browse/FLINK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142092#comment-17142092
]
Fabian Hueske commented on FLINK-17845:
---
[~jark] I completely agree that the invalid configuration key should not be
ignored.
But it would be good to be able to fix the problem without dropping and
recreating the table ;)
> Can't remove a table connector property with ALTER TABLE
>
>
> Key: FLINK-17845
> URL: https://issues.apache.org/jira/browse/FLINK-17845
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Fabian Hueske
>Priority: Major
>
> It is not possible to remove an existing table property from a table.
> Looking at the [source
> code|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/sqlexec/SqlToOperationConverter.java#L295]
> this seems to be the intended semantics, but it seems counter-intuitive to
> me.
> If I create a table with the following statement:
> {code}
> CREATE TABLE `testTable` (
> id INT
> )
> WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topicX' = 'test', -- Woops, I made a typo here
> [...]
> )
> {code}
> The statement will be successfully executed. However, the table cannot be
> used due to the typo.
> Fixing the typo with the following DDL is not possible:
> {code}
> ALTER TABLE `testTable` SET (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'test', -- Fixing the typo
> )
> {code}
> because the key {{connector.topicX}} is not removed.
> Right now it seems that the only way to fix a table with an invalid key is to
> DROP and CREATE it. I think that this use case should be supported by ALTER
> TABLE.
> I would even argue that the expected behavior is that previous properties are
> removed and replaced by the new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17845) Can't remove a table connector property with ALTER TABLE
[
https://issues.apache.org/jira/browse/FLINK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142008#comment-17142008
]
Fabian Hueske commented on FLINK-17845:
---
OK, I understand the reason for the current behavior.
You're proposal to add {{ALTER TABLE table_name REPLACE ('key1' = 'value1',
...)}} makes sense to me.
Regarding your question:
{quote}{{Another question is why would a property typo make the table
unusable?}}
{quote}
It might depend on the connector, but in Flink 1.10 the Kafka connector failed
when the table properties contained a key that the connector could not handle.
Not sure if that was recently changed.
> Can't remove a table connector property with ALTER TABLE
>
>
> Key: FLINK-17845
> URL: https://issues.apache.org/jira/browse/FLINK-17845
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Fabian Hueske
>Priority: Major
>
> It is not possible to remove an existing table property from a table.
> Looking at the [source
> code|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/sqlexec/SqlToOperationConverter.java#L295]
> this seems to be the intended semantics, but it seems counter-intuitive to
> me.
> If I create a table with the following statement:
> {code}
> CREATE TABLE `testTable` (
> id INT
> )
> WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topicX' = 'test', -- Woops, I made a typo here
> [...]
> )
> {code}
> The statement will be successfully executed. However, the table cannot be
> used due to the typo.
> Fixing the typo with the following DDL is not possible:
> {code}
> ALTER TABLE `testTable` SET (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'test', -- Fixing the typo
> )
> {code}
> because the key {{connector.topicX}} is not removed.
> Right now it seems that the only way to fix a table with an invalid key is to
> DROP and CREATE it. I think that this use case should be supported by ALTER
> TABLE.
> I would even argue that the expected behavior is that previous properties are
> removed and replaced by the new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141997#comment-17141997
]
Fabian Hueske commented on FLINK-16835:
---
{quote} # Maybe we can remove the NULL_CHECK configuration. I guess it is never
used by users and we are going to refactor code generation to remove null check
when the type is NOT NULL automatically.{quote}
Sounds good to me
{quote} # I'm thinking to merge {{table.idle-state.retention.min}} and
{{table.idle-state.retention.max}} into one configuration, e.g.
{{table.exec.state.ttl}}. Sometimes this configuration is a bit confusing that
what's the max/min is used for. Besides, we are moving to use
{{StateTtlConfig}} which is more lightweight than cleanup timers and can
decouple state cleanup from operator logic.
{quote}
So we would only set the min retention time and derive the max retention time
by adding a constant (or value that's relative to the min retention time)? As
long as the switch to {{StateTtlConfig}} isn't completed yet, we would need to
set two durations.
{quote} # The value of {{MathContext.DECIMAL128.toString()}} is {{precision=34
roundingMode=HALF_EVEN}} which is rather hard to configure. Could we provide
some predefined enums, e.g. {{DECIMAL128, DECIMAL64, DECIMAL32, UNLIMITED}}. We
can support custom decimal context in the future when it is needed.
{quote}
Sounds good to me as well.
What do you think about the configuration keys?
Are they OK or should we change the prefix from {{table}} to {{table.exec}} ?
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16835) Replace TableConfig with Configuration
[
https://issues.apache.org/jira/browse/FLINK-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140575#comment-17140575
]
Fabian Hueske commented on FLINK-16835:
---
I'd like to make a proposal for the configuration options for configuring the
TableConfig.
{code:java}
public static final ConfigOption NULL_CHECK =
key("table.null-check.enabled")
.booleanType()
.defaultValue(true);
public static final ConfigOption MIN_IDLE_STATE_RETENTION =
key("table.idle-state.retention.min")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_IDLE_STATE_RETENTION =
key("table.idle-state.retention.max")
.durationType()
.defaultValue(Duration.ofMillis(0));
public static final ConfigOption MAX_LENGTH_GENERATED_CODE =
key("table.generated-code.max-length")
.intType()
.defaultValue(64000);
public static final ConfigOption DECIMAL_CONTEXT =
key("table.decimal-context")
.stringType()
.defaultValue(MathContext.DECIMAL128.toString());
public static final ConfigOption LOCAL_TIMEZONE =
key("table.local-timezone")
.stringType()
.defaultValue(ZoneId.systemDefault().toString());
{code}
The default values are taken from {{TableConfig}}. The ZoneId and MathContext
need to be parsed from the String representation in the configuration
(MathContext and ZoneId have corresponding parsing methods).
I'd like to get some feedback especially on the option keys.
CC: [~twalthr], [~jark]
> Replace TableConfig with Configuration
> --
>
> Key: FLINK-16835
> URL: https://issues.apache.org/jira/browse/FLINK-16835
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> In order to allow reading and writing of configuration from a file or
> string-based properties. We should consider removing {{TableConfig}} and
> fully rely on a Configuration-based object with {{ConfigOptions}}.
> This effort was partially already started which is why
> {{TableConfig.getConfiguration}} exists.
> However, we should clarify if we would like to have control and traceability
> over layered configurations such as {{flink-conf,yaml <
> StreamExecutionEnvironment < TableEnvironment < Query}}. Maybe the
> {{Configuration}} class is not the right abstraction for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18378) CatalogManager checks for CatalogTableImpl instead of CatalogTable
[
https://issues.apache.org/jira/browse/FLINK-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140547#comment-17140547
]
Fabian Hueske commented on FLINK-18378:
---
[~dwysakowicz] Thanks for the explanation! (y)
> CatalogManager checks for CatalogTableImpl instead of CatalogTable
> --
>
> Key: FLINK-18378
> URL: https://issues.apache.org/jira/browse/FLINK-18378
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
> Fix For: 1.11.0
>
>
> The {{CatalogManager}} checks for {{CatalogTableImpl}} instead of
> {{CatalogTable}} to decide whether to resolve the table schema. See
> https://github.com/apache/flink/blob/release-1.11/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java#L369
> Resolving the table schema adjusts the type of fields which are referenced by
> watermarks, i.e., changes their type from {{TIMESTAMP(3)}} to {{TIMESTAMP(3)
> ROWTIME}}. If table schema is not properly resolved some queries involving
> time attributes will fail during type validation.
> However, {{CatalogTableImpl}} is an internal implementation of the public
> {{CatalogTable}} interface. Hence, external {{Catalog}} implementations will
> not work with {{CatalogTableImpl}} but rather {{CatalogTable}} and hence
> might fail to work correctly with queries that involve event-time attributes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18378) CatalogManager checks for CatalogTableImpl instead of CatalogTable
[
https://issues.apache.org/jira/browse/FLINK-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140519#comment-17140519
]
Fabian Hueske commented on FLINK-18378:
---
Is it necessary to preserve the original table type?
Wouldn't it be possible to create a new {{CatalogTableImpl}} with the resolved
schema and all other fields (properties, comment, partitionKeys, ...) and
return it?
Btw. what's the purpose of the check? Is it to distinguish between tables and
views?
> CatalogManager checks for CatalogTableImpl instead of CatalogTable
> --
>
> Key: FLINK-18378
> URL: https://issues.apache.org/jira/browse/FLINK-18378
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
> Fix For: 1.11.0
>
>
> The {{CatalogManager}} checks for {{CatalogTableImpl}} instead of
> {{CatalogTable}} to decide whether to resolve the table schema. See
> https://github.com/apache/flink/blob/release-1.11/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java#L369
> Resolving the table schema adjusts the type of fields which are referenced by
> watermarks, i.e., changes their type from {{TIMESTAMP(3)}} to {{TIMESTAMP(3)
> ROWTIME}}. If table schema is not properly resolved some queries involving
> time attributes will fail during type validation.
> However, {{CatalogTableImpl}} is an internal implementation of the public
> {{CatalogTable}} interface. Hence, external {{Catalog}} implementations will
> not work with {{CatalogTableImpl}} but rather {{CatalogTable}} and hence
> might fail to work correctly with queries that involve event-time attributes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18378) CatalogManager checks for CatalogTableImpl instead of CatalogTable
Fabian Hueske created FLINK-18378:
-
Summary: CatalogManager checks for CatalogTableImpl instead of
CatalogTable
Key: FLINK-18378
URL: https://issues.apache.org/jira/browse/FLINK-18378
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: Fabian Hueske
The {{CatalogManager}} checks for {{CatalogTableImpl}} instead of
{{CatalogTable}} to decide whether to resolve the table schema. See
https://github.com/apache/flink/blob/release-1.11/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java#L369
Resolving the table schema adjusts the type of fields which are referenced by
watermarks, i.e., changes their type from {{TIMESTAMP(3)}} to {{TIMESTAMP(3)
ROWTIME}}. If table schema is not properly resolved some queries involving time
attributes will fail during type validation.
However, {{CatalogTableImpl}} is an internal implementation of the public
{{CatalogTable}} interface. Hence, external {{Catalog}} implementations will
not work with {{CatalogTableImpl}} but rather {{CatalogTable}} and hence might
fail to work correctly with queries that involve event-time attributes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-18298) Rename TableResult headers of SHOW statements
[
https://issues.apache.org/jira/browse/FLINK-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske reassigned FLINK-18298:
-
Assignee: godfrey he
> Rename TableResult headers of SHOW statements
> -
>
> Key: FLINK-18298
> URL: https://issues.apache.org/jira/browse/FLINK-18298
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Assignee: godfrey he
>Priority: Major
>
> The SHOW TABLES and SHOW FUNCTIONS commands are listing all tables and
> functions of the currently selected database.
> With FLIP-84, the result is passed back as a TableResult object, that
> includes the schema of the result as a TableSchema.
> The column name for the result of SHOW TABLES and SHOW FUNCTION is "result":
>
> {code:java}
> SHOW TABLES;
> result
> myTable1
> myTable2
> {code}
>
> I think this name is not very descriptive and too generic.
> IMO it would be nice to change it to "table name" and "function name",
> respectively.
> {code:java}
> SHOW TABLES;
> table names
> myTable1
> myTable2{code}
> Would be nice to get this little improvement in before the 1.11 release.
> cc [~godfreyhe], [~twalthr]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18298) Rename TableResult headers of SHOW statements
[
https://issues.apache.org/jira/browse/FLINK-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135703#comment-17135703
]
Fabian Hueske commented on FLINK-18298:
---
Thanks for the swift response [~godfreyhe]!
The suggested names look good :)
> Rename TableResult headers of SHOW statements
> -
>
> Key: FLINK-18298
> URL: https://issues.apache.org/jira/browse/FLINK-18298
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
>
> The SHOW TABLES and SHOW FUNCTIONS commands are listing all tables and
> functions of the currently selected database.
> With FLIP-84, the result is passed back as a TableResult object, that
> includes the schema of the result as a TableSchema.
> The column name for the result of SHOW TABLES and SHOW FUNCTION is "result":
>
> {code:java}
> SHOW TABLES;
> result
> myTable1
> myTable2
> {code}
>
> I think this name is not very descriptive and too generic.
> IMO it would be nice to change it to "table name" and "function name",
> respectively.
> {code:java}
> SHOW TABLES;
> table names
> myTable1
> myTable2{code}
> Would be nice to get this little improvement in before the 1.11 release.
> cc [~godfreyhe], [~twalthr]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18298) Rename TableResult headers of SHOW statements
Fabian Hueske created FLINK-18298:
-
Summary: Rename TableResult headers of SHOW statements
Key: FLINK-18298
URL: https://issues.apache.org/jira/browse/FLINK-18298
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Fabian Hueske
The SHOW TABLES and SHOW FUNCTIONS commands are listing all tables and
functions of the currently selected database.
With FLIP-84, the result is passed back as a TableResult object, that includes
the schema of the result as a TableSchema.
The column name for the result of SHOW TABLES and SHOW FUNCTION is "result":
{code:java}
SHOW TABLES;
result
myTable1
myTable2
{code}
I think this name is not very descriptive and too generic.
IMO it would be nice to change it to "table name" and "function name",
respectively.
{code:java}
SHOW TABLES;
table names
myTable1
myTable2{code}
Would be nice to get this little improvement in before the 1.11 release.
cc [~godfreyhe], [~twalthr]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18209) Replace "charged" words in the Flink codebase.
[ https://issues.apache.org/jira/browse/FLINK-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133274#comment-17133274 ] Fabian Hueske commented on FLINK-18209: --- +1 > Replace "charged" words in the Flink codebase. > -- > > Key: FLINK-18209 > URL: https://issues.apache.org/jira/browse/FLINK-18209 > Project: Flink > Issue Type: Task >Reporter: Marta Paes Moreira >Priority: Trivial > > The Flink codebase still makes use of "charged" words that are considered > outdated and have a negative connotation. In particular, this is the number > of occurrences of such words: > "Master": 1366 > "Slave": 229 > > "Whitelist": 23 > > "Blacklist": 28 > > I'd like to propose that we rethink the use of these words [1] and consider > existing alternatives that other open source projects are also adopting. > Replacing all occurrences is non-trivial in many cases and may involve > breaking-changes, though, so one idea would be to break this effort down into > different phases that can be tackled over time, depending on the impact of > the required changes. > [1] [https://lethargy.org/~jesus/writes/a-guide-to-nomenclature-selection/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17845) Can't remove a table connector property with ALTER TABLE
Fabian Hueske created FLINK-17845:
-
Summary: Can't remove a table connector property with ALTER TABLE
Key: FLINK-17845
URL: https://issues.apache.org/jira/browse/FLINK-17845
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Fabian Hueske
It is not possible to remove an existing table property from a table.
Looking at the [source
code|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/sqlexec/SqlToOperationConverter.java#L295]
this seems to be the intended semantics, but it seems counter-intuitive to me.
If I create a table with the following statement:
{code}
CREATE TABLE `testTable` (
id INT
)
WITH (
'connector.type' = 'kafka',
'connector.version' = 'universal',
'connector.topicX' = 'test', -- Woops, I made a typo here
[...]
)
{code}
The statement will be successfully executed. However, the table cannot be used
due to the typo.
Fixing the typo with the following DDL is not possible:
{code}
ALTER TABLE `testTable` SET (
'connector.type' = 'kafka',
'connector.version' = 'universal',
'connector.topic' = 'test', -- Fixing the typo
)
{code}
because the key {{connector.topicX}} is not removed.
Right now it seems that the only way to fix a table with an invalid key is to
DROP and CREATE it. I think that this use case should be supported by ALTER
TABLE.
I would even argue that the expected behavior is that previous properties are
removed and replaced by the new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-9528) Incorrect results: Filter does not treat Upsert messages correctly.
[
https://issues.apache.org/jira/browse/FLINK-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078248#comment-17078248
]
Fabian Hueske commented on FLINK-9528:
--
Thanks for the update [~hequn8128]!
I'll reopen the ticket then.
> Incorrect results: Filter does not treat Upsert messages correctly.
> ---
>
> Key: FLINK-9528
> URL: https://issues.apache.org/jira/browse/FLINK-9528
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.3.3, 1.4.2, 1.5.0
>Reporter: Fabian Hueske
>Priority: Critical
>
> Currently, Filters (i.e., Calcs with predicates) do not distinguish between
> retraction and upsert mode. A Calc looks at record (regardless of its update
> semantics) and either discard it (predicate evaluates to false) or pass it on
> (predicate evaluates to true).
> This works fine for messages with retraction semantics but is not correct for
> upsert messages.
> The following test case (can be pasted into {{TableSinkITCase}}) shows the
> problem:
> {code:java}
> @Test
> def testUpsertsWithFilter(): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.getConfig.enableObjectReuse()
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val t = StreamTestData.get3TupleDataStream(env)
> .assignAscendingTimestamps(_._1.toLong)
> .toTable(tEnv, 'id, 'num, 'text)
> t.select('text.charLength() as 'len)
> .groupBy('len)
> .select('len, 'len.count as 'cnt)
> // .where('cnt < 7)
> .writeToSink(new TestUpsertSink(Array("len"), false))
> env.execute()
> val results = RowCollector.getAndClearValues
> val retracted = RowCollector.upsertResults(results, Array(0)).sorted
> val expectedWithoutFilter = List(
> "2,1", "5,1", "9,9", "10,7", "11,1", "14,1", "25,1").sorted
> val expectedWithFilter = List(
> "2,1", "5,1", "11,1", "14,1", "25,1").sorted
> assertEquals(expectedWithoutFilter, retracted)
> // assertEquals(expectedWithFilter, retracted)
> }
> {code}
> When we add a filter on the aggregation result, we would expect that all rows
> that do not fulfill the condition are removed from the result. However, the
> filter only removes the upsert message such that the previous version remains
> in the result.
> One solution could be to make a filter aware of the update semantics (retract
> or upsert) and convert the upsert message into a delete message if the
> predicate evaluates to false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (FLINK-9528) Incorrect results: Filter does not treat Upsert messages correctly.
[
https://issues.apache.org/jira/browse/FLINK-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske reopened FLINK-9528:
--
> Incorrect results: Filter does not treat Upsert messages correctly.
> ---
>
> Key: FLINK-9528
> URL: https://issues.apache.org/jira/browse/FLINK-9528
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.3.3, 1.4.2, 1.5.0
>Reporter: Fabian Hueske
>Priority: Critical
>
> Currently, Filters (i.e., Calcs with predicates) do not distinguish between
> retraction and upsert mode. A Calc looks at record (regardless of its update
> semantics) and either discard it (predicate evaluates to false) or pass it on
> (predicate evaluates to true).
> This works fine for messages with retraction semantics but is not correct for
> upsert messages.
> The following test case (can be pasted into {{TableSinkITCase}}) shows the
> problem:
> {code:java}
> @Test
> def testUpsertsWithFilter(): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.getConfig.enableObjectReuse()
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val t = StreamTestData.get3TupleDataStream(env)
> .assignAscendingTimestamps(_._1.toLong)
> .toTable(tEnv, 'id, 'num, 'text)
> t.select('text.charLength() as 'len)
> .groupBy('len)
> .select('len, 'len.count as 'cnt)
> // .where('cnt < 7)
> .writeToSink(new TestUpsertSink(Array("len"), false))
> env.execute()
> val results = RowCollector.getAndClearValues
> val retracted = RowCollector.upsertResults(results, Array(0)).sorted
> val expectedWithoutFilter = List(
> "2,1", "5,1", "9,9", "10,7", "11,1", "14,1", "25,1").sorted
> val expectedWithFilter = List(
> "2,1", "5,1", "11,1", "14,1", "25,1").sorted
> assertEquals(expectedWithoutFilter, retracted)
> // assertEquals(expectedWithFilter, retracted)
> }
> {code}
> When we add a filter on the aggregation result, we would expect that all rows
> that do not fulfill the condition are removed from the result. However, the
> filter only removes the upsert message such that the previous version remains
> in the result.
> One solution could be to make a filter aware of the update semantics (retract
> or upsert) and convert the upsert message into a delete message if the
> predicate evaluates to false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-9528) Incorrect results: Filter does not treat Upsert messages correctly.
[
https://issues.apache.org/jira/browse/FLINK-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078225#comment-17078225
]
Fabian Hueske commented on FLINK-9528:
--
I agree with [~twalthr].
I don't think we should close bug reports unless we know that the bug was fixed
or is not relevant anymore (because a component was completely removed or ...).
> Incorrect results: Filter does not treat Upsert messages correctly.
> ---
>
> Key: FLINK-9528
> URL: https://issues.apache.org/jira/browse/FLINK-9528
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.3.3, 1.4.2, 1.5.0
>Reporter: Fabian Hueske
>Assignee: Hequn Cheng
>Priority: Critical
>
> Currently, Filters (i.e., Calcs with predicates) do not distinguish between
> retraction and upsert mode. A Calc looks at record (regardless of its update
> semantics) and either discard it (predicate evaluates to false) or pass it on
> (predicate evaluates to true).
> This works fine for messages with retraction semantics but is not correct for
> upsert messages.
> The following test case (can be pasted into {{TableSinkITCase}}) shows the
> problem:
> {code:java}
> @Test
> def testUpsertsWithFilter(): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.getConfig.enableObjectReuse()
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val t = StreamTestData.get3TupleDataStream(env)
> .assignAscendingTimestamps(_._1.toLong)
> .toTable(tEnv, 'id, 'num, 'text)
> t.select('text.charLength() as 'len)
> .groupBy('len)
> .select('len, 'len.count as 'cnt)
> // .where('cnt < 7)
> .writeToSink(new TestUpsertSink(Array("len"), false))
> env.execute()
> val results = RowCollector.getAndClearValues
> val retracted = RowCollector.upsertResults(results, Array(0)).sorted
> val expectedWithoutFilter = List(
> "2,1", "5,1", "9,9", "10,7", "11,1", "14,1", "25,1").sorted
> val expectedWithFilter = List(
> "2,1", "5,1", "11,1", "14,1", "25,1").sorted
> assertEquals(expectedWithoutFilter, retracted)
> // assertEquals(expectedWithFilter, retracted)
> }
> {code}
> When we add a filter on the aggregation result, we would expect that all rows
> that do not fulfill the condition are removed from the result. However, the
> filter only removes the upsert message such that the previous version remains
> in the result.
> One solution could be to make a filter aware of the update semantics (retract
> or upsert) and convert the upsert message into a delete message if the
> predicate evaluates to false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16716) Update Roadmap after Flink 1.10 release
Fabian Hueske created FLINK-16716: - Summary: Update Roadmap after Flink 1.10 release Key: FLINK-16716 URL: https://issues.apache.org/jira/browse/FLINK-16716 Project: Flink Issue Type: Bug Components: Project Website Reporter: Fabian Hueske The roadmap on the Flink website needs to be updated to reflect the new features of Flink 1.10 and the planned features and improvements of future releases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16540) Fully specify bugfix version of Docker containers in Flink Playground docker-compose.yaml files
[ https://issues.apache.org/jira/browse/FLINK-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-16540. - Fix Version/s: 1.9.0 1.10.0 1.11.0 Resolution: Fixed > Fully specify bugfix version of Docker containers in Flink Playground > docker-compose.yaml files > --- > > Key: FLINK-16540 > URL: https://issues.apache.org/jira/browse/FLINK-16540 > Project: Flink > Issue Type: Improvement > Components: Examples >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0, 1.10.0, 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > I recently noticed that we do not guarantee client-jobmanager compatibility > among different bugfix versions of the same minor version (see > https://github.com/ververica/sql-training/issues/8 for details). > In this case, a job submitted via a Flink 1.9.0 client could not be executed > on a Flink 1.9.2 cluster. > For the playgrounds this can easily happen, because we build a custom Docker > image (with a fixed bugfix version) and load the latest Flink images for the > latest bugfix version of the same minor version. > We can avoid such problems by fixing the bugfix version of the Flink images > that we run in the playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-1640) FileOutputFormat writes to wrong path if path ends with '/'
[
https://issues.apache.org/jira/browse/FLINK-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057003#comment-17057003
]
Fabian Hueske commented on FLINK-1640:
--
OK, sounds good to me.
> FileOutputFormat writes to wrong path if path ends with '/'
> ---
>
> Key: FLINK-1640
> URL: https://issues.apache.org/jira/browse/FLINK-1640
> Project: Flink
> Issue Type: Bug
> Components: API / Scala
>Affects Versions: 0.9, 0.8.1
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Major
> Fix For: 0.9, 0.8.2
>
>
> The FileOutputFormat duplicates the last directory of a path, if the path
> ends with a slash '/'.
> For example, if the output path is specified as {{/home/myuser/outputPath/}}
> the output is written to {{/home/myuser/outputPath/outputPath/}}.
> This bug was introduced by commit 8fc04e4da8a36866e10564205c3f900894f4f6e0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16540) Fully specify bugfix version of Docker containers in Flink Playground docker-compose.yaml files
Fabian Hueske created FLINK-16540: - Summary: Fully specify bugfix version of Docker containers in Flink Playground docker-compose.yaml files Key: FLINK-16540 URL: https://issues.apache.org/jira/browse/FLINK-16540 Project: Flink Issue Type: Improvement Reporter: Fabian Hueske Assignee: Fabian Hueske I recently noticed that we do not guarantee client-jobmanager compatibility among different bugfix versions of the same minor version (see https://github.com/ververica/sql-training/issues/8 for details). In this case, a job submitted via a Flink 1.9.0 client could not be executed on a Flink 1.9.2 cluster. For the playgrounds this can easily happen, because we build a custom Docker image (with a fixed bugfix version) and load the latest Flink images for the latest bugfix version of the same minor version. We can avoid such problems by fixing the bugfix version of the Flink images that we run in the playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16148) Update Operations Playground to Flink 1.10.0
[ https://issues.apache.org/jira/browse/FLINK-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-16148. - Resolution: Fixed Done with aca293d8b20874555c9491c593f7c3991f670ad1 > Update Operations Playground to Flink 1.10.0 > > > Key: FLINK-16148 > URL: https://issues.apache.org/jira/browse/FLINK-16148 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.10.0 >Reporter: Oleg Bonar >Assignee: David Anderson >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Update the operations playground to Flink 1.10.0 > This includes: > * Updating the flink-playgrounds repository > * Updating the "Getting Started/Docker Playgrounds" section in the > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-1640) FileOutputFormat writes to wrong path if path ends with '/'
[
https://issues.apache.org/jira/browse/FLINK-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056776#comment-17056776
]
Fabian Hueske commented on FLINK-1640:
--
Oh, that's a fairly old issue ;-)
I guess I removed whitespaces to unify the paths.
I don't think that tailing whitespaces should be a problem.
Maybe you can just change this logic to remove leading whitespaces?
> FileOutputFormat writes to wrong path if path ends with '/'
> ---
>
> Key: FLINK-1640
> URL: https://issues.apache.org/jira/browse/FLINK-1640
> Project: Flink
> Issue Type: Bug
> Components: API / Scala
>Affects Versions: 0.9, 0.8.1
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Major
> Fix For: 0.9, 0.8.2
>
>
> The FileOutputFormat duplicates the last directory of a path, if the path
> ends with a slash '/'.
> For example, if the output path is specified as {{/home/myuser/outputPath/}}
> the output is written to {{/home/myuser/outputPath/outputPath/}}.
> This bug was introduced by commit 8fc04e4da8a36866e10564205c3f900894f4f6e0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16309) ElasticSearch 7 connector is missing in SQL connector list
Fabian Hueske created FLINK-16309: - Summary: ElasticSearch 7 connector is missing in SQL connector list Key: FLINK-16309 URL: https://issues.apache.org/jira/browse/FLINK-16309 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Fabian Hueske The ES7 connector is not listed on https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16308) SQL connector download links are broken
[ https://issues.apache.org/jira/browse/FLINK-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-16308: - Assignee: Fabian Hueske > SQL connector download links are broken > --- > > Key: FLINK-16308 > URL: https://issues.apache.org/jira/browse/FLINK-16308 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > The download links for the SQL connectors on > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html > are broken because central.maven.org is down. > The URLs should be updated to > https://repo.maven.apache.org/maven2/org/apache/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16308) SQL connector download links are broken
Fabian Hueske created FLINK-16308: - Summary: SQL connector download links are broken Key: FLINK-16308 URL: https://issues.apache.org/jira/browse/FLINK-16308 Project: Flink Issue Type: Bug Components: Documentation Reporter: Fabian Hueske The download links for the SQL connectors on https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html are broken because central.maven.org is down. The URLs should be updated to https://repo.maven.apache.org/maven2/org/apache/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16067) Flink's CalciteParser swallows error position information
Fabian Hueske created FLINK-16067:
-
Summary: Flink's CalciteParser swallows error position information
Key: FLINK-16067
URL: https://issues.apache.org/jira/browse/FLINK-16067
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: Fabian Hueske
The parser used to parse SQL queries submitted through
{{TableEnvironmentImpl.sqlUpdate}} does not add the original exception from
Calcite as a cause to Flink's
{{org.apache.flink.table.api.SqlParserException}}.
However, Calcite's exception contains the position in the SQL query where the
parser failed.
This info would help users to fix their queries.
This used to work with Flink 1.9.x.
CC [~dwysakowicz]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15826) Add renameFunction() to Catalog
[
https://issues.apache.org/jira/browse/FLINK-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027284#comment-17027284
]
Fabian Hueske commented on FLINK-15826:
---
What do you think [~phoenixjiangnan]?
> Add renameFunction() to Catalog
> ---
>
> Key: FLINK-15826
> URL: https://issues.apache.org/jira/browse/FLINK-15826
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
>
> The {{Catalog}} interface lacks a method to rename a function.
> It is possible to change all properties (via {{alterFunction()}}) but it is
> not possible to rename a database.
> A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15825) Add renameDatabase() to Catalog
[
https://issues.apache.org/jira/browse/FLINK-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027283#comment-17027283
]
Fabian Hueske commented on FLINK-15825:
---
What do you think [~phoenixjiangnan]?
> Add renameDatabase() to Catalog
> ---
>
> Key: FLINK-15825
> URL: https://issues.apache.org/jira/browse/FLINK-15825
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Priority: Major
>
> The {{Catalog}} interface lacks a method to rename a database.
> It is possible to change all properties (via {{alterDatabase()}}) but it is
> not possible to rename a database.
> A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15826) Add renameFunction() to Catalog
Fabian Hueske created FLINK-15826:
-
Summary: Add renameFunction() to Catalog
Key: FLINK-15826
URL: https://issues.apache.org/jira/browse/FLINK-15826
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Fabian Hueske
The {{Catalog}} interface lacks a method to rename a function.
It is possible to change all properties (via {{alterFunction()}}) but it is not
possible to rename a database.
A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15825) Add renameDatabase() to Catalog
Fabian Hueske created FLINK-15825:
-
Summary: Add renameDatabase() to Catalog
Key: FLINK-15825
URL: https://issues.apache.org/jira/browse/FLINK-15825
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Fabian Hueske
The {{Catalog}} interface lacks a method to rename a database.
It is possible to change all properties (via {{alterDatabase()}}) but it is not
possible to rename a database.
A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15644) Add support for SQL query validation
[
https://issues.apache.org/jira/browse/FLINK-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019427#comment-17019427
]
Fabian Hueske commented on FLINK-15644:
---
Hi [~zjffdu] , I didn't consider it initially, but I think it shouldn't be too
hard.
Syntactic and semantic checks should be no problem.
The more extensive checks of DDL for Flink support might be a problem if it
would require that the connector dependencies are on the classpath to validate
that all required connector properties are set.
Not sure if we should require that.
> Add support for SQL query validation
> -
>
> Key: FLINK-15644
> URL: https://issues.apache.org/jira/browse/FLINK-15644
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Fabian Hueske
>Priority: Major
>
> It would be good if the {{TableEnvironment}} would offer methods to check the
> validity of SQL queries. Such a method could be used by services (CLI query
> shells, notebooks, SQL UIs) that are backed by Flink and execute their
> queries on Flink.
> Validation should be available in two levels:
> # Validation of syntax and semantics: This includes parsing the query,
> checking the catalog for dbs, tables, fields, type checks for expressions and
> functions, etc. This will check if the query is a valid SQL query.
> # Validation that query is supported: Checks if Flink can execute the given
> query. Some syntactically and semantically valid SQL queries are not
> supported, esp. in a streaming context. This requires running the optimizer.
> If the optimizer generates an execution plan, the query can be executed. This
> check includes the first step and is more expensive.
> The reason for this separation is that the first check can be done much fast
> as it does not involve calling the optimizer. Hence, it would be suitable for
> fast checks in an interactive query editor. The second check might take more
> time (depending on the complexity of the query) and might not be suitable for
> rapid checks but only on explicit user request.
> Requirements:
> * validation does not modify the state of the {{TableEnvironment}}, i.e. it
> does not add plan operators
> * validation does not require connector dependencies
> * validation can identify the update mode of a continuous query result
> (append-only, upsert, retraction).
> Out of scope for this issue:
> * better error messages for unsupported features as suggested by FLINK-7217
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15644) Add support for SQL query validation
Fabian Hueske created FLINK-15644:
-
Summary: Add support for SQL query validation
Key: FLINK-15644
URL: https://issues.apache.org/jira/browse/FLINK-15644
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Reporter: Fabian Hueske
It would be good if the {{TableEnvironment}} would offer methods to check the
validity of SQL queries. Such a method could be used by services (CLI query
shells, notebooks, SQL UIs) that are backed by Flink and execute their queries
on Flink.
Validation should be available in two levels:
# Validation of syntax and semantics: This includes parsing the query,
checking the catalog for dbs, tables, fields, type checks for expressions and
functions, etc. This will check if the query is a valid SQL query.
# Validation that query is supported: Checks if Flink can execute the given
query. Some syntactically and semantically valid SQL queries are not supported,
esp. in a streaming context. This requires running the optimizer. If the
optimizer generates an execution plan, the query can be executed. This check
includes the first step and is more expensive.
The reason for this separation is that the first check can be done much fast as
it does not involve calling the optimizer. Hence, it would be suitable for fast
checks in an interactive query editor. The second check might take more time
(depending on the complexity of the query) and might not be suitable for rapid
checks but only on explicit user request.
Requirements:
* validation does not modify the state of the {{TableEnvironment}}, i.e. it
does not add plan operators
* validation does not require connector dependencies
* validation can identify the update mode of a continuous query result
(append-only, upsert, retraction).
Out of scope for this issue:
* better error messages for unsupported features as suggested by FLINK-7217
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15419) Validate SQL syntax not need to depend on connector jar
[
https://issues.apache.org/jira/browse/FLINK-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-15419:
--
Summary: Validate SQL syntax not need to depend on connector jar (was:
Add support for SQL query validation)
> Validate SQL syntax not need to depend on connector jar
>
>
> Key: FLINK-15419
> URL: https://issues.apache.org/jira/browse/FLINK-15419
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Kaibo Zhou
>Priority: Major
> Fix For: 1.11.0
>
>
> As a platform user, I want to integrate Flink SQL in my platform.
> The users will register Source/Sink Tables and Functions to catalog service
> through UI, and write SQL scripts on Web SQLEditor. I want to validate the
> SQL syntax and validate that all catalog objects exist (table, fields, UDFs).
> After some investigation, I decided to use the `tEnv.sqlUpdate/sqlQuery` API
> to do this.`SqlParser` and`FlinkSqlParserImpl` is not a good choice, as it
> will not read the catalog.
> The users have registered *Kafka* source/sink table in the catalog, so the
> validation logic will be:
> {code:java}
> TableEnvironment tableEnv =
> tEnv.registerCatalog(CATALOG_NAME, catalog);
> tEnv.useCatalog(CATALOG_NAME);
> tEnv.useDatabase(DB_NAME);
> tEnv.sqlUpdate("INSERT INTO sinkTable SELECT f1,f2 FROM sourceTable");
> or
> tEnv.sqlQuery("SELECT * FROM tableName")
> {code}
> It will through exception on Flink 1.9.0 because I do not have
> `flink-connector-kafka_2.11-1.9.0.jar` in my classpath.
> {code:java}
> org.apache.flink.table.api.ValidationException: SQL validation failed.
> findAndCreateTableSource
> failed.org.apache.flink.table.api.ValidationException: SQL validation failed.
> findAndCreateTableSource failed. at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:125)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:82)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.parse(PlannerBase.scala:132)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:335)
> The following factories have been considered:
> org.apache.flink.formats.json.JsonRowFormatFactory
> org.apache.flink.table.planner.delegation.BlinkPlannerFactory
> org.apache.flink.table.planner.delegation.BlinkExecutorFactory
> org.apache.flink.table.catalog.GenericInMemoryCatalogFactory
> org.apache.flink.table.sources.CsvBatchTableSourceFactory
> org.apache.flink.table.sources.CsvAppendTableSourceFactory
> org.apache.flink.table.sinks.CsvBatchTableSinkFactory
> org.apache.flink.table.sinks.CsvAppendTableSinkFactory
> at
> org.apache.flink.table.factories.TableFactoryService.filterByContext(TableFactoryService.java:283)
> at
> org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:191)
> at
> org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:144)
> at
> org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:97)
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:64)
> {code}
> For a platform provider, the user's SQL may depend on *ANY* connector or even
> a custom connector. It is complicated to do dynamic loading connector jar
> after parser the connector type in SQL. And this requires the users must
> upload their custom connector jar before doing a syntax check.
> I hope that Flink can provide a friendly way to verify the syntax of SQL
> whose tables/functions are already registered in the catalog, *NOT* need to
> depend on the jar of the connector. This makes it easier for SQL to be
> integrated by external platforms.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15419) Add support for SQL query validation
[
https://issues.apache.org/jira/browse/FLINK-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-15419:
--
Summary: Add support for SQL query validation (was: Validate SQL syntax
not need to depend on connector jar)
> Add support for SQL query validation
>
>
> Key: FLINK-15419
> URL: https://issues.apache.org/jira/browse/FLINK-15419
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Kaibo Zhou
>Priority: Major
> Fix For: 1.11.0
>
>
> As a platform user, I want to integrate Flink SQL in my platform.
> The users will register Source/Sink Tables and Functions to catalog service
> through UI, and write SQL scripts on Web SQLEditor. I want to validate the
> SQL syntax and validate that all catalog objects exist (table, fields, UDFs).
> After some investigation, I decided to use the `tEnv.sqlUpdate/sqlQuery` API
> to do this.`SqlParser` and`FlinkSqlParserImpl` is not a good choice, as it
> will not read the catalog.
> The users have registered *Kafka* source/sink table in the catalog, so the
> validation logic will be:
> {code:java}
> TableEnvironment tableEnv =
> tEnv.registerCatalog(CATALOG_NAME, catalog);
> tEnv.useCatalog(CATALOG_NAME);
> tEnv.useDatabase(DB_NAME);
> tEnv.sqlUpdate("INSERT INTO sinkTable SELECT f1,f2 FROM sourceTable");
> or
> tEnv.sqlQuery("SELECT * FROM tableName")
> {code}
> It will through exception on Flink 1.9.0 because I do not have
> `flink-connector-kafka_2.11-1.9.0.jar` in my classpath.
> {code:java}
> org.apache.flink.table.api.ValidationException: SQL validation failed.
> findAndCreateTableSource
> failed.org.apache.flink.table.api.ValidationException: SQL validation failed.
> findAndCreateTableSource failed. at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:125)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:82)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.parse(PlannerBase.scala:132)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:335)
> The following factories have been considered:
> org.apache.flink.formats.json.JsonRowFormatFactory
> org.apache.flink.table.planner.delegation.BlinkPlannerFactory
> org.apache.flink.table.planner.delegation.BlinkExecutorFactory
> org.apache.flink.table.catalog.GenericInMemoryCatalogFactory
> org.apache.flink.table.sources.CsvBatchTableSourceFactory
> org.apache.flink.table.sources.CsvAppendTableSourceFactory
> org.apache.flink.table.sinks.CsvBatchTableSinkFactory
> org.apache.flink.table.sinks.CsvAppendTableSinkFactory
> at
> org.apache.flink.table.factories.TableFactoryService.filterByContext(TableFactoryService.java:283)
> at
> org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:191)
> at
> org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:144)
> at
> org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:97)
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:64)
> {code}
> For a platform provider, the user's SQL may depend on *ANY* connector or even
> a custom connector. It is complicated to do dynamic loading connector jar
> after parser the connector type in SQL. And this requires the users must
> upload their custom connector jar before doing a syntax check.
> I hope that Flink can provide a friendly way to verify the syntax of SQL
> whose tables/functions are already registered in the catalog, *NOT* need to
> depend on the jar of the connector. This makes it easier for SQL to be
> integrated by external platforms.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012807#comment-17012807 ] Fabian Hueske commented on FLINK-14684: --- Thanks for picking this issue up [~Jiawei Wu]! > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-14684: - Assignee: Jiawei Wu > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14213) Link from Flink website to Getting Started Overview page
[ https://issues.apache.org/jira/browse/FLINK-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-14213. - Resolution: Done Done with d23026ac16470157ad8b50e1d753f1471dae593e > Link from Flink website to Getting Started Overview page > > > Key: FLINK-14213 > URL: https://issues.apache.org/jira/browse/FLINK-14213 > Project: Flink > Issue Type: Sub-task > Components: Project Website >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The "Tutorials" link on the Flink website currently links to the "Local Setup > Tutorial" of the documentation. > We should replace that link with a "Getting Started" link that points to the > Getting Started Overview page (which lists code walkthroughs and playgrounds). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14313) Add Gojek to Chinese Powered By page
Fabian Hueske created FLINK-14313: - Summary: Add Gojek to Chinese Powered By page Key: FLINK-14313 URL: https://issues.apache.org/jira/browse/FLINK-14313 Project: Flink Issue Type: Task Components: chinese-translation, Project Website Reporter: Fabian Hueske Add Gojek to Chinese Powered By Flink page. The relevant commit is: 7fc857030998ea8ce6366bfec63850e08e24c563 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13002) Expand Concept -> Glossary Section
[ https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941755#comment-16941755 ] Fabian Hueske commented on FLINK-13002: --- * Data Stream * Data Set are both used to define the several other terms but are not defined themselves. > Expand Concept -> Glossary Section > -- > > Key: FLINK-13002 > URL: https://issues.apache.org/jira/browse/FLINK-13002 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Konstantin Knauf >Priority: Major > > We use this ticket to collect terms, we would like to add to the Glossary in > the future: > * Snapshot > * Checkpoint > * Savepoint > * Parallelism > * Backpressure > * TaskSlot -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14266) Introduce RowCsvInputFormat to new CSV module
[ https://issues.apache.org/jira/browse/FLINK-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940767#comment-16940767 ] Fabian Hueske commented on FLINK-14266: --- This is a lot more tricky than it seems. The difference between reading CSV records from a file and a Kafka topic is that a Kafka topic provides already individual records that just need to be parsed. When reading from a file, you need to split the records first. The standard record delimiter is a new line, however, new lines can also be included in escaped text fields. Something that makes it even more complicated is that splitting large files into multiple splits cannot be easily done, because you do not know whether the next new-line character is a record delimiter or contained in a string field. > Introduce RowCsvInputFormat to new CSV module > - > > Key: FLINK-14266 > URL: https://issues.apache.org/jira/browse/FLINK-14266 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > Now, we have an old CSV, but that is not standard CSV support. we should > support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14227) Add Razorpay to Chinese Powered By page
Fabian Hueske created FLINK-14227: - Summary: Add Razorpay to Chinese Powered By page Key: FLINK-14227 URL: https://issues.apache.org/jira/browse/FLINK-14227 Project: Flink Issue Type: Task Components: chinese-translation, Project Website Reporter: Fabian Hueske Razorpay was added to the English Powered By page with commit: [87a034140e97be42616e1a3dbe58e4f7a014e560|https://github.com/apache/flink-web/commit/87a034140e97be42616e1a3dbe58e4f7a014e560]. It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14215) Add Docs for TM and JM Environment Variable Setting
[
https://issues.apache.org/jira/browse/FLINK-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske reassigned FLINK-14215:
-
Assignee: Zhenqiu Huang
> Add Docs for TM and JM Environment Variable Setting
> ---
>
> Key: FLINK-14215
> URL: https://issues.apache.org/jira/browse/FLINK-14215
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.9.0
>
>
> Add description for
> /**
>* Prefix for passing custom environment variables to Flink's master
> process.
>* For example for passing LD_LIBRARY_PATH as an env variable to the
> AppMaster, set:
>* containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"
>* in the flink-conf.yaml.
>*/
> public static final String CONTAINERIZED_MASTER_ENV_PREFIX =
> "containerized.master.env.";
> /**
>* Similar to the {@see CONTAINERIZED_MASTER_ENV_PREFIX}, this
> configuration prefix allows
>* setting custom environment variables for the workers (TaskManagers).
>*/
> public static final String CONTAINERIZED_TASK_MANAGER_ENV_PREFIX =
> "containerized.taskmanager.env.";
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14213) Link from Flink website to Getting Started Overview page
Fabian Hueske created FLINK-14213: - Summary: Link from Flink website to Getting Started Overview page Key: FLINK-14213 URL: https://issues.apache.org/jira/browse/FLINK-14213 Project: Flink Issue Type: Sub-task Components: Project Website Reporter: Fabian Hueske Assignee: Fabian Hueske The "Tutorials" link on the Flink website currently links to the "Local Setup Tutorial" of the documentation. We should replace that link with a "Getting Started" link that points to the Getting Started Overview page (which lists code walkthroughs and playgrounds). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-14160) Extend Operations Playground with --backpressure option
[ https://issues.apache.org/jira/browse/FLINK-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske resolved FLINK-14160. --- Fix Version/s: 1.9.1 1.10.0 Assignee: David Anderson Resolution: Implemented Implemented for flink.git with release-1.9: 5997accdf9d5a8c5a934b7888f6826ca9fd1acf8 master: c83c18671bc0056a341877f312ba293ae5811953 Implemented for flink-playgrounds.git with release-1.9: 41acc3b90bbf43e6879f2e3d9cdded0cac980524 master: 1c7c254fc7827e74db7c3c387348e7ca2219788a > Extend Operations Playground with --backpressure option > --- > > Key: FLINK-14160 > URL: https://issues.apache.org/jira/browse/FLINK-14160 > Project: Flink > Issue Type: New Feature > Components: Documentation >Reporter: David Anderson >Assignee: David Anderson >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Add a --backpressure option to the ClickEventCount job used in the operations > playground. This will insert an optional operator into the job that causes > severe, periodic backpressure that can be observed in the metrics and web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14117) Translate changes on documentation index page to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933345#comment-16933345 ] Fabian Hueske commented on FLINK-14117: --- Yes, it's about different pages. Sorry for not making this clear. This issue is about https://ci.apache.org/projects/flink/flink-docs-master/index.html FLINK-14116 is about https://ci.apache.org/projects/flink/flink-docs-master/getting-started/index.html > Translate changes on documentation index page to Chinese > > > Key: FLINK-14117 > URL: https://issues.apache.org/jira/browse/FLINK-14117 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Priority: Major > > The changes of commit > [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-1a523bd9fa0dbf998008b37579210e12] > on the documentation index page should be translated to Chinese. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14115) Translate DataStream Code Walkthrough to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-14115: - Assignee: gaofeilong > Translate DataStream Code Walkthrough to Chinese > > > Key: FLINK-14115 > URL: https://issues.apache.org/jira/browse/FLINK-14115 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Assignee: gaofeilong >Priority: Major > > The new DataStream Code Walkthrough should be translated to Chinese: > https://github.com/apache/flink/blob/master/docs/getting-started/walkthroughs/datastream_api.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14115) Translate DataStream Code Walkthrough to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933344#comment-16933344 ] Fabian Hueske commented on FLINK-14115: --- Hi [~gaofeilong], sure! Done :-) > Translate DataStream Code Walkthrough to Chinese > > > Key: FLINK-14115 > URL: https://issues.apache.org/jira/browse/FLINK-14115 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Assignee: gaofeilong >Priority: Major > > The new DataStream Code Walkthrough should be translated to Chinese: > https://github.com/apache/flink/blob/master/docs/getting-started/walkthroughs/datastream_api.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14116) Translate changes on Getting Started Overview to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932265#comment-16932265 ] Fabian Hueske commented on FLINK-14116: --- Thanks! I've assigned the ticket to you :-) > Translate changes on Getting Started Overview to Chinese > > > Key: FLINK-14116 > URL: https://issues.apache.org/jira/browse/FLINK-14116 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Assignee: Rong Zhang >Priority: Major > > The changes of commit > [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] > on the Getting Started Overview should be translated to Chinese -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14116) Translate changes on Getting Started Overview to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-14116: - Assignee: Rong Zhang > Translate changes on Getting Started Overview to Chinese > > > Key: FLINK-14116 > URL: https://issues.apache.org/jira/browse/FLINK-14116 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Assignee: Rong Zhang >Priority: Major > > The changes of commit > [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] > on the Getting Started Overview should be translated to Chinese -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14095) Add Pravega in Flink ecosystem page
[ https://issues.apache.org/jira/browse/FLINK-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1693#comment-1693 ] Fabian Hueske edited comment on FLINK-14095 at 9/18/19 8:58 AM: Yes, that would be the right place IMO. Please check our contribution guide for the website: https://flink.apache.org/contributing/improve-website.html Thanks was (Author: fhueske): Yes, that would be the right place IMO. Thanks > Add Pravega in Flink ecosystem page > --- > > Key: FLINK-14095 > URL: https://issues.apache.org/jira/browse/FLINK-14095 > Project: Flink > Issue Type: Improvement > Components: Documentation, Project Website >Reporter: Yumin Zhou >Assignee: Yumin Zhou >Priority: Minor > > Flink Ecosystem page ([http://flink.apache.org/ecosystem.html]) lists a set > of third-party projects that support working with Flink. > Pravega ([http://pravega.io/]) provides a new storage abstraction - a stream > - for continuous and unbounded data. It is a high-performance and > strongly-consistant stream storage. We also developed Flink connector > ([https://github.com/pravega/flink-connectors]) to adapt Flink's source, sink > and Table API to support Flink computation. I am wondering if I could create > a PR to add our project - Pravega in the ecosystem page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14095) Add Pravega in Flink ecosystem page
[ https://issues.apache.org/jira/browse/FLINK-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1693#comment-1693 ] Fabian Hueske commented on FLINK-14095: --- Yes, that would be the right place IMO. Thanks > Add Pravega in Flink ecosystem page > --- > > Key: FLINK-14095 > URL: https://issues.apache.org/jira/browse/FLINK-14095 > Project: Flink > Issue Type: Improvement > Components: Documentation, Project Website >Reporter: Yumin Zhou >Assignee: Yumin Zhou >Priority: Minor > > Flink Ecosystem page ([http://flink.apache.org/ecosystem.html]) lists a set > of third-party projects that support working with Flink. > Pravega ([http://pravega.io/]) provides a new storage abstraction - a stream > - for continuous and unbounded data. It is a high-performance and > strongly-consistant stream storage. We also developed Flink connector > ([https://github.com/pravega/flink-connectors]) to adapt Flink's source, sink > and Table API to support Flink computation. I am wondering if I could create > a PR to add our project - Pravega in the ecosystem page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14116) Translate changes on Getting Started Overview to Chinese
[ https://issues.apache.org/jira/browse/FLINK-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske updated FLINK-14116: -- Description: The changes of commit [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] on the Getting Started Overview should be translated to Chinese (was: The changes of commit [https://github.com/apache/flink/blob/master/docs/getting-started/walkthroughs/datastream_api.zh.md|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] on the Getting Started Overview should be translated to Chinese) > Translate changes on Getting Started Overview to Chinese > > > Key: FLINK-14116 > URL: https://issues.apache.org/jira/browse/FLINK-14116 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.10.0 >Reporter: Fabian Hueske >Priority: Major > > The changes of commit > [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] > on the Getting Started Overview should be translated to Chinese -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14117) Translate changes on documentation index page to Chinese
Fabian Hueske created FLINK-14117: - Summary: Translate changes on documentation index page to Chinese Key: FLINK-14117 URL: https://issues.apache.org/jira/browse/FLINK-14117 Project: Flink Issue Type: Task Components: chinese-translation, Documentation Affects Versions: 1.10.0 Reporter: Fabian Hueske The changes of commit [ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-1a523bd9fa0dbf998008b37579210e12] on the documentation index page should be translated to Chinese. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14116) Translate changes on Getting Started Overview to Chinese
Fabian Hueske created FLINK-14116: - Summary: Translate changes on Getting Started Overview to Chinese Key: FLINK-14116 URL: https://issues.apache.org/jira/browse/FLINK-14116 Project: Flink Issue Type: Task Components: chinese-translation, Documentation Affects Versions: 1.10.0 Reporter: Fabian Hueske The changes of commit [https://github.com/apache/flink/blob/master/docs/getting-started/walkthroughs/datastream_api.zh.md|https://github.com/apache/flink/commit/ee0d6fdf0604d74bd1cf9a6eb9cf5338ac1aa4f9#diff-759f29741e3adc9d9cdc95c996f25869] on the Getting Started Overview should be translated to Chinese -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14115) Translate DataStream Code Walkthrough to Chinese
Fabian Hueske created FLINK-14115: - Summary: Translate DataStream Code Walkthrough to Chinese Key: FLINK-14115 URL: https://issues.apache.org/jira/browse/FLINK-14115 Project: Flink Issue Type: Task Components: chinese-translation, Documentation Affects Versions: 1.10.0 Reporter: Fabian Hueske The new DataStream Code Walkthrough should be translated to Chinese: https://github.com/apache/flink/blob/master/docs/getting-started/walkthroughs/datastream_api.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-12746) Getting Started - DataStream Example Walkthrough
[ https://issues.apache.org/jira/browse/FLINK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske resolved FLINK-12746. --- Fix Version/s: 1.10.0 Resolution: Implemented Implemented for 1.10 with df8f9a586143bbd719b6e9f03592e02e45629a9a > Getting Started - DataStream Example Walkthrough > > > Key: FLINK-12746 > URL: https://issues.apache.org/jira/browse/FLINK-12746 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Konstantin Knauf >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The planned structure for the new Getting Started Guide is > * Flink Overview (~ two pages) > * Project Setup > ** Java > ** Scala > ** Python > * Quickstarts > ** Example Walkthrough - Table API / SQL > ** Example Walkthrough - DataStream API > * Docker Playgrounds > ** Flink Cluster Playground > ** Flink Interactive SQL Playground > In this ticket we should add "Project Setup" and "Quickstarts -> Example > Walkthrough - DataStream API", which covers everything what we have today. > This will replace the current "Tutorials" and "Examples" section, which can > be removed as part of this ticket as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-992) Create CollectionDataSets by reading (client) local files.
[
https://issues.apache.org/jira/browse/FLINK-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931529#comment-16931529
]
Fabian Hueske commented on FLINK-992:
-
Hi [~alfredlu], this is a very old issue.
We are currently redesigning the architecture of the submission client.
Not sure if this still fits into the new design.
Actually, I'd rather close this issue.
Best, Fabian
> Create CollectionDataSets by reading (client) local files.
> --
>
> Key: FLINK-992
> URL: https://issues.apache.org/jira/browse/FLINK-992
> Project: Flink
> Issue Type: New Feature
> Components: API / DataSet, API / Python
>Reporter: Fabian Hueske
>Priority: Minor
> Labels: starter
>
> {{CollectionDataSets}} are a nice way to feed data into programs.
> We could add support to read a client-local file at program construction time
> using a FileInputFormat, put its data into a CollectionDataSet, and ship its
> data together with the program.
> This would remove the need to upload small files into DFS which are used
> together with some large input (stored in DFS).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-14095) Add Pravega in Flink ecosystem page
[ https://issues.apache.org/jira/browse/FLINK-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931495#comment-16931495 ] Fabian Hueske commented on FLINK-14095: --- Yes, please add Pravega and open a PR! Thanks, Fabian > Add Pravega in Flink ecosystem page > --- > > Key: FLINK-14095 > URL: https://issues.apache.org/jira/browse/FLINK-14095 > Project: Flink > Issue Type: Improvement > Components: Documentation, Project Website >Reporter: Yumin Zhou >Assignee: Yumin Zhou >Priority: Minor > > Flink Ecosystem page ([http://flink.apache.org/ecosystem.html]) lists a set > of third-party projects that support working with Flink. > Pravega ([http://pravega.io/]) provides a new storage abstraction - a stream > - for continuous and unbounded data. It is a high-performance and > strongly-consistant stream storage. We also developed Flink connector > ([https://github.com/pravega/flink-connectors]) to adapt Flink's source, sink > and Table API to support Flink computation. I am wondering if I could create > a PR to add our project - Pravega in the ecosystem page. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13862) Remove or rewrite Execution Plan docs
[
https://issues.apache.org/jira/browse/FLINK-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929054#comment-16929054
]
Fabian Hueske commented on FLINK-13862:
---
I found the plan visualizer quite handy when debugging / explaining more
complex user programs.
The WebUI requires a running job (AFAIK), the visualizer just the JSON
representation of the JobGraph that can be more easily shared (no JAR, no
environment dependency, ...).
> Remove or rewrite Execution Plan docs
> -
>
> Key: FLINK-13862
> URL: https://issues.apache.org/jira/browse/FLINK-13862
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.10.0, 1.9.1
>
>
> The *Execution Plans* section is totally outdated and refers to the old
> {{tools/planVisalizer.html}} file that has been removed for two years.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/execution_plans.html
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-7151) Add a function SQL DDL
[
https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927323#comment-16927323
]
Fabian Hueske commented on FLINK-7151:
--
Hi [~phoenixjiangnan], I agree. Splitting the issue in two is a good approach.
There are a few architectural decisions in the UDF in external JAR feature that
require in my opinion a broader attention.
I don't think we need a full-blown FLIP for the basic syntax, but it would make
sense to start a discussion thread on the dev mailing list to raise the
awareness of others.
Cheers, Fabian
> Add a function SQL DDL
> --
>
> Key: FLINK-7151
> URL: https://issues.apache.org/jira/browse/FLINK-7151
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: yuemeng
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.10.0
>
>
> Based on create function and table.we can register a udf,udaf,udtf use sql:
> {code}
> CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS
> class_name [WITH (properties.)];
> DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name;
> ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO
> new_name;
> {code}
> {code}
> CREATE function 'TOPK' AS
> 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF';
> INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP
> BY id;
> {code}
> This ticket can assume that the function class is already loaded in classpath
> by users. Advanced syntax like to how to dynamically load udf libraries from
> external locations can be on a separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-7151) Add a function SQL DDL
[
https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926733#comment-16926733
]
Fabian Hueske commented on FLINK-7151:
--
Hi [~ZhenqiuHuang], thanks for sharing the proposal!
Just a few remarks:
* the proposal assumes that the client side (the one submitting the SQL / Table
API query) can access the external library. It would first be downloaded to the
client and from there shipped to the Flink master and from there distributed
via the blob service
* downloading external code (possibly from a public network) is a security
risk. We should probably add a switch to disable this feature.
* Do we need to think about how to load libraries (JARs) from external catalogs
(like Metastore) or is this is a separate discussion?
Thanks, Fabian
> Add a function SQL DDL
> --
>
> Key: FLINK-7151
> URL: https://issues.apache.org/jira/browse/FLINK-7151
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: yuemeng
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.10.0
>
>
> Based on create function and table.we can register a udf,udaf,udtf use sql:
> {code}
> CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS
> class_name [WITH (properties.)];
> DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name;
> ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO
> new_name;
> {code}
> {code}
> CREATE function 'TOPK' AS
> 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF';
> INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP
> BY id;
> {code}
> This ticket can assume that the function class is already loaded in classpath
> by users. Advanced syntax like to how to dynamically load udf libraries from
> external locations can be on a separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (FLINK-13942) Add Overview page for Getting Started section
[ https://issues.apache.org/jira/browse/FLINK-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske resolved FLINK-13942. --- Resolution: Fixed Done for 1.10 with 275fe9641a8be0815fee152388bf105941b23418 Done for 1.9 with fcdad7265866b5f5b44bb09f7b036e1540f7e27f > Add Overview page for Getting Started section > - > > Key: FLINK-13942 > URL: https://issues.apache.org/jira/browse/FLINK-13942 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.9.0, 1.10.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The Getting Started section provide different types of tutorials that target > users with different interests and backgrounds. > We should add a brief overview page that describes the different tutorials > such that users easily find the material that they need to get started with > Flink. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13994) Translate "Getting Started" overview to Chinese
Fabian Hueske created FLINK-13994: - Summary: Translate "Getting Started" overview to Chinese Key: FLINK-13994 URL: https://issues.apache.org/jira/browse/FLINK-13994 Project: Flink Issue Type: Task Components: chinese-translation, Documentation Reporter: Fabian Hueske The "Getting Started" overview page needs to be translated to Chinese: https://github.com/apache/flink/blob/master/docs/getting-started/index.zh.md -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13872) Translate Operations Playground to Chinese
[ https://issues.apache.org/jira/browse/FLINK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924208#comment-16924208 ] Fabian Hueske commented on FLINK-13872: --- Hi [~a64011144], The playground on the master branch was updated. You can go ahead with the translation. Thank you, Fabian > Translate Operations Playground to Chinese > -- > > Key: FLINK-13872 > URL: https://issues.apache.org/jira/browse/FLINK-13872 > Project: Flink > Issue Type: Task > Components: chinese-translation, Documentation >Affects Versions: 1.9.0 >Reporter: Fabian Hueske >Assignee: Lord i Will >Priority: Major > > The [Operations > Playground|https://ci.apache.org/projects/flink/flink-docs-release-1.9/getting-started/docker-playgrounds/flink_operations_playground.html] > is a quick and convenient way to learn about Flink's operational features > (job submission, failure recovery, job updates, scaling, metrics). > We should translate it to Chinese as well. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924205#comment-16924205
]
Fabian Hueske commented on FLINK-12749:
---
Playground job code removed from master (1.10) with
8998b2fe286d4ac788011f03fc6953f8b040b119
Playground readded to master with a17623ba51f7cae83bc789cd4f8ffc7f105a8715
added to release-1.9 with cd756d6609064254cfe50191635ea762af477793
added to release-1.8 with 9868ae1dde0ac6709b4b7a8afa6e7121cfb72b4d
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (FLINK-13821) Website must link to License etc
[ https://issues.apache.org/jira/browse/FLINK-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske resolved FLINK-13821. --- Resolution: Fixed Fixed with https://github.com/apache/flink-web/commit/8c02afc727e2770db11e0698f2de45db46f57fd0 We're all green :-) > Website must link to License etc > > > Key: FLINK-13821 > URL: https://issues.apache.org/jira/browse/FLINK-13821 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Sebb >Assignee: Robert Metzger >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > ASF project websites must have certain links: > Apachecon > License > Thanks > Security > Sponsor/Donat > Please see: > https://whimsy.apache.org/site/project/flink -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13821) Website must link to License etc
[ https://issues.apache.org/jira/browse/FLINK-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924018#comment-16924018 ] Fabian Hueske commented on FLINK-13821: --- Thanks [~sebb], I'll have a look. > Website must link to License etc > > > Key: FLINK-13821 > URL: https://issues.apache.org/jira/browse/FLINK-13821 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Sebb >Assignee: Robert Metzger >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > ASF project websites must have certain links: > Apachecon > License > Thanks > Security > Sponsor/Donat > Please see: > https://whimsy.apache.org/site/project/flink -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13975) Translate "Upcoming Events" on Chinese index.html
Fabian Hueske created FLINK-13975: - Summary: Translate "Upcoming Events" on Chinese index.html Key: FLINK-13975 URL: https://issues.apache.org/jira/browse/FLINK-13975 Project: Flink Issue Type: Task Components: chinese-translation, Project Website Reporter: Fabian Hueske We recently added a section for "Upcoming Events" to the index page of the Flink website. We need to translate "Upcoming Events" on the Chinese version of the main page. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-13925) ClassLoader in BlobLibraryCacheManager is not using context class loader
[ https://issues.apache.org/jira/browse/FLINK-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-13925: - Assignee: Jan Lukavský > ClassLoader in BlobLibraryCacheManager is not using context class loader > > > Key: FLINK-13925 > URL: https://issues.apache.org/jira/browse/FLINK-13925 > Project: Flink > Issue Type: Bug >Affects Versions: 1.8.1, 1.9.0 >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Labels: pull-request-available > Fix For: 1.9.1, 1.8.3 > > Time Spent: 10m > Remaining Estimate: 0h > > Use thread's current context classloader as parent class loader of flink user > code class loaders. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13942) Add Overview page for Getting Started section
Fabian Hueske created FLINK-13942: - Summary: Add Overview page for Getting Started section Key: FLINK-13942 URL: https://issues.apache.org/jira/browse/FLINK-13942 Project: Flink Issue Type: Sub-task Components: Documentation Affects Versions: 1.9.0, 1.10.0 Reporter: Fabian Hueske Assignee: Fabian Hueske The Getting Started section provide different types of tutorials that target users with different interests and backgrounds. We should add a brief overview page that describes the different tutorials such that users easily find the material that they need to get started with Flink. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-12750) Gettting Started - Docker Playground - Interactive SQL Playground
[ https://issues.apache.org/jira/browse/FLINK-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-12750: - Assignee: Fabian Hueske > Gettting Started - Docker Playground - Interactive SQL Playground > - > > Key: FLINK-12750 > URL: https://issues.apache.org/jira/browse/FLINK-12750 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Konstantin Knauf >Assignee: Fabian Hueske >Priority: Major > > The planned structure for the new Getting Started Guide is > * Flink Overview (~ two pages) > * Project Setup > ** Java > ** Scala > ** Python > * Quickstarts > ** Example Walkthrough - Table API / SQL > ** Example Walkthrough - DataStream API > * Docker Playgrounds > ** Flink Cluster Playground > ** Flink Interactive SQL Playground > In this ticket we add the Flink Cluster Playground, a docker-compose based > setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster) and > an SQL-Client. > The general setup should be in line with FLINK-12749. > **Open Questions** > * Where to host the SQL Client image? Can we somehow also use existing plain > Flink images? -- This message was sent by Atlassian Jira (v8.3.2#803003)
