[jira] [Updated] (FLINK-8865) Add CLI query code completion in SQL Client
[ https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-8865: -- Labels: pull-request-available (was: ) > Add CLI query code completion in SQL Client > --- > > Key: FLINK-8865 > URL: https://issues.apache.org/jira/browse/FLINK-8865 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Timo Walther >Assignee: xueyu >Priority: Major > Labels: pull-request-available > > This issue is a subtask of part two "Full Embedded SQL Client" of the > implementation plan mentioned in > [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client]. > Calcite already offers a code completion functionality. It would be great if > we could expose this feature also through the SQL CLI Client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client
[ https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637827#comment-16637827 ] ASF GitHub Bot commented on FLINK-8865: --- xueyumusic opened a new pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client URL: https://github.com/apache/flink/pull/6791 ## What is the purpose of the change This PR adds cli query code completion in SQL Client based on Calcite SqlAdvisor ## Brief change log This change added tests and can be verified as follows: - *Added sql completer test* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not documented, add afterward) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add CLI query code completion in SQL Client > --- > > Key: FLINK-8865 > URL: https://issues.apache.org/jira/browse/FLINK-8865 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Timo Walther >Assignee: xueyu >Priority: Major > Labels: pull-request-available > > This issue is a subtask of part two "Full Embedded SQL Client" of the > implementation plan mentioned in > [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client]. > Calcite already offers a code completion functionality. It would be great if > we could expose this feature also through the SQL CLI Client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] xueyumusic opened a new pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client
xueyumusic opened a new pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client URL: https://github.com/apache/flink/pull/6791 ## What is the purpose of the change This PR adds cli query code completion in SQL Client based on Calcite SqlAdvisor ## Brief change log This change added tests and can be verified as follows: - *Added sql completer test* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not documented, add afterward) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637707#comment-16637707
]
ASF GitHub Bot commented on FLINK-10487:
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222518000
##
File path:
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.java.StreamTableEnvironment;
+
+import java.util.Arrays;
+
+/**
+ * Simple example for demonstrating the use of SQL on a Stream Table in Java.
+ *
+ * This example shows how to:
+ * - Convert DataStreams to Tables
+ * - Register a Table under a name
+ * - Run a StreamSQL query on the registered Table
+ *
+ */
+public class StreamSQLExample {
+
+ //
*
+ // PROGRAM
+ //
*
+
+ public static void main(String[] args) throws Exception {
+
+ // set up execution environment
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream orderA = env.fromCollection(Arrays.asList(
+ new Order(1L, "beer", 3),
+ new Order(1L, "diaper", 4),
+ new Order(3L, "rubber", 2)));
+
+ DataStream orderB = env.fromCollection(Arrays.asList(
+ new Order(2L, "pen", 3),
+ new Order(2L, "rubber", 3),
+ new Order(4L, "beer", 1)));
+
+ // register the DataSet as table "WordCount"
+ Table tableA = tEnv.fromDataStream(orderA, "user, product,
amount");
+ tEnv.registerDataStream("OrderB", orderB, "user, product,
amount");
+
+ // run a SQL query on the Table and retrieve the result as a
new Table
Review comment:
Change to "union the two tables" making it consistent to the comments in
scala？
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in
[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637705#comment-16637705
]
ASF GitHub Bot commented on FLINK-10487:
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222517996
##
File path:
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.java.StreamTableEnvironment;
+
+import java.util.Arrays;
+
+/**
+ * Simple example for demonstrating the use of SQL on a Stream Table in Java.
+ *
+ * This example shows how to:
+ * - Convert DataStreams to Tables
+ * - Register a Table under a name
+ * - Run a StreamSQL query on the registered Table
+ *
+ */
+public class StreamSQLExample {
+
+ //
*
+ // PROGRAM
+ //
*
+
+ public static void main(String[] args) throws Exception {
+
+ // set up execution environment
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream orderA = env.fromCollection(Arrays.asList(
+ new Order(1L, "beer", 3),
+ new Order(1L, "diaper", 4),
+ new Order(3L, "rubber", 2)));
+
+ DataStream orderB = env.fromCollection(Arrays.asList(
+ new Order(2L, "pen", 3),
+ new Order(2L, "rubber", 3),
+ new Order(4L, "beer", 1)));
+
+ // register the DataSet as table "WordCount"
Review comment:
Correct the comments. I think we can add a comment for each of the following
two fromDataStream.
One is "convert the DataStream into a Table"
The other is "register the DataStream under the name "OrderB""
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637706#comment-16637706
]
ASF GitHub Bot commented on FLINK-10487:
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222518009
##
File path:
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamSQLExample.scala
##
@@ -54,12 +54,12 @@ object StreamSQLExample {
Order(4L, "beer", 1)))
// register the DataStreams under the name "OrderA" and "OrderB"
-tEnv.registerDataStream("OrderA", orderA, 'user, 'product, 'amount)
+var tableA = tEnv.fromDataStream(orderA, 'user, 'product, 'amount)
Review comment:
Adapt the comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222518000
##
File path:
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.java.StreamTableEnvironment;
+
+import java.util.Arrays;
+
+/**
+ * Simple example for demonstrating the use of SQL on a Stream Table in Java.
+ *
+ * This example shows how to:
+ * - Convert DataStreams to Tables
+ * - Register a Table under a name
+ * - Run a StreamSQL query on the registered Table
+ *
+ */
+public class StreamSQLExample {
+
+ //
*
+ // PROGRAM
+ //
*
+
+ public static void main(String[] args) throws Exception {
+
+ // set up execution environment
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream orderA = env.fromCollection(Arrays.asList(
+ new Order(1L, "beer", 3),
+ new Order(1L, "diaper", 4),
+ new Order(3L, "rubber", 2)));
+
+ DataStream orderB = env.fromCollection(Arrays.asList(
+ new Order(2L, "pen", 3),
+ new Order(2L, "rubber", 3),
+ new Order(4L, "beer", 1)));
+
+ // register the DataSet as table "WordCount"
+ Table tableA = tEnv.fromDataStream(orderA, "user, product,
amount");
+ tEnv.registerDataStream("OrderB", orderB, "user, product,
amount");
+
+ // run a SQL query on the Table and retrieve the result as a
new Table
Review comment:
Change to "union the two tables" making it consistent to the comments in
scala？
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222517996
##
File path:
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.java.StreamTableEnvironment;
+
+import java.util.Arrays;
+
+/**
+ * Simple example for demonstrating the use of SQL on a Stream Table in Java.
+ *
+ * This example shows how to:
+ * - Convert DataStreams to Tables
+ * - Register a Table under a name
+ * - Run a StreamSQL query on the registered Table
+ *
+ */
+public class StreamSQLExample {
+
+ //
*
+ // PROGRAM
+ //
*
+
+ public static void main(String[] args) throws Exception {
+
+ // set up execution environment
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream orderA = env.fromCollection(Arrays.asList(
+ new Order(1L, "beer", 3),
+ new Order(1L, "diaper", 4),
+ new Order(3L, "rubber", 2)));
+
+ DataStream orderB = env.fromCollection(Arrays.asList(
+ new Order(2L, "pen", 3),
+ new Order(2L, "rubber", 3),
+ new Order(4L, "beer", 1)));
+
+ // register the DataSet as table "WordCount"
Review comment:
Correct the comments. I think we can add a comment for each of the following
two fromDataStream.
One is "convert the DataStream into a Table"
The other is "register the DataStream under the name "OrderB""
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API
hequn8128 commented on a change in pull request #6790: [FLINK-10487] fix
invalid Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222518009
##
File path:
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamSQLExample.scala
##
@@ -54,12 +54,12 @@ object StreamSQLExample {
Order(4L, "beer", 1)))
// register the DataStreams under the name "OrderA" and "OrderB"
-tEnv.registerDataStream("OrderA", orderA, 'user, 'product, 'amount)
+var tableA = tEnv.fromDataStream(orderA, 'user, 'product, 'amount)
Review comment:
Adapt the comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637651#comment-16637651
]
ASF GitHub Bot commented on FLINK-10487:
isunjin commented on a change in pull request #6790: [FLINK-10487] fix invalid
Flink SQL example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#discussion_r222506848
##
File path:
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java
##
@@ -0,0 +1,85 @@
+package org.apache.flink.table.examples.java;
Review comment:
you need add Apache Licence header to prevent buid failure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] isunjin commented on a change in pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API
isunjin commented on a change in pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API URL: https://github.com/apache/flink/pull/6790#discussion_r222506848 ## File path: flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/StreamSQLExample.java ## @@ -0,0 +1,85 @@ +package org.apache.flink.table.examples.java; Review comment: you need add Apache Licence header to prevent buid failure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637608#comment-16637608
]
ASF GitHub Bot commented on FLINK-10487:
bowenli86 opened a new pull request #6790: [FLINK-10487] fix invalid Flink SQL
example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790
## What is the purpose of the change
Fix invalid java example (which uses a non-existing API) in doc
## Brief change log
- Fix the invalid java example to use correct SQL API
- Add a runnable Flink SQL example for Java. The example's logic is the
same as the corresponding example in scala
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
The example can be run manually to verify it.
## Does this pull request potentially affect one of the following parts:
none
## Documentation
- Does this pull request introduce a new feature? (no)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10487:
---
Labels: pull-request-available (was: )
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] bowenli86 opened a new pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API
bowenli86 opened a new pull request #6790: [FLINK-10487] fix invalid Flink SQL example and add runnable SQL example for Java API URL: https://github.com/apache/flink/pull/6790 ## What is the purpose of the change Fix invalid java example (which uses a non-existing API) in doc ## Brief change log - Fix the invalid java example to use correct SQL API - Add a runnable Flink SQL example for Java. The example's logic is the same as the corresponding example in scala ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. The example can be run manually to verify it. ## Does this pull request potentially affect one of the following parts: none ## Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Description:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
...
{code}
But there's no API of {{toTable()}} in StreamTableEnvironment
cc [~fhueske]
was:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
cc [~fhueske]
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Description:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
cc [~fhue...@gmail.com]
was:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no APIs like {{toTable()}} in StreamTableEnvironment
> cc [~fhue...@gmail.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Description:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
cc [~fhueske]
was:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
cc [~fhue...@gmail.com]
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no APIs like {{toTable()}} in StreamTableEnvironment
> cc [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix deprecated Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Description:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}} in StreamTableEnvironment
was:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}}, {{registerDataStream()}},
{{registerDataStream()}} in StreamTableEnvironment
> fix deprecated Flink SQL example
>
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no APIs like {{toTable()}} in StreamTableEnvironment
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix invalid Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Summary: fix invalid Flink SQL example (was: fix deprecated Flink SQL
example)
> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no APIs like {{toTable()}} in StreamTableEnvironment
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix deprecated Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Description:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no APIs like {{toTable()}}, {{registerDataStream()}},
{{registerDataStream()}} in StreamTableEnvironment
was:
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no {{toTable()}} and {{registerDataStream()}} API in
StreamTableEnvironment anymore
> fix deprecated Flink SQL example
>
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no APIs like {{toTable()}}, {{registerDataStream()}},
> {{registerDataStream()}} in StreamTableEnvironment
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10487) fix deprecated Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li reassigned FLINK-10487:
Assignee: Bowen Li
> fix deprecated Flink SQL example
>
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no {{toTable()}} and {{registerDataStream()}} API in
> StreamTableEnvironment anymore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10487) fix deprecated Flink SQL example
[
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-10487:
-
Component/s: Table API & SQL
> fix deprecated Flink SQL example
>
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table API SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Priority: Major
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html
> , the example is :
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
> "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> // SQL query with a registered table
> // register the DataStream as table "Orders"
> tableEnv.registerDataStream("Orders", ds, "user, product, amount");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result2 = tableEnv.sqlQuery(
> "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no {{toTable()}} and {{registerDataStream()}} API in
> StreamTableEnvironment anymore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10487) fix deprecated Flink SQL example
Bowen Li created FLINK-10487:
Summary: fix deprecated Flink SQL example
Key: FLINK-10487
URL: https://issues.apache.org/jira/browse/FLINK-10487
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.6.1, 1.6.0, 1.7.0
Reporter: Bowen Li
Fix For: 1.7.0
On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html ,
the example is :
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream> ds = env.addSource(...);
// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sqlQuery(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
Table result2 = tableEnv.sqlQuery(
"SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
...
{code}
But there's no {{toTable()}} and {{registerDataStream()}} API in
StreamTableEnvironment anymore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10279) Make jython limitations more obvious in documentation
[ https://issues.apache.org/jira/browse/FLINK-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637487#comment-16637487 ] ASF GitHub Bot commented on FLINK-10279: mxm commented on a change in pull request #6761: [FLINK-10279] [documentation] Make jython limitations more obvious in documentation. URL: https://github.com/apache/flink/pull/6761#discussion_r222458664 ## File path: docs/dev/stream/python.md ## @@ -50,7 +54,12 @@ existing Java streaming APIs. There are two main constraints for using Jython: * The latest Python supported version is 2.7 -* It is not straightforward to use Python C extensions +* It is not straightforward to use Python C extensions, which may prevent reuse of some libraries Review comment: Would rephrase `reuse` to `use`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make jython limitations more obvious in documentation > - > > Key: FLINK-10279 > URL: https://issues.apache.org/jira/browse/FLINK-10279 > Project: Flink > Issue Type: Task > Components: Documentation, Python API >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > > The "Python Programming Guide (Streaming) Beta" at > [https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/python.html] > does not sufficiently highlight limitations of the API. It should probably > have a prominent disclaimer right at the top stating that this actually isn't > a "Python" API but Jython, which likely means that the user looking for a > solution to run native Python code won't be able to use many important > libraries, which is often the reason to look for Python support in first > place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] mxm commented on a change in pull request #6761: [FLINK-10279] [documentation] Make jython limitations more obvious in documentation.
mxm commented on a change in pull request #6761: [FLINK-10279] [documentation] Make jython limitations more obvious in documentation. URL: https://github.com/apache/flink/pull/6761#discussion_r222458664 ## File path: docs/dev/stream/python.md ## @@ -50,7 +54,12 @@ existing Java streaming APIs. There are two main constraints for using Jython: * The latest Python supported version is 2.7 -* It is not straightforward to use Python C extensions +* It is not straightforward to use Python C extensions, which may prevent reuse of some libraries Review comment: Would rephrase `reuse` to `use`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10475) Standalone HA - Leader election is not triggered on loss of leader (ZK 3.5.3-beta only)
[ https://issues.apache.org/jira/browse/FLINK-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637433#comment-16637433 ] Till Rohrmann commented on FLINK-10475: --- Good to know [~Jamalarm]. We could also add it to the HA documentation. Do you wanna contribute a PR for that? Then I guess we can close this issue, right? > Standalone HA - Leader election is not triggered on loss of leader (ZK > 3.5.3-beta only) > --- > > Key: FLINK-10475 > URL: https://issues.apache.org/jira/browse/FLINK-10475 > Project: Flink > Issue Type: Bug >Affects Versions: 1.6.1, 1.5.4 >Reporter: Thomas Wozniakowski >Priority: Minor > Attachments: t1.log, t2.log, t3.log > > > Hey Guys, > Just testing the new bugfix release of 1.5.4 (edit: also happens with 1.6.1). > Happy to see that the issue of jobgraphs hanging around forever has been > resolved in standalone/zookeeper HA mode, but now I'm seeing a different > issue. > It looks like the HA failover is never triggered. I set up a 3/3/3 cluster of > zookeeper/jobmanager/taskmanagers. Started my job, all fine with the new > version. I then proceeded to kill the leading jobmanager to test the failover. > The remaining jobmanagers never triggered a leader election, and simply got > stuck. > Please give me a shout if I can provide any more useful information > EDIT > Jobmanager logs attached below. You can see that I brought up a fresh > cluster, one JM was elected leader (no taskmanagers or actual jobs in this > case). I then let the cluster sit there for half an hour or so, before > killing the leader. The log files are snapshotted maybe half an hour after > that. You can see that a second election was never triggered. > In case it's useful, our zookeeper quorum is running "3.5.3-beta". This setup > previously worked with 1.4.3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-10371) Allow to enable SSL mutual authentication on REST endpoints by configuration
[ https://issues.apache.org/jira/browse/FLINK-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-10371. --- Resolution: Fixed Fixed via 1.7.0: 12e820c69819f3850597e54d08dde82373147a18 1.6.2: 5e7237a02c77787225f98760fb025c34136af0b4 > Allow to enable SSL mutual authentication on REST endpoints by configuration > > > Key: FLINK-10371 > URL: https://issues.apache.org/jira/browse/FLINK-10371 > Project: Flink > Issue Type: Improvement > Components: Client, REST, Security >Affects Versions: 1.6.0, 1.7.0 >Reporter: Johannes Dillmann >Assignee: Johannes Dillmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0, 1.6.2 > > > With Flink 1.6 SSL mutual authentication was introduced for internal > connectivity in FLINK-9312. > SSL support for external connectivity was also introduced in regard to > encryption of the connection and verification of the Flink REST endpoint from > the client side. > But _mutual authentication between the REST endpoint and clients is not > supported yet_. > The [documentation suggests > |https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-ssl.html] > using a side car proxy to enable SSL mutual auth on the REST endpoint and > points out the advantages of using a feature rich proxy. > While this is a good rationale, there are still important use cases for > support of simple mutual authentication directly in Flink: Mainly support > for using standard images in a containerized environment. > There are tools used to setup Flink Jobs (for example on Kubernetes clusters) > and act as gateways to the Flink REST endpoint and the Flink web interface. > To prevent unauthorised access to Flink the connectivity has to be secured. > As the tools acts as gateway it is easy to create and pass a shared keystore > and truststore used for mutual authentication to the Flink instances > configurations. > To enable for SSL mutual authentication on REST endpoints, I am suggesting to > add a the configuration parameter `security.ssl.rest.authentication-enabled` > which defaults to `false`. > If it is set to `true` the `SSLUtils` factories for creating the REST server > endpoint and the REST clients should set authentication to required and share > `security.ssl.rest.keystore` and `security.ssl.rest.truststore` to setup SSL > mutual authenticated connections. > > I have a working prototype which I would gladly submit as a PR to get further > feedback. The changes to Flink are minimal and the default behaviour won't > change. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9739) Regression in supported filesystems for RocksDB
[
https://issues.apache.org/jira/browse/FLINK-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637380#comment-16637380
]
ASF GitHub Bot commented on FLINK-9739:
---
alexeyt820 commented on issue #6452: [FLINK-9739] Regression in supported
filesystems for RocksDB
URL: https://github.com/apache/flink/pull/6452#issuecomment-426756071
Is any way to workaround problem to use rocksdb with 1.6.0 at all?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Regression in supported filesystems for RocksDB
> ---
>
> Key: FLINK-9739
> URL: https://issues.apache.org/jira/browse/FLINK-9739
> Project: Flink
> Issue Type: Bug
> Components: FileSystem, State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chesnay Schepler
>Assignee: Sampath Bhat
>Priority: Major
> Labels: pull-request-available
>
> A user reporter on the [mailing
> list|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpointing-in-Flink-1-5-0-tt21173.html]
> has reported that the {{RocksDBStateBackend}} no longer supports GlusterFS
> mounted volumes.
> Configuring {{file:///home/abc/share}} lead to an exception that claimed the
> path to not be absolute.
> This was working fine in 1.4.2.
> In FLINK-6557 the {{RocksDBStateBackend}} was refactored to use java
> {{Files}} instead of Flink {{Paths}}, potentially causing the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] alexeyt820 commented on issue #6452: [FLINK-9739] Regression in supported filesystems for RocksDB
alexeyt820 commented on issue #6452: [FLINK-9739] Regression in supported filesystems for RocksDB URL: https://github.com/apache/flink/pull/6452#issuecomment-426756071 Is any way to workaround problem to use rocksdb with 1.6.0 at all? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10484) New latency tracking metrics format causes metrics cardinality explosion
[ https://issues.apache.org/jira/browse/FLINK-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637332#comment-16637332 ] Jamie Grier commented on FLINK-10484: - [~Zentol] Great. I didn't see that this had already been addressed in 1.7. What do you think about the difficulty of backporting to 1.5 and 1.6? Currently, it's a pretty big problem for people trying to run Flink at any reasonable scale – and since latency tracking is on by default basically everything breaks as soon as you upgrade a job from 1.4 to 1.5. Also, latency tracking is something that has to be disabled from application code rather than in the flink-conf.yaml file so it's very hard for infra teams supporting Flink to enforce. It's also not just a problem for Flink JM – but in our case we actually caused an observability incident company wide just because of the sheer volume of metrics being thrown at our metrics servers. > New latency tracking metrics format causes metrics cardinality explosion > > > Key: FLINK-10484 > URL: https://issues.apache.org/jira/browse/FLINK-10484 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.6.0, 1.6.1, 1.5.4 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Critical > > The new metrics format for latency tracking causes huge metrics cardinality > explosion due to the format and the fact that there is a metric reported for > a every combination of source subtask index and operator subtask index. > Yikes! > This format is actually responsible for basically taking down our metrics > system due to DDOSing our metrics servers (at Lyft). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10469) FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)
[ https://issues.apache.org/jira/browse/FLINK-10469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637271#comment-16637271 ] ASF GitHub Bot commented on FLINK-10469: isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400795 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/BufferSpiller.java ## @@ -38,6 +38,8 @@ import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.atomic.AtomicInteger; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > FileChannel may not write the whole buffer in a single call to > FileChannel.write(Buffer buffer) > --- > > Key: FLINK-10469 > URL: https://issues.apache.org/jira/browse/FLINK-10469 > Project: Flink > Issue Type: Bug > Components: Core, Network >Affects Versions: 1.4.1, 1.4.2, 1.5.3, 1.6.0, 1.6.1, 1.7.0, 1.5.4, 1.6.2 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0, 1.6.2, 1.5.5 > > > Currently all the calls to _FileChannel.write(ByteBuffer src)_ assumes that > this method will not return before the whole buffer is written, like the one > in _AsynchronousFileIOChannel.write()._ > > However, this assumption may not be right for all the environments. We have > encountered the case that only part of a buffer was written on a cluster with > a high IO load, and the target file got messy. > > To fix this issue, I think we should add a utility method in the > org.apache.flink.util.IOUtils to ensure the whole buffer is written with a > loop，and replace all the calls to _FileChannel.write(ByteBuffer)_ with this > new method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10469) FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)
[ https://issues.apache.org/jira/browse/FLINK-10469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637269#comment-16637269 ] ASF GitHub Bot commented on FLINK-10469: isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400575 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/AsynchronousFileIOChannel.java ## @@ -28,6 +28,7 @@ import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > FileChannel may not write the whole buffer in a single call to > FileChannel.write(Buffer buffer) > --- > > Key: FLINK-10469 > URL: https://issues.apache.org/jira/browse/FLINK-10469 > Project: Flink > Issue Type: Bug > Components: Core, Network >Affects Versions: 1.4.1, 1.4.2, 1.5.3, 1.6.0, 1.6.1, 1.7.0, 1.5.4, 1.6.2 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0, 1.6.2, 1.5.5 > > > Currently all the calls to _FileChannel.write(ByteBuffer src)_ assumes that > this method will not return before the whole buffer is written, like the one > in _AsynchronousFileIOChannel.write()._ > > However, this assumption may not be right for all the environments. We have > encountered the case that only part of a buffer was written on a cluster with > a high IO load, and the target file got messy. > > To fix this issue, I think we should add a utility method in the > org.apache.flink.util.IOUtils to ensure the whole buffer is written with a > loop，and replace all the calls to _FileChannel.write(ByteBuffer)_ with this > new method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10469) FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)
[ https://issues.apache.org/jira/browse/FLINK-10469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637270#comment-16637270 ] ASF GitHub Bot commented on FLINK-10469: isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400725 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpillingAdaptiveSpanningRecordDeserializer.java ## @@ -39,6 +39,8 @@ import java.util.Arrays; import java.util.Random; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > FileChannel may not write the whole buffer in a single call to > FileChannel.write(Buffer buffer) > --- > > Key: FLINK-10469 > URL: https://issues.apache.org/jira/browse/FLINK-10469 > Project: Flink > Issue Type: Bug > Components: Core, Network >Affects Versions: 1.4.1, 1.4.2, 1.5.3, 1.6.0, 1.6.1, 1.7.0, 1.5.4, 1.6.2 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0, 1.6.2, 1.5.5 > > > Currently all the calls to _FileChannel.write(ByteBuffer src)_ assumes that > this method will not return before the whole buffer is written, like the one > in _AsynchronousFileIOChannel.write()._ > > However, this assumption may not be right for all the environments. We have > encountered the case that only part of a buffer was written on a cluster with > a high IO load, and the target file got messy. > > To fix this issue, I think we should add a utility method in the > org.apache.flink.util.IOUtils to ensure the whole buffer is written with a > loop，and replace all the calls to _FileChannel.write(ByteBuffer)_ with this > new method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel
isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400795 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/BufferSpiller.java ## @@ -38,6 +38,8 @@ import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.atomic.AtomicInteger; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel
isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400725 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpillingAdaptiveSpanningRecordDeserializer.java ## @@ -39,6 +39,8 @@ import java.util.Arrays; import java.util.Random; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel
isunjin commented on a change in pull request #6788: [FLINK-10469][core] make sure to always write the whole buffer to FileChannel URL: https://github.com/apache/flink/pull/6788#discussion_r222400575 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/AsynchronousFileIOChannel.java ## @@ -28,6 +28,7 @@ import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import static org.apache.flink.util.FileUtils.writeCompletely; Review comment: i would avoid import static This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10316) Kinesalite bug fix (FLINK-9402) doesn't fully fix the problem.
[
https://issues.apache.org/jira/browse/FLINK-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637252#comment-16637252
]
ASF GitHub Bot commented on FLINK-10316:
diego-carvallo-tx opened a new pull request #6789: [FLINK-10316][kinesis] bug
was preventing FlinkKinesisProducer to connect to Kinesalite
URL: https://github.com/apache/flink/pull/6789
## What is the purpose of the change
- Some time ago
[FLINK-4197](https://issues.apache.org/jira/browse/FLINK-4197) added the
ability to connect to a local Kinesis endpoint but also introduced some bugs
- Some time ago
[FLINK-9402](https://issues.apache.org/jira/browse/FLINK-9402) added some fixes
for those bugs but for FlinkKinesisConsumer only
- This PR addresses
[FLINK-10316](https://issues.apache.org/jira/browse/FLINK-10316) to fix the
remaining bugs to allow FlinkKinesisProducer also to connect to a local Kinesis
endpoint
## Brief change log
- The method `KinesisConfigUtil.validateAwsConfiguration(Properties
config)` is used by both `FlinkKinesisConsumer` and `FlinkKinesisProducer` but
AWS_REGION/AWS_ENDPOINT validation was performed only for Consumer who needs
only one of them to be set. On the other side Producer requires AWS_REGION to
be set even if AWS_ENDPOINT is defined to connect to local Kinesis. So the
change in this PR is to change the method signature to have a boolean to
identify if it is a Consumer or Producer so the validation can be done properly
for each case:
`KinesisConfigUtil.validateAwsConfiguration(Properties config, boolean
isProducer)`
- Changed the Unit Tests accordingly
## Jira Note
There are 2 other tickets that could be closed along with this one since
they all refer to the same issue:
[FLINK-9618](https://issues.apache.org/jira/browse/FLINK-9618) and
[FLINK-8936](https://issues.apache.org/jira/browse/FLINK-8936)
## Verifying this change
This change is already covered by existing tests, such as *(please describe
tests)*.
- KinesisConfigUtilTest.testMissingAwsRegionInConfig
- KinesisConfigUtilTest.testAwsRegionOrEndpointInConfig
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kinesalite bug fix (FLINK-9402) doesn't fully fix the problem.
> --
>
> Key: FLINK-10316
> URL: https://issues.apache.org/jira/browse/FLINK-10316
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joseph Sims
>Priority: Minor
> Labels: pull-request-available
>
> Related to: https://issues.apache.org/jira/browse/FLINK-9402
> A fix (Flink-9402) was introduced to force the configurations to have
> *exactly* one of "aws.region" or "aws.endpoint", but the underlying problem
> wasn't entirely fixed.
> If "aws.endpoint" is set and "aws.region" is unset, a different exception is
> thrown, instead.
> In *KinesisConfigUtil.java*, *getValidatedProducerConfiguration()* calls:
> {code:java}
> validateAwsConfiguration(config)
> {code}
> Where the original fix was introduced ("aws.region" XOR "aws.endpoint")
> But a couple of lines later, *getValidatedProducerConfiguration()* then
> calls:
> {code:java}
> kpc.setRegion(config.getProperty(AWSConfigConstants.AWS_REGION));
> {code}
> Which explicitly checks for the existence of "aws.region", and throws an
> exception if it is not set.
> Thus, this class needs to be fixed such that "aws.region" is only required
> if "aws.endpoint" is unset, as the original ticket indicated.
> Also, *validateAwsConfiguration()* has a minor bug where is labels the
> "aws.endpoint" incorrectly.
> Current:
> {code:java}
> if (!(config.containsKey(AWSConfigConstants.AWS_REGION) ^
> config.containsKey(ConsumerConfigConstants.AWS_ENDPOINT))) {
>// per validation in AwsClientBuilder
>throw new
[jira] [Updated] (FLINK-10316) Kinesalite bug fix (FLINK-9402) doesn't fully fix the problem.
[
https://issues.apache.org/jira/browse/FLINK-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10316:
---
Labels: pull-request-available (was: )
> Kinesalite bug fix (FLINK-9402) doesn't fully fix the problem.
> --
>
> Key: FLINK-10316
> URL: https://issues.apache.org/jira/browse/FLINK-10316
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joseph Sims
>Priority: Minor
> Labels: pull-request-available
>
> Related to: https://issues.apache.org/jira/browse/FLINK-9402
> A fix (Flink-9402) was introduced to force the configurations to have
> *exactly* one of "aws.region" or "aws.endpoint", but the underlying problem
> wasn't entirely fixed.
> If "aws.endpoint" is set and "aws.region" is unset, a different exception is
> thrown, instead.
> In *KinesisConfigUtil.java*, *getValidatedProducerConfiguration()* calls:
> {code:java}
> validateAwsConfiguration(config)
> {code}
> Where the original fix was introduced ("aws.region" XOR "aws.endpoint")
> But a couple of lines later, *getValidatedProducerConfiguration()* then
> calls:
> {code:java}
> kpc.setRegion(config.getProperty(AWSConfigConstants.AWS_REGION));
> {code}
> Which explicitly checks for the existence of "aws.region", and throws an
> exception if it is not set.
> Thus, this class needs to be fixed such that "aws.region" is only required
> if "aws.endpoint" is unset, as the original ticket indicated.
> Also, *validateAwsConfiguration()* has a minor bug where is labels the
> "aws.endpoint" incorrectly.
> Current:
> {code:java}
> if (!(config.containsKey(AWSConfigConstants.AWS_REGION) ^
> config.containsKey(ConsumerConfigConstants.AWS_ENDPOINT))) {
>// per validation in AwsClientBuilder
>throw new IllegalArgumentException(String.format("Either AWS region ('%s')
> or AWS endpoint ('%s') must be set in the config.",
> AWSConfigConstants.AWS_REGION, AWSConfigConstants.AWS_REGION));
> }
> {code}
> But should be:
> {code:java}
> if (!(config.containsKey(AWSConfigConstants.AWS_REGION) ^
> config.containsKey(ConsumerConfigConstants.AWS_ENDPOINT))) {
>// per validation in AwsClientBuilder
>throw new IllegalArgumentException(String.format("Either AWS region ('%s')
> or AWS endpoint ('%s') must be set in the config.",
> AWSConfigConstants.AWS_REGION, ConsumerConfigConstants.AWS_ENDPOINT));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] diego-carvallo-tx opened a new pull request #6789: [FLINK-10316][kinesis] bug was preventing FlinkKinesisProducer to connect to Kinesalite
diego-carvallo-tx opened a new pull request #6789: [FLINK-10316][kinesis] bug was preventing FlinkKinesisProducer to connect to Kinesalite URL: https://github.com/apache/flink/pull/6789 ## What is the purpose of the change - Some time ago [FLINK-4197](https://issues.apache.org/jira/browse/FLINK-4197) added the ability to connect to a local Kinesis endpoint but also introduced some bugs - Some time ago [FLINK-9402](https://issues.apache.org/jira/browse/FLINK-9402) added some fixes for those bugs but for FlinkKinesisConsumer only - This PR addresses [FLINK-10316](https://issues.apache.org/jira/browse/FLINK-10316) to fix the remaining bugs to allow FlinkKinesisProducer also to connect to a local Kinesis endpoint ## Brief change log - The method `KinesisConfigUtil.validateAwsConfiguration(Properties config)` is used by both `FlinkKinesisConsumer` and `FlinkKinesisProducer` but AWS_REGION/AWS_ENDPOINT validation was performed only for Consumer who needs only one of them to be set. On the other side Producer requires AWS_REGION to be set even if AWS_ENDPOINT is defined to connect to local Kinesis. So the change in this PR is to change the method signature to have a boolean to identify if it is a Consumer or Producer so the validation can be done properly for each case: `KinesisConfigUtil.validateAwsConfiguration(Properties config, boolean isProducer)` - Changed the Unit Tests accordingly ## Jira Note There are 2 other tickets that could be closed along with this one since they all refer to the same issue: [FLINK-9618](https://issues.apache.org/jira/browse/FLINK-9618) and [FLINK-8936](https://issues.apache.org/jira/browse/FLINK-8936) ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. - KinesisConfigUtilTest.testMissingAwsRegionInConfig - KinesisConfigUtilTest.testAwsRegionOrEndpointInConfig ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-10478) Kafka Producer wrongly formats "%" for transaction ID
[
https://issues.apache.org/jira/browse/FLINK-10478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang reassigned FLINK-10478:
Assignee: vinoyang
> Kafka Producer wrongly formats "%" for transaction ID
> -
>
> Key: FLINK-10478
> URL: https://issues.apache.org/jira/browse/FLINK-10478
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Affects Versions: 1.4.2
> Environment: Flink 1.4.2
> Scala 2.11.12
> jdk1.8.0_162
> Running on local embedded Flink mini cluster (This happened on a standalone
> cluster with another code)
>Reporter: Obi Tetsuya
>Assignee: vinoyang
>Priority: Minor
>
> Kafka Producer with exactly-once feature uses its task name for a transaction
> ID. Because the Producer uses the name as a format string directly, in the
> case it contains "%" the job fails.
> Code to reproduce:
> {code:scala}
> object ExampleRunner {
> def main(args: Array[String]): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime)
> env.enableCheckpointing(1000)
> env.getConfig.disableSysoutLogging()
> env.setRestartStrategy(RestartStrategies.noRestart)
> val p = new java.util.Properties
> Map("bootstrap.servers" -> "192.168.1.100:9092", "transaction.timeout.ms"
> -> (10 * 60 * 1000).toString).foreach { case (k,v) => p.setProperty(k,v) }
> env
> .fromCollection(100 to 200)
> .map(_.toString)
> .addSink(new FlinkKafkaProducer011(
> "test",
> new KeyedSerializationSchemaWrapper(new SimpleStringSchema),
> p,
> Semantic.EXACTLY_ONCE)).name("100%")
> env.execute()
> }
> }
> {code}
> Raised exception:
> {code}
> 2018-10-02 17:00:12.918 [Map -> Sink: 100% (1/8)] INFO
> o.a.flink.runtime.taskmanager.Task - Map -> Sink: 100% (1/8)
> (25190aeccdce738afdd00e9320903d7b) switched from RUNNING to FAILED.
> java.util.MissingFormatWidthException: %-%
> at java.util.Formatter$FormatSpecifier.checkText(Formatter.java:3040)
> at java.util.Formatter$FormatSpecifier.(Formatter.java:2733)
> at java.util.Formatter.parse(Formatter.java:2560)
> at java.util.Formatter.format(Formatter.java:2501)
> at java.util.Formatter.format(Formatter.java:2455)
> at java.lang.String.format(String.java:2940)
> at
> org.apache.flink.streaming.connectors.kafka.internal.TransactionalIdsGenerator.generateTransactionalId(TransactionalIdsGenerator.java:91)
> at
> org.apache.flink.streaming.connectors.kafka.internal.TransactionalIdsGenerator.generateIdsToUse(TransactionalIdsGenerator.java:72)
> at
> org.apache.flink.streaming.connectors.kafka.internal.TransactionalIdsGenerator.generateIdsToAbort(TransactionalIdsGenerator.java:85)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:850)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10477) Add rescale button to web interface
[ https://issues.apache.org/jira/browse/FLINK-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned FLINK-10477: Assignee: vinoyang > Add rescale button to web interface > --- > > Key: FLINK-10477 > URL: https://issues.apache.org/jira/browse/FLINK-10477 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: Sander Ploegsma >Assignee: vinoyang >Priority: Minor > > Instead of having to use the REST API to rescale a running job, it would be > much easier if we were able to rescale a job from the web interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637149#comment-16637149
]
ASF GitHub Bot commented on FLINK-10292:
yanghua commented on issue #6786: [FLINK-10292][rest] Separate REST and
Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#issuecomment-426703943
hi @zentol, It seems like this PR should be linked to
[FLINK-10282](https://issues.apache.org/jira/browse/FLINK-10282)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools
yanghua commented on issue #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools URL: https://github.com/apache/flink/pull/6786#issuecomment-426703943 hi @zentol, It seems like this PR should be linked to [FLINK-10282](https://issues.apache.org/jira/browse/FLINK-10282)? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10340) Add Cosh math function supported in Table API and SQL
[
https://issues.apache.org/jira/browse/FLINK-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637141#comment-16637141
]
ASF GitHub Bot commented on FLINK-10340:
yanghua commented on a change in pull request #6700: [FLINK-10340][table] Add
Cosh math function supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6700#discussion_r222377224
##
File path: docs/dev/table/functions.md
##
@@ -1274,6 +1274,17 @@ ATAN2(numeric1, numeric2)
+
+
+{% highlight text %}
+COSH(numeric)
+{% endhighlight %}
+
+
+Returns the hyperbolic cosine of NUMERIC. Return value
type is DOUBLE.
Review comment:
I have updated this PR based on your suggestions. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Cosh math function supported in Table API and SQL
> -
>
> Key: FLINK-10340
> URL: https://issues.apache.org/jira/browse/FLINK-10340
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Affects Versions: 1.6.0
>Reporter: Sergey Tsvetkov
>Assignee: vinoyang
>Priority: Minor
> Labels: pull-request-available
>
> Implement udf of cosh, just like in oracle
> [https://docs.oracle.com/cd/B28359_01/server.111/b28286/functions031.htm#SQLRF00623]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on a change in pull request #6700: [FLINK-10340][table] Add Cosh math function supported in Table API and SQL
yanghua commented on a change in pull request #6700: [FLINK-10340][table] Add
Cosh math function supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6700#discussion_r222377224
##
File path: docs/dev/table/functions.md
##
@@ -1274,6 +1274,17 @@ ATAN2(numeric1, numeric2)
+
+
+{% highlight text %}
+COSH(numeric)
+{% endhighlight %}
+
+
+Returns the hyperbolic cosine of NUMERIC. Return value
type is DOUBLE.
Review comment:
I have updated this PR based on your suggestions. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-8799) Make AbstractYarnClusterDescriptor immutable
[
https://issues.apache.org/jira/browse/FLINK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637131#comment-16637131
]
ASF GitHub Bot commented on FLINK-8799:
---
yanghua commented on issue #5617: [FLINK-8799][YARN] Make
AbstractYarnClusterDescriptor immutable
URL: https://github.com/apache/flink/pull/5617#issuecomment-426698448
@GJL is right, the JIRA issue should not been closed, I can not fix this
issue, so I closed my PR and released this issue. However, this action just
make others feel free to fix it. So we can not close the JIRA issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make AbstractYarnClusterDescriptor immutable
>
>
> Key: FLINK-8799
> URL: https://issues.apache.org/jira/browse/FLINK-8799
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{AbstractYarnClusterDescriptor}} should be made immutable. Currently, its
> internal configuration is modified from different places which makes it
> difficult to reason about the code. For example, it should not be possible to
> modify the {{zookeeperNamespace}} using a setter method. A user of this class
> should be forced to provide all information prior to creating the instance,
> e.g., by passing a {{org.apache.flink.configuration.Configuration}} object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #5617: [FLINK-8799][YARN] Make AbstractYarnClusterDescriptor immutable
yanghua commented on issue #5617: [FLINK-8799][YARN] Make AbstractYarnClusterDescriptor immutable URL: https://github.com/apache/flink/pull/5617#issuecomment-426698448 @GJL is right, the JIRA issue should not been closed, I can not fix this issue, so I closed my PR and released this issue. However, this action just make others feel free to fix it. So we can not close the JIRA issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9990) Add regexp_extract supported in TableAPI and SQL
[
https://issues.apache.org/jira/browse/FLINK-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637124#comment-16637124
]
ASF GitHub Bot commented on FLINK-9990:
---
yanghua commented on issue #6448: [FLINK-9990] [table] Add regexp_extract
supported in TableAPI and SQL
URL: https://github.com/apache/flink/pull/6448#issuecomment-426697438
@xccui Thanks for reviewing again, I have updated this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add regexp_extract supported in TableAPI and SQL
>
>
> Key: FLINK-9990
> URL: https://issues.apache.org/jira/browse/FLINK-9990
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Minor
> Labels: pull-request-available
>
> regex_extract is a very useful function, it returns a string based on a regex
> pattern and a index.
> For example :
> {code:java}
> regexp_extract('foothebar', 'foo(.*?)(bar)', 2) // returns 'bar.'
> {code}
> It is provided as a UDF in Hive, more details please see[1].
> [1]: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #6448: [FLINK-9990] [table] Add regexp_extract supported in TableAPI and SQL
yanghua commented on issue #6448: [FLINK-9990] [table] Add regexp_extract supported in TableAPI and SQL URL: https://github.com/apache/flink/pull/6448#issuecomment-426697438 @xccui Thanks for reviewing again, I have updated this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-8799) Make AbstractYarnClusterDescriptor immutable
[
https://issues.apache.org/jira/browse/FLINK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637078#comment-16637078
]
ASF GitHub Bot commented on FLINK-8799:
---
GJL commented on issue #5617: [FLINK-8799][YARN] Make
AbstractYarnClusterDescriptor immutable
URL: https://github.com/apache/flink/pull/5617#issuecomment-426678446
@isunjin Why? The issue is not resolved yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make AbstractYarnClusterDescriptor immutable
>
>
> Key: FLINK-8799
> URL: https://issues.apache.org/jira/browse/FLINK-8799
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{AbstractYarnClusterDescriptor}} should be made immutable. Currently, its
> internal configuration is modified from different places which makes it
> difficult to reason about the code. For example, it should not be possible to
> modify the {{zookeeperNamespace}} using a setter method. A user of this class
> should be forced to provide all information prior to creating the instance,
> e.g., by passing a {{org.apache.flink.configuration.Configuration}} object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] GJL commented on issue #5617: [FLINK-8799][YARN] Make AbstractYarnClusterDescriptor immutable
GJL commented on issue #5617: [FLINK-8799][YARN] Make AbstractYarnClusterDescriptor immutable URL: https://github.com/apache/flink/pull/5617#issuecomment-426678446 @isunjin Why? The issue is not resolved yet. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10486) Recover local mode on flink startup
Ismaël Mejía created FLINK-10486: Summary: Recover local mode on flink startup Key: FLINK-10486 URL: https://issues.apache.org/jira/browse/FLINK-10486 Project: Flink Issue Type: Improvement Components: Cluster Management Reporter: Ismaël Mejía Assignee: Ismaël Mejía After the move to FLIP-6 Flink lost the local mode execution option. This has been a common use of the docker flink images for development purposes. We need to start a Dispatcher, ResourceManager and TaskExecutor. But unlike the old local mode, they should still expose their ports to the outside world. This can be fixed via a bash script or a special entrypoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9126) Creation of the CassandraPojoInputFormat class to output data into a Custom Cassandra Annotated Pojo
[
https://issues.apache.org/jira/browse/FLINK-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636986#comment-16636986
]
ASF GitHub Bot commented on FLINK-9126:
---
bmeriaux commented on a change in pull request #6735: [FLINK-9126] New
CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
URL: https://github.com/apache/flink/pull/6735#discussion_r222314129
##
File path:
flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java
##
@@ -382,31 +383,31 @@ public void testCassandraCommitter() throws Exception {
@Test
public void testCassandraTupleAtLeastOnceSink() throws Exception {
CassandraTupleSink> sink = new
CassandraTupleSink<>(injectTableName(INSERT_DATA_QUERY), builder);
-
- sink.open(new Configuration());
-
- for (Tuple3 value : collection) {
- sink.send(value);
+ try {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Creation of the CassandraPojoInputFormat class to output data into a Custom
> Cassandra Annotated Pojo
>
>
> Key: FLINK-9126
> URL: https://issues.apache.org/jira/browse/FLINK-9126
> Project: Flink
> Issue Type: New Feature
> Components: DataSet API
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Jeffrey Carter
>Assignee: Jeffrey Carter
>Priority: Minor
> Labels: InputFormat, cassandra, features, pull-request-available
> Fix For: 1.7.0
>
> Attachments: CassandraPojoInputFormatText.rtf
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently the DataSet API only has the ability to output data received from
> Cassandra as a source in as a Tuple. This would be allow the data to be
> output as a custom POJO that the user has created that has been annotated
> using Datastax API. This would remove the need of very long Tuples to be
> created by the DataSet and then mapped to the custom POJO.
>
> -The changes to the CassandraInputFormat object would be minimal, but would
> require importing the Datastax API into the class-. Another option is to make
> a similar, but slightly different class called CassandraPojoInputFormat.
> I have already gotten code for this working in my own project, but want other
> thoughts as to the best way this should go about being implemented.
>
> //Example of its use in main
> CassandraPojoInputFormat cassandraInputFormat = new
> CassandraPojoInputFormat<>(queryToRun, defaultClusterBuilder,
> CustomCassandraPojo.class);
> cassandraInputFormat.configure(null);
> cassandraInputFormat.open(null);
> DataSet outputTestSet =
> exEnv.createInput(cassandraInputFormat, TypeInformation.of(new
> TypeHint(){}));
>
> //The class that I currently have set up
> [^CassandraPojoInputFormatText.rtf]
>
> Will make another Jira Issue for the Output version next if this is approved
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] bmeriaux commented on a change in pull request #6735: [FLINK-9126] New CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
bmeriaux commented on a change in pull request #6735: [FLINK-9126] New
CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
URL: https://github.com/apache/flink/pull/6735#discussion_r222314129
##
File path:
flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java
##
@@ -382,31 +383,31 @@ public void testCassandraCommitter() throws Exception {
@Test
public void testCassandraTupleAtLeastOnceSink() throws Exception {
CassandraTupleSink> sink = new
CassandraTupleSink<>(injectTableName(INSERT_DATA_QUERY), builder);
-
- sink.open(new Configuration());
-
- for (Tuple3 value : collection) {
- sink.send(value);
+ try {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-9126) Creation of the CassandraPojoInputFormat class to output data into a Custom Cassandra Annotated Pojo
[
https://issues.apache.org/jira/browse/FLINK-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636985#comment-16636985
]
ASF GitHub Bot commented on FLINK-9126:
---
bmeriaux commented on a change in pull request #6735: [FLINK-9126] New
CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
URL: https://github.com/apache/flink/pull/6735#discussion_r222314062
##
File path:
flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/batch/connectors/cassandra/CassandraOutputFormatBase.java
##
@@ -41,7 +41,7 @@
* @param Type of the elements to write.
*/
public abstract class CassandraOutputFormatBase extends
RichOutputFormat {
- private static final Logger LOG =
LoggerFactory.getLogger(CassandraOutputFormatBase.class);
+ private final Logger logger = LoggerFactory.getLogger(getClass());
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Creation of the CassandraPojoInputFormat class to output data into a Custom
> Cassandra Annotated Pojo
>
>
> Key: FLINK-9126
> URL: https://issues.apache.org/jira/browse/FLINK-9126
> Project: Flink
> Issue Type: New Feature
> Components: DataSet API
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Jeffrey Carter
>Assignee: Jeffrey Carter
>Priority: Minor
> Labels: InputFormat, cassandra, features, pull-request-available
> Fix For: 1.7.0
>
> Attachments: CassandraPojoInputFormatText.rtf
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently the DataSet API only has the ability to output data received from
> Cassandra as a source in as a Tuple. This would be allow the data to be
> output as a custom POJO that the user has created that has been annotated
> using Datastax API. This would remove the need of very long Tuples to be
> created by the DataSet and then mapped to the custom POJO.
>
> -The changes to the CassandraInputFormat object would be minimal, but would
> require importing the Datastax API into the class-. Another option is to make
> a similar, but slightly different class called CassandraPojoInputFormat.
> I have already gotten code for this working in my own project, but want other
> thoughts as to the best way this should go about being implemented.
>
> //Example of its use in main
> CassandraPojoInputFormat cassandraInputFormat = new
> CassandraPojoInputFormat<>(queryToRun, defaultClusterBuilder,
> CustomCassandraPojo.class);
> cassandraInputFormat.configure(null);
> cassandraInputFormat.open(null);
> DataSet outputTestSet =
> exEnv.createInput(cassandraInputFormat, TypeInformation.of(new
> TypeHint(){}));
>
> //The class that I currently have set up
> [^CassandraPojoInputFormatText.rtf]
>
> Will make another Jira Issue for the Output version next if this is approved
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] bmeriaux commented on a change in pull request #6735: [FLINK-9126] New CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
bmeriaux commented on a change in pull request #6735: [FLINK-9126] New
CassandraPojoInputFormat to output data as a custom annotated Cassandra Pojo
URL: https://github.com/apache/flink/pull/6735#discussion_r222314062
##
File path:
flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/batch/connectors/cassandra/CassandraOutputFormatBase.java
##
@@ -41,7 +41,7 @@
* @param Type of the elements to write.
*/
public abstract class CassandraOutputFormatBase extends
RichOutputFormat {
- private static final Logger LOG =
LoggerFactory.getLogger(CassandraOutputFormatBase.class);
+ private final Logger logger = LoggerFactory.getLogger(getClass());
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308614
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
It would be the `WebMonitorEndpoint` task to do so. The contract would be
that the `WebMonitorEndpoint` is responsible for managing this executor.
In general, it is a good idea to not create things in the constructor of a
component, because it makes it really hard to reuse it in a different
environment. Instead, it is imo better to pass in instances of some interface.
That way, we don't couple components.
Even if we don't use it for testing right now, we should not remove
functionality which allowed it in the first place.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636960#comment-16636960
]
ASF GitHub Bot commented on FLINK-10292:
tillrohrmann commented on issue #6786: [FLINK-10292][rest] Separate REST and
Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#issuecomment-426639274
Thanks for addressing some of my components @zentol. I think we should still
change how the `ExecutorService` is instantiated. Instead of doing it in the
constructor it should happen imo outside of the component. That way, we don't
couple different components (e.g. which `ExecutorService` is used by the
`WebMonitorEndpoint`). Moreover, it was already like this before and, thus, we
would introduce a regression (in terms of architecture).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on issue #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools
tillrohrmann commented on issue #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools URL: https://github.com/apache/flink/pull/6786#issuecomment-426639274 Thanks for addressing some of my components @zentol. I think we should still change how the `ExecutorService` is instantiated. Instead of doing it in the constructor it should happen imo outside of the component. That way, we don't couple different components (e.g. which `ExecutorService` is used by the `WebMonitorEndpoint`). Moreover, it was already like this before and, thus, we would introduce a regression (in terms of architecture). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308964
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
In general, I think the REST API is less important than the cluster
components. Bumping the priority of the `Dispatcher` could also be a solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636956#comment-16636956
]
ASF GitHub Bot commented on FLINK-10292:
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308614
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
It would be the `WebMonitorEndpoint` task to do so. The contract would be
that the `WebMonitorEndpoint` is responsible for managing this executor.
In general, it is a good idea to not create things in the constructor of a
component, because it makes it really hard to reuse it in a different
environment. Instead, it is imo better to pass in instances of some interface.
That way, we don't couple components.
Even if we don't use it for testing right now, we should not remove
functionality which allowed it in the first place.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636952#comment-16636952
]
ASF GitHub Bot commented on FLINK-10292:
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308964
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
In general, I think the REST API is less important than the cluster
components. Bumping the priority of the `Dispatcher` could also be a solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636950#comment-16636950
]
ASF GitHub Bot commented on FLINK-10292:
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308614
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
It would be the `WebMonitorEndpoint` task to do so. The contract would be
that the `WebMonitorEndpoint` is responsible for managing this executor.
In general, it is a good idea to not create things in the constructor of a
component, because it makes it really hard to reuse it in a different
environment. Instead, it is imo better to pass in instances of some interface.
That way, we don't couple components.
Even if we don't do it right now, we should not remove functionality which
allowed it in the first place.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest] Separate REST and Dispatcher RPC thread pools
tillrohrmann commented on a change in pull request #6786: [FLINK-10292][rest]
Separate REST and Dispatcher RPC thread pools
URL: https://github.com/apache/flink/pull/6786#discussion_r222308614
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -180,7 +183,7 @@ public WebMonitorEndpoint(
this.restConfiguration =
Preconditions.checkNotNull(restConfiguration);
this.resourceManagerRetriever =
Preconditions.checkNotNull(resourceManagerRetriever);
this.transientBlobService =
Preconditions.checkNotNull(transientBlobService);
- this.executor = Preconditions.checkNotNull(executor);
+ this.executor =
Executors.newFixedThreadPool(endpointConfiguration.getNumThreads(), new
ExecutorThreadFactory("Flink-" + getClass().getSimpleName()));
Review comment:
It would be the `WebMonitorEndpoint` task to do so. The contract would be
that the `WebMonitorEndpoint` is responsible for managing this executor.
In general, it is a good idea to not create things in the constructor of a
component, because it makes it really hard to reuse it in a different
environment. Instead, it is imo better to pass in instances of some interface.
That way, we don't couple components.
Even if we don't do it right now, we should not remove functionality which
allowed it in the first place.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10354) Savepoints should be counted as retained checkpoints
[ https://issues.apache.org/jira/browse/FLINK-10354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636942#comment-16636942 ] ASF GitHub Bot commented on FLINK-10354: tillrohrmann commented on issue #6704: [FLINK-10354] Revert "[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore" URL: https://github.com/apache/flink/pull/6704#issuecomment-426635766 @aljoscha I'm not aware of any changes in the meantime which could break things. I think this is the task of the reviewer to guarantee that this is not the case. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Savepoints should be counted as retained checkpoints > > > Key: FLINK-10354 > URL: https://issues.apache.org/jira/browse/FLINK-10354 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > This task is about reverting [FLINK-6328]. > The problem is that you can get incorrect results with exactly-once sinks if > there is a failure after taking a savepoint but before taking the next > checkpoint because the savepoint will also have manifested side effects to > the sink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tillrohrmann commented on issue #6704: [FLINK-10354] Revert "[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore"
tillrohrmann commented on issue #6704: [FLINK-10354] Revert "[FLINK-6328] [chkPts] Don't add savepoints to CompletedCheckpointStore" URL: https://github.com/apache/flink/pull/6704#issuecomment-426635766 @aljoscha I'm not aware of any changes in the meantime which could break things. I think this is the task of the reviewer to guarantee that this is not the case. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10474) Don't translate IN to JOIN with VALUES for streaming queries
[ https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636930#comment-16636930 ] Fabian Hueske commented on FLINK-10474: --- Oh. I had assumed that the conversion happens in a rule during optimization. Sorry for the confusion. +1 for your approach! > Don't translate IN to JOIN with VALUES for streaming queries > > > Key: FLINK-10474 > URL: https://issues.apache.org/jira/browse/FLINK-10474 > Project: Flink > Issue Type: Improvement > Components: Table API SQL >Affects Versions: 1.6.1, 1.7.0 >Reporter: Fabian Hueske >Assignee: Hequn Cheng >Priority: Major > > IN clauses are translated to JOIN with VALUES if the number of elements in > the IN clause exceeds a certain threshold. This should not be done, because a > streaming join is very heavy and materializes both inputs (which is fine for > the VALUES) input but not for the other. > There are two ways to solve this: > # don't translate IN to a JOIN at all > # translate it to a JOIN but have a special join strategy if one input is > bound and final (non-updating) > Option 1. should be easy to do, option 2. requires much more effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10371) Allow to enable SSL mutual authentication on REST endpoints by configuration
[ https://issues.apache.org/jira/browse/FLINK-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636925#comment-16636925 ] ASF GitHub Bot commented on FLINK-10371: asfgit closed pull request #6727: [FLINK-10371] Allow to enable SSL mutual authentication on REST endpoints by configuration URL: https://github.com/apache/flink/pull/6727 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/_includes/generated/security_configuration.html b/docs/_includes/generated/security_configuration.html index 19d0287df99..680c1c02434 100644 --- a/docs/_includes/generated/security_configuration.html +++ b/docs/_includes/generated/security_configuration.html @@ -62,6 +62,11 @@ "TLSv1.2" The SSL protocol version to be supported for the ssl transport. Note that it doesn’t support comma separated list. + +security.ssl.rest.authentication-enabled +false +Turns on mutual SSL authentication for external communication via the REST endpoints. + security.ssl.rest.enabled false diff --git a/docs/ops/security-ssl.md b/docs/ops/security-ssl.md index ed5f4d771bc..6ea686203ee 100644 --- a/docs/ops/security-ssl.md +++ b/docs/ops/security-ssl.md @@ -43,11 +43,11 @@ Internal connectivity includes: - Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager - The data plane: The connections between TaskManagers to exchange data during shuffles, broadcasts, redistribution, etc. - - The Blob Service (distribution of libraries and other artifacts). + - The Blob Service (distribution of libraries and other artifacts). All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**, meaning both server and client side of each connection need to present the certificate to each other. The certificate acts effectively as a shared -secret. +secret. A common setup is to generate a dedicated certificate (may be self-signed) for a Flink deployment. The certificate for internal communication is not needed by any other party to interact with Flink, and can be simply added to the container images, or attached to the YARN deployment. @@ -61,15 +61,14 @@ All external connectivity is exposed via an HTTP/REST endpoint, used for example - Communication with the *Dispatcher* to submit jobs (session clusters) - Communication with the *JobManager* to inspect and modify a running job/application -The REST endpoints can be configured to require SSL connections. The server will, however, accept connections from any client, meaning the REST endpoint does not authenticate the client. +The REST endpoints can be configured to require SSL connections. The server will, however, accept connections from any client by default, meaning the REST endpoint does not authenticate the client. -If authentication of connections to the REST endpoint is required, we recommend to deploy a "side car proxy": +Simple mutual authentication may be enabled by configuration if authentication of connections to the REST endpoint is required, but we recommend to deploy a "side car proxy": Bind the REST endpoint to the loopback interface (or the pod-local interface in Kubernetes) and start a REST proxy that authenticates and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/) or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html). -The rationale behind delegating authentication to a proxy is that such proxies offer many more authentication options than the Flink project could reasonably implement itself, -and thus offer better integration into existing infrastructures. +The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety of authentication options and thus better integration into existing infrastructures. Queryable State @@ -115,10 +114,12 @@ security.ssl.internal.truststore-password: truststore_password **REST Endpoints (external connectivity)** -For REST endpoints, the keystore is used by the server endpoint, and the truststore is used by the REST clients (including the CLI client) +For REST endpoints, by default the keystore is used by the server endpoint, and the truststore is used by the REST clients (including the CLI client) to accept the server's certificate. In the case where the REST keystore has a self-signed certificate, the truststore must trust that certificate directly. If the REST endpoint uses a certificate that is signed through a
[GitHub] asfgit closed pull request #6727: [FLINK-10371] Allow to enable SSL mutual authentication on REST endpoints by configuration
asfgit closed pull request #6727: [FLINK-10371] Allow to enable SSL mutual authentication on REST endpoints by configuration URL: https://github.com/apache/flink/pull/6727 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/_includes/generated/security_configuration.html b/docs/_includes/generated/security_configuration.html index 19d0287df99..680c1c02434 100644 --- a/docs/_includes/generated/security_configuration.html +++ b/docs/_includes/generated/security_configuration.html @@ -62,6 +62,11 @@ "TLSv1.2" The SSL protocol version to be supported for the ssl transport. Note that it doesn’t support comma separated list. + +security.ssl.rest.authentication-enabled +false +Turns on mutual SSL authentication for external communication via the REST endpoints. + security.ssl.rest.enabled false diff --git a/docs/ops/security-ssl.md b/docs/ops/security-ssl.md index ed5f4d771bc..6ea686203ee 100644 --- a/docs/ops/security-ssl.md +++ b/docs/ops/security-ssl.md @@ -43,11 +43,11 @@ Internal connectivity includes: - Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager - The data plane: The connections between TaskManagers to exchange data during shuffles, broadcasts, redistribution, etc. - - The Blob Service (distribution of libraries and other artifacts). + - The Blob Service (distribution of libraries and other artifacts). All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**, meaning both server and client side of each connection need to present the certificate to each other. The certificate acts effectively as a shared -secret. +secret. A common setup is to generate a dedicated certificate (may be self-signed) for a Flink deployment. The certificate for internal communication is not needed by any other party to interact with Flink, and can be simply added to the container images, or attached to the YARN deployment. @@ -61,15 +61,14 @@ All external connectivity is exposed via an HTTP/REST endpoint, used for example - Communication with the *Dispatcher* to submit jobs (session clusters) - Communication with the *JobManager* to inspect and modify a running job/application -The REST endpoints can be configured to require SSL connections. The server will, however, accept connections from any client, meaning the REST endpoint does not authenticate the client. +The REST endpoints can be configured to require SSL connections. The server will, however, accept connections from any client by default, meaning the REST endpoint does not authenticate the client. -If authentication of connections to the REST endpoint is required, we recommend to deploy a "side car proxy": +Simple mutual authentication may be enabled by configuration if authentication of connections to the REST endpoint is required, but we recommend to deploy a "side car proxy": Bind the REST endpoint to the loopback interface (or the pod-local interface in Kubernetes) and start a REST proxy that authenticates and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/) or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html). -The rationale behind delegating authentication to a proxy is that such proxies offer many more authentication options than the Flink project could reasonably implement itself, -and thus offer better integration into existing infrastructures. +The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety of authentication options and thus better integration into existing infrastructures. Queryable State @@ -115,10 +114,12 @@ security.ssl.internal.truststore-password: truststore_password **REST Endpoints (external connectivity)** -For REST endpoints, the keystore is used by the server endpoint, and the truststore is used by the REST clients (including the CLI client) +For REST endpoints, by default the keystore is used by the server endpoint, and the truststore is used by the REST clients (including the CLI client) to accept the server's certificate. In the case where the REST keystore has a self-signed certificate, the truststore must trust that certificate directly. If the REST endpoint uses a certificate that is signed through a proper certification hierarchy, the roots of that hierarchy should -be in the trust store. +be in the trust store. + +If mutual authentication is enabled, the keystore and the truststore are used by both, the server endpoint and the REST
[jira] [Commented] (FLINK-10484) New latency tracking metrics format causes metrics cardinality explosion
[ https://issues.apache.org/jira/browse/FLINK-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636850#comment-16636850 ] Chesnay Schepler commented on FLINK-10484: -- In FLINK-10243 we introduced a switch to reduce the amount of data for the latency source, see https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#metrics-latency-granularity. This can be used to drastically reduce the number of latency metrics. We could look into back-porting this. The "cardinality explosion" is caused by introducing proper support for custom tags, which we used here for consistency purposes as it was always a bit odd that you only had a tag for the receiving operator ID, but not the source. The issue of effectively uncontrollable tags (since they're unaffected by scope formats) was raised before, like in FLINK-7935, but I haven't found time to address it as it requires a more thorough rework of the internals. All the tag-based scope goodies were pretty much tacked on after the fact, and now things are scattered all over the place :( > New latency tracking metrics format causes metrics cardinality explosion > > > Key: FLINK-10484 > URL: https://issues.apache.org/jira/browse/FLINK-10484 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.6.0, 1.6.1, 1.5.4 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Critical > > The new metrics format for latency tracking causes huge metrics cardinality > explosion due to the format and the fact that there is a metric reported for > a every combination of source subtask index and operator subtask index. > Yikes! > This format is actually responsible for basically taking down our metrics > system due to DDOSing our metrics servers (at Lyft). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10474) Don't translate IN to JOIN with VALUES for streaming queries
[ https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636841#comment-16636841 ] Hequn Cheng commented on FLINK-10474: - Hi [~fhueske] , I guess I miss something? The IN is translated during SqlToRelConverter in Calcite before optimization. It will be converted into a join or a OR predicate. I think we can force usage of OR in all cases. I plan to add a rule to convert these predicates back to IN, for example, convert predicate: (x = 1 OR x = 2 OR x = 3) AND y = 4 back to predicate: x IN (1, 2, 3) AND y = 4. We can generate code using a HashSet for the IN. > Don't translate IN to JOIN with VALUES for streaming queries > > > Key: FLINK-10474 > URL: https://issues.apache.org/jira/browse/FLINK-10474 > Project: Flink > Issue Type: Improvement > Components: Table API SQL >Affects Versions: 1.6.1, 1.7.0 >Reporter: Fabian Hueske >Assignee: Hequn Cheng >Priority: Major > > IN clauses are translated to JOIN with VALUES if the number of elements in > the IN clause exceeds a certain threshold. This should not be done, because a > streaming join is very heavy and materializes both inputs (which is fine for > the VALUES) input but not for the other. > There are two ways to solve this: > # don't translate IN to a JOIN at all > # translate it to a JOIN but have a special join strategy if one input is > bound and final (non-updating) > Option 1. should be easy to do, option 2. requires much more effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10484) New latency tracking metrics format causes metrics cardinality explosion
[ https://issues.apache.org/jira/browse/FLINK-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636838#comment-16636838 ] Aljoscha Krettek commented on FLINK-10484: -- Hi [~jgrier] :) There is this issue and a bunch of other metrics-related issues that are being worked on for 1.7 (and also as bugfixes for earlier versions): FLINK-10246 > New latency tracking metrics format causes metrics cardinality explosion > > > Key: FLINK-10484 > URL: https://issues.apache.org/jira/browse/FLINK-10484 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.6.0, 1.6.1, 1.5.4 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Critical > > The new metrics format for latency tracking causes huge metrics cardinality > explosion due to the format and the fact that there is a metric reported for > a every combination of source subtask index and operator subtask index. > Yikes! > This format is actually responsible for basically taking down our metrics > system due to DDOSing our metrics servers (at Lyft). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10422) Follow AWS specs in Kinesis Consumer
[
https://issues.apache.org/jira/browse/FLINK-10422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636775#comment-16636775
]
eugen yushin commented on FLINK-10422:
--
[~tzulitai] looks like you've contributed to Kinesis connector a much, can you
please check proposed changes or let me know who is the better person to ask
for a Code Review?
Regards
> Follow AWS specs in Kinesis Consumer
> -
>
> Key: FLINK-10422
> URL: https://issues.apache.org/jira/browse/FLINK-10422
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.6.1
>Reporter: eugen yushin
>Assignee: eugen yushin
>Priority: Major
> Labels: pull-request-available
>
> *Related conversation in mailing list:*
> [https://lists.apache.org/thread.html/96de3bac9761564767cf283b58d664f5ae1b076e0c4431620552af5b@%3Cdev.flink.apache.org%3E]
> *Summary:*
> Flink Kinesis consumer checks shards id for a particular pattern:
> {noformat}
> "^shardId-\\d{12}"
> {noformat}
> [https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/model/StreamShardHandle.java#L132]
> While this inlines with current Kinesis streams server implementation (all
> streams follows this pattern), it confronts with AWS docs:
>
> {code:java}
> ShardId
> The unique identifier of the shard within the stream.
> Type: String
> Length Constraints: Minimum length of 1. Maximum length of 128.
> Pattern: [a-zA-Z0-9_.-]+
> Required: Yes
> {code}
>
> [https://docs.aws.amazon.com/kinesis/latest/APIReference/API_Shard.html]
> *Intention:*
> We have no guarantees and can't rely on patterns other than provided in AWS
> manifest.
> Any custom implementation of Kinesis mock should rely on AWS manifest which
> claims ShardID to be alfanums. This prevents anyone to use Flink with such
> kind of mocks.
> The reason behind the scene to use particular pattern "^shardId-d12" is to
> create Flink's custom Shard comparator, filter already seen shards, and pass
> latest shard for client.listShards only to limit the scope for RPC call to
> AWS.
> In the meantime, I think we can get rid of this logic at all. The current
> usage in project is:
> - fix Kinesalite bug (I've already opened an issue to cover this:
> [https://github.com/mhart/kinesalite/issues/76] and opened PR:
> [https://github.com/mhart/kinesalite/pull/77]). We can move this logic to
> test code base to keep production code clean for now
>
> [https://github.com/apache/flink/blob/50d076ab6ad325907690a2c115ee2cb1c45775c9/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L464]
> - adjust last seen shard id. We can simply omit this cause' AWS client won't
> return already seen shards and we will have new ids only or nothing.
> [https://github.com/apache/flink/blob/50d076ab6ad325907690a2c115ee2cb1c45775c9/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L475]
>
> [https://github.com/apache/flink/blob/50d076ab6ad325907690a2c115ee2cb1c45775c9/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L406]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10474) Don't translate IN to JOIN with VALUES for streaming queries
[ https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636726#comment-16636726 ] Fabian Hueske commented on FLINK-10474: --- Wouldn't it be easier to remove the rule that translates the IN? > Don't translate IN to JOIN with VALUES for streaming queries > > > Key: FLINK-10474 > URL: https://issues.apache.org/jira/browse/FLINK-10474 > Project: Flink > Issue Type: Improvement > Components: Table API SQL >Affects Versions: 1.6.1, 1.7.0 >Reporter: Fabian Hueske >Assignee: Hequn Cheng >Priority: Major > > IN clauses are translated to JOIN with VALUES if the number of elements in > the IN clause exceeds a certain threshold. This should not be done, because a > streaming join is very heavy and materializes both inputs (which is fine for > the VALUES) input but not for the other. > There are two ways to solve this: > # don't translate IN to a JOIN at all > # translate it to a JOIN but have a special join strategy if one input is > bound and final (non-updating) > Option 1. should be easy to do, option 2. requires much more effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7811) Add support for Scala 2.12
[
https://issues.apache.org/jira/browse/FLINK-7811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636646#comment-16636646
]
ASF GitHub Bot commented on FLINK-7811:
---
aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12
URL: https://github.com/apache/flink/pull/6784#issuecomment-426564163
@zentol Also, the only ASM-related imports are
```
import org.apache.xbean.asm6.{ClassReader, ClassVisitor, MethodVisitor, Type}
import org.apache.xbean.asm6.Opcodes._
```
which seem to be exactly the same as for our earlier asm5 dependency.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support for Scala 2.12
> --
>
> Key: FLINK-7811
> URL: https://issues.apache.org/jira/browse/FLINK-7811
> Project: Flink
> Issue Type: Sub-task
> Components: Scala API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10474) Don't translate IN to JOIN with VALUES for streaming queries
[
https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636645#comment-16636645
]
Hequn Cheng commented on FLINK-10474:
-
Hi, thanks for the discussion and ideas. I also prefer the first option.
I think we can override {{getInSubQueryThreshold}} in
{{SqlToRelConverter.Config}} to avoid converting to a join. And we can use a
more efficient HashSet during code gen.
> Don't translate IN to JOIN with VALUES for streaming queries
>
>
> Key: FLINK-10474
> URL: https://issues.apache.org/jira/browse/FLINK-10474
> Project: Flink
> Issue Type: Improvement
> Components: Table API SQL
>Affects Versions: 1.6.1, 1.7.0
>Reporter: Fabian Hueske
>Assignee: Hequn Cheng
>Priority: Major
>
> IN clauses are translated to JOIN with VALUES if the number of elements in
> the IN clause exceeds a certain threshold. This should not be done, because a
> streaming join is very heavy and materializes both inputs (which is fine for
> the VALUES) input but not for the other.
> There are two ways to solve this:
> # don't translate IN to a JOIN at all
> # translate it to a JOIN but have a special join strategy if one input is
> bound and final (non-updating)
> Option 1. should be easy to do, option 2. requires much more effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12
aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12
URL: https://github.com/apache/flink/pull/6784#issuecomment-426564163
@zentol Also, the only ASM-related imports are
```
import org.apache.xbean.asm6.{ClassReader, ClassVisitor, MethodVisitor, Type}
import org.apache.xbean.asm6.Opcodes._
```
which seem to be exactly the same as for our earlier asm5 dependency.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-7811) Add support for Scala 2.12
[ https://issues.apache.org/jira/browse/FLINK-7811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636642#comment-16636642 ] ASF GitHub Bot commented on FLINK-7811: --- aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12 URL: https://github.com/apache/flink/pull/6784#issuecomment-426563401 > API compatibility failed: > > ``` > 11:55:19.619 [ERROR] Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project flink-streaming-scala_2.11: Breaking the build because there is at least one incompatibility: > org.apache.flink.streaming.api.scala.DataStream$$anon$3.DataStream$$anon$3(org.apache.flink.streaming.api.scala.DataStream,org.apache.flink.api.common.typeinfo.TypeInformation,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$4.map(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$4.DataStream$$anon$4(org.apache.flink.streaming.api.scala.DataStream,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.api.common.functions.MapFunction[org.apache.flink.api.common.functions.MapFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$5.flatMap(java.lang.Object,org.apache.flink.util.Collector):METHOD_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$5.DataStream$$anon$5(org.apache.flink.streaming.api.scala.DataStream,scala.Function2):CONSTRUCTOR_REMOVED, > org.apache.flink.api.common.functions.FlatMapFunction[org.apache.flink.api.common.functions.FlatMapFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$6.DataStream$$anon$6(org.apache.flink.streaming.api.scala.DataStream,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$7.filter(java.lang.Object):METHOD_REMOVED, > org.apache.flink.api.common.functions.FilterFunction[org.apache.flink.api.common.functions.FilterFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$8.invoke(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.functions.sink.SinkFunction[org.apache.flink.streaming.api.functions.sink.SinkFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$9.select(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.collector.selector.OutputSelector[org.apache.flink.streaming.api.collector.selector.OutputSelector]:INTERFACE_REMOVED -> [Help 1] > ``` @zentol that was resolved by one of the last commits. The problem was that our exclusion pattern for `anon$$...` classes was not correct. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support for Scala 2.12 > -- > > Key: FLINK-7811 > URL: https://issues.apache.org/jira/browse/FLINK-7811 > Project: Flink > Issue Type: Sub-task > Components: Scala API >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12
aljoscha commented on issue #6784: [FLINK-7811] Add support for Scala 2.12 URL: https://github.com/apache/flink/pull/6784#issuecomment-426563401 > API compatibility failed: > > ``` > 11:55:19.619 [ERROR] Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project flink-streaming-scala_2.11: Breaking the build because there is at least one incompatibility: > org.apache.flink.streaming.api.scala.DataStream$$anon$3.DataStream$$anon$3(org.apache.flink.streaming.api.scala.DataStream,org.apache.flink.api.common.typeinfo.TypeInformation,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$4.map(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$4.DataStream$$anon$4(org.apache.flink.streaming.api.scala.DataStream,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.api.common.functions.MapFunction[org.apache.flink.api.common.functions.MapFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$5.flatMap(java.lang.Object,org.apache.flink.util.Collector):METHOD_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$5.DataStream$$anon$5(org.apache.flink.streaming.api.scala.DataStream,scala.Function2):CONSTRUCTOR_REMOVED, > org.apache.flink.api.common.functions.FlatMapFunction[org.apache.flink.api.common.functions.FlatMapFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$6.DataStream$$anon$6(org.apache.flink.streaming.api.scala.DataStream,scala.Function1):CONSTRUCTOR_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$7.filter(java.lang.Object):METHOD_REMOVED, > org.apache.flink.api.common.functions.FilterFunction[org.apache.flink.api.common.functions.FilterFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$8.invoke(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.functions.sink.SinkFunction[org.apache.flink.streaming.api.functions.sink.SinkFunction]:INTERFACE_REMOVED, > org.apache.flink.streaming.api.scala.DataStream$$anon$9.select(java.lang.Object):METHOD_REMOVED, > org.apache.flink.streaming.api.collector.selector.OutputSelector[org.apache.flink.streaming.api.collector.selector.OutputSelector]:INTERFACE_REMOVED -> [Help 1] > ``` @zentol that was resolved by one of the last commits. The problem was that our exclusion pattern for `anon$$...` classes was not correct. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
