[GitHub] Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function in Table API and SQL

[GitBox]

Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function 
in Table API and SQL
URL: https://github.com/apache/flink/pull/6773#issuecomment-427545460
 
 
   Hi, @yanghua thanks for your suggestion, I've add some examples in docs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10152) Add reverse supported in TableAPI and SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640557#comment-16640557
 ] 

ASF GitHub Bot commented on FLINK-10152:


Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function 
in Table API and SQL
URL: https://github.com/apache/flink/pull/6773#issuecomment-427545460
 
 
   Hi, @yanghua thanks for your suggestion, I've add some examples in docs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add reverse supported in TableAPI and SQL
> -
>
> Key: FLINK-10152
> URL: https://issues.apache.org/jira/browse/FLINK-10152
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Guibo Pan
>Assignee: Guibo Pan
>Priority: Major
>  Labels: pull-request-available
>
> reverse is useful function for procession String.
> for example:
> {code:java}
> select reverse("abcd") // return "dcba"
> {code}
> It is supported as a UDF in Hive, more details please see[1] [1]: 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10496) CommandLineParser arguments interleaving

[eugen yushin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

eugen yushin updated FLINK-10496:
-
Issue Type: Bug  (was: Improvement)

> CommandLineParser arguments interleaving
> 
>
> Key: FLINK-10496
> URL: https://issues.apache.org/jira/browse/FLINK-10496
> Project: Flink
>  Issue Type: Bug
>  Components: Configuration, Core, Docker, Java API
>Affects Versions: 1.6.1, 1.7.0
>Reporter: eugen yushin
>Assignee: eugen yushin
>Priority: Major
>  Labels: pull-request-available
>
> *Business case:*
> Run Flink job cluster within Docker/k8s. Job takes an argument called 
> `--config` which can't be recognized in runtime.
> {code:java}
> Caused by: java.lang.RuntimeException: No data for required key 'config'
> {code}
> *Problem statement:*
> Command line parser can't recognize job specific arguments when they have the 
> same prefix as Flink's ones.
> e.g.
> [https://github.com/apache/flink/blob/master/flink-container/src/test/java/org/apache/flink/container/entrypoint/StandaloneJobClusterConfigurationParserFactoryTest.java#L52]
> the following args results in failure:
> {code:java}
> final String arg1 = "--config";
> final String arg2 = "/path/to/job.yaml";{code}
> *Reason*:
> Apache CLI parser use string prefix matching to parse options and adds extra 
> --configDir to result set instead of adding new --config.
> https://github.com/apache/commons-cli/blob/cli-1.3.1/src/main/java/org/apache/commons/cli/DefaultParser.java#L391
> *Scope*:
> Update commons-cli dependency with version 1.4 which has flag to disable 
> partial matching.
> https://github.com/apache/commons-cli/commit/bdb4a09ceaceab7e3d214b1beadb93bd9c911342
> Update Flink's command line parser to utilize this feature.
> https://github.com/apache/flink/blob/6258a4c333ce9dba914621b13eac2f7d91f5cb72/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/parser/CommandLineParser.java#L45



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-10421:
---
Labels: pull-request-available test-stability  (was: test-stability)

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640191#comment-16640191
 ] 

ASF GitHub Bot commented on FLINK-10421:


tillrohrmann opened a new pull request #6800: [FLINK-10421][tests] Exclude 
FluentPropertyBeanIntrospector info message in common.sh
URL: https://github.com/apache/flink/pull/6800
 
 
   ## What is the purpose of the change
   
   FluentPropertyBeanIntrospector logs an info message containing the word 
error. This can cause
   end-to-end tests to fail. In order to prevent this, the respective logging 
statement is excluded.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] tillrohrmann opened a new pull request #6800: [FLINK-10421][tests] Exclude FluentPropertyBeanIntrospector info message in common.sh

[GitBox]

tillrohrmann opened a new pull request #6800: [FLINK-10421][tests] Exclude 
FluentPropertyBeanIntrospector info message in common.sh
URL: https://github.com/apache/flink/pull/6800
 
 
   ## What is the purpose of the change
   
   FluentPropertyBeanIntrospector logs an info message containing the word 
error. This can cause
   end-to-end tests to fail. In order to prevent this, the respective logging 
statement is excluded.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640188#comment-16640188
 ] 

ASF GitHub Bot commented on FLINK-10454:


tillrohrmann closed pull request #6799: [FLINK-10454][tests] Start MiniCluster 
with rest port 0
URL: https://github.com/apache/flink/pull/6799
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
index 6d58928d5e9..7782a8e2feb 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
@@ -20,6 +20,7 @@
 
 import org.apache.flink.configuration.AkkaOptions;
 import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.runtime.client.JobExecutionException;
 import org.apache.flink.runtime.execution.Environment;
 import org.apache.flink.runtime.io.network.api.reader.RecordReader;
@@ -41,6 +42,11 @@
 
 import java.util.BitSet;
 
+/**
+ * Tests that Flink can execute jobs with a higher parallelism than available 
number
+ * of slots. This effectively tests that Flink can execute jobs with blocking 
results
+ * in a staged fashion.
+ */
 public class SlotCountExceedingParallelismTest extends TestLogger {
 
// Test configuration
@@ -55,6 +61,7 @@
@BeforeClass
public static void setUp() throws Exception {
final Configuration config = new Configuration();
+   config.setInteger(RestOptions.PORT, 0);
config.setString(AkkaOptions.ASK_TIMEOUT, 
TestingUtils.DEFAULT_AKKA_ASK_TIMEOUT());
 
final MiniClusterConfiguration miniClusterConfiguration = new 
MiniClusterConfiguration.Builder()
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
index aee526dc153..eab4eabd6aa 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
@@ -20,6 +20,7 @@
 
 import org.apache.flink.configuration.AkkaOptions;
 import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.runtime.execution.Environment;
 import org.apache.flink.runtime.io.network.api.writer.RecordWriter;
 import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
@@ -44,6 +45,10 @@
 
 import static 
org.apache.flink.runtime.jobmanager.SlotCountExceedingParallelismTest.SubtaskIndexReceiver.CONFIG_KEY;
 
+/**
+ * Tests for the lazy scheduling/updating of consumers depending on the
+ * producers result.
+ */
 public class ScheduleOrUpdateConsumersTest extends TestLogger {
 
private static final int NUMBER_OF_TMS = 2;
@@ -55,6 +60,7 @@
@BeforeClass
public static void setUp() throws Exception {
final Configuration config = new Configuration();
+   config.setInteger(RestOptions.PORT, 0);
config.setString(AkkaOptions.ASK_TIMEOUT, 
TestingUtils.DEFAULT_AKKA_ASK_TIMEOUT());
 
final MiniClusterConfiguration miniClusterConfiguration = new 
MiniClusterConfiguration.Builder()
@@ -89,7 +95,7 @@ public static void tearDown() throws Exception {
 * +--+
 * 
 *
-* The pipelined receiver gets deployed after the first buffer is 
available and the blocking
+* The pipelined receiver gets deployed after the first buffer is 
available and the blocking
 * one after all subtasks are finished.
 */
@Test
@@ -136,9 +142,13 @@ public void testMixedPipelinedAndBlockingResults() throws 
Exception {
 
// 
-
 
+   /**
+* Invokable which writes a configurable number of events to a pipelined
+* and blocking partition alternatingly.
+*/
public static class BinaryRoundRobinSubtaskIndexSender extends 
AbstractInvokable {
 
-   public static final String CONFIG_KEY = 
"number-of-times-to-send";
+   static final String CONFIG_KEY = 

[jira] [Resolved] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-10454.
---
Resolution: Fixed

Fixed via 
https://github.com/apache/flink/commit/c7e0cdd1921df569e496a5c18f1b63ae247a050d

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] tillrohrmann closed pull request #6799: [FLINK-10454][tests] Start MiniCluster with rest port 0

[GitBox]

tillrohrmann closed pull request #6799: [FLINK-10454][tests] Start MiniCluster 
with rest port 0
URL: https://github.com/apache/flink/pull/6799
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
index 6d58928d5e9..7782a8e2feb 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/SlotCountExceedingParallelismTest.java
@@ -20,6 +20,7 @@
 
 import org.apache.flink.configuration.AkkaOptions;
 import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.runtime.client.JobExecutionException;
 import org.apache.flink.runtime.execution.Environment;
 import org.apache.flink.runtime.io.network.api.reader.RecordReader;
@@ -41,6 +42,11 @@
 
 import java.util.BitSet;
 
+/**
+ * Tests that Flink can execute jobs with a higher parallelism than available 
number
+ * of slots. This effectively tests that Flink can execute jobs with blocking 
results
+ * in a staged fashion.
+ */
 public class SlotCountExceedingParallelismTest extends TestLogger {
 
// Test configuration
@@ -55,6 +61,7 @@
@BeforeClass
public static void setUp() throws Exception {
final Configuration config = new Configuration();
+   config.setInteger(RestOptions.PORT, 0);
config.setString(AkkaOptions.ASK_TIMEOUT, 
TestingUtils.DEFAULT_AKKA_ASK_TIMEOUT());
 
final MiniClusterConfiguration miniClusterConfiguration = new 
MiniClusterConfiguration.Builder()
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
index aee526dc153..eab4eabd6aa 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleOrUpdateConsumersTest.java
@@ -20,6 +20,7 @@
 
 import org.apache.flink.configuration.AkkaOptions;
 import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.runtime.execution.Environment;
 import org.apache.flink.runtime.io.network.api.writer.RecordWriter;
 import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
@@ -44,6 +45,10 @@
 
 import static 
org.apache.flink.runtime.jobmanager.SlotCountExceedingParallelismTest.SubtaskIndexReceiver.CONFIG_KEY;
 
+/**
+ * Tests for the lazy scheduling/updating of consumers depending on the
+ * producers result.
+ */
 public class ScheduleOrUpdateConsumersTest extends TestLogger {
 
private static final int NUMBER_OF_TMS = 2;
@@ -55,6 +60,7 @@
@BeforeClass
public static void setUp() throws Exception {
final Configuration config = new Configuration();
+   config.setInteger(RestOptions.PORT, 0);
config.setString(AkkaOptions.ASK_TIMEOUT, 
TestingUtils.DEFAULT_AKKA_ASK_TIMEOUT());
 
final MiniClusterConfiguration miniClusterConfiguration = new 
MiniClusterConfiguration.Builder()
@@ -89,7 +95,7 @@ public static void tearDown() throws Exception {
 * +--+
 * 
 *
-* The pipelined receiver gets deployed after the first buffer is 
available and the blocking
+* The pipelined receiver gets deployed after the first buffer is 
available and the blocking
 * one after all subtasks are finished.
 */
@Test
@@ -136,9 +142,13 @@ public void testMixedPipelinedAndBlockingResults() throws 
Exception {
 
// 
-
 
+   /**
+* Invokable which writes a configurable number of events to a pipelined
+* and blocking partition alternatingly.
+*/
public static class BinaryRoundRobinSubtaskIndexSender extends 
AbstractInvokable {
 
-   public static final String CONFIG_KEY = 
"number-of-times-to-send";
+   static final String CONFIG_KEY = "number-of-times-to-send";
 
public BinaryRoundRobinSubtaskIndexSender(Environment 
environment) {
super(environment);


 


This is an automated 

[GitHub] azagrebin commented on issue #6793: [hotfix] [docs] Update cluster setup docs to reflect the new syntax of jobmanager.sh script.

[GitBox]

azagrebin commented on issue #6793: [hotfix] [docs] Update cluster setup docs 
to reflect the new syntax of jobmanager.sh script.
URL: https://github.com/apache/flink/pull/6793#issuecomment-427433251
 
 
   @markcho, one of committers will merge it if there is no more comments from 
them.
   cc @tillrohrmann 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10289) Classify Exceptions to different category for apply different failover strategy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640071#comment-16640071
 ] 

ASF GitHub Bot commented on FLINK-10289:


isunjin commented on issue #6739: [FLINK-10289] [JobManager] Classify 
Exceptions to different category for apply different failover strategy
URL: https://github.com/apache/flink/pull/6739#issuecomment-427431004
 
 
   @StefanRRichter Thank you, pushed the change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Classify Exceptions to different category for apply different failover 
> strategy
> ---
>
> Key: FLINK-10289
> URL: https://issues.apache.org/jira/browse/FLINK-10289
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: JIN SUN
>Assignee: JIN SUN
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> We need to classify exceptions and treat them with different strategies. To 
> do this, we propose to introduce the following Throwable Types, and the 
> corresponding exceptions:
>  * NonRecoverable
>  ** We shouldn’t retry if an exception was classified as NonRecoverable
>  ** For example, NoResouceAvailiableException is a NonRecoverable Exception
>  ** Introduce a new Exception UserCodeException to wrap all exceptions that 
> throw from user code
>  * PartitionDataMissingError
>  ** In certain scenarios producer data was transferred in blocking mode or 
> data was saved in persistent store. If the partition was missing, we need to 
> revoke/rerun the produce task to regenerate the data.
>  ** Introduce a new exception PartitionDataMissingException to wrap all those 
> kinds of issues.
>  * EnvironmentError
>  ** It happened due to hardware, or software issues that were related to 
> specific environments. The assumption is that a task will succeed if we run 
> it in a different environment, and other task run in this bad environment 
> will very likely fail. If multiple task failures in the same machine due to 
> EnvironmentError, we need to consider adding the bad machine to blacklist, 
> and avoiding schedule task on it.
>  ** Introduce a new exception EnvironmentException to wrap all those kind of 
> issues.
>  * Recoverable
>  ** We assume other issues are recoverable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] isunjin commented on issue #6739: [FLINK-10289] [JobManager] Classify Exceptions to different category for apply different failover strategy

[GitBox]

isunjin commented on issue #6739: [FLINK-10289] [JobManager] Classify 
Exceptions to different category for apply different failover strategy
URL: https://github.com/apache/flink/pull/6739#issuecomment-427431004
 
 
   @StefanRRichter Thank you, pushed the change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] markcho commented on issue #6793: [hotfix] [docs] Update cluster setup docs to reflect the new syntax of jobmanager.sh script.

[GitBox]

markcho commented on issue #6793: [hotfix] [docs] Update cluster setup docs to 
reflect the new syntax of jobmanager.sh script.
URL: https://github.com/apache/flink/pull/6793#issuecomment-427422038
 
 
   @azagrebin Does the reviewer merge the change into the master branch? It 
should also be applied to release-1.5 and release-1.6 branch since the change 
was made in Flink 1.5.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10412) toString field in AbstractID should be transient to avoid been serialized

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640034#comment-16640034
 ] 

ASF GitHub Bot commented on FLINK-10412:


yanghua commented on issue #6755: [FLINK-10412] toString field in AbstractID 
should be transient to avoid been serialized
URL: https://github.com/apache/flink/pull/6755#issuecomment-427420831
 
 
   @tillrohrmann and @zentol What's your opinion?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> toString field in AbstractID should be transient to avoid been serialized
> -
>
> Key: FLINK-10412
> URL: https://issues.apache.org/jira/browse/FLINK-10412
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.7.0
>Reporter: Zhu Zhu
>Assignee: vinoyang
>Priority: Major
>  Labels: deploy,deployment, pull-request-available, serialization
>
> The toString field in AbstractID will be serialized currently, which makes 
> RPC messages body like InputChannelDeploymentDescriptor and PartitionInfo 
> larger (50%+).
> It adds more pressure to JM memory especially in large scale job scheduling 
> (1x1 ALL-to-ALL connection).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] yanghua commented on issue #6755: [FLINK-10412] toString field in AbstractID should be transient to avoid been serialized

[GitBox]

yanghua commented on issue #6755: [FLINK-10412] toString field in AbstractID 
should be transient to avoid been serialized
URL: https://github.com/apache/flink/pull/6755#issuecomment-427420831
 
 
   @tillrohrmann and @zentol What's your opinion?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640032#comment-16640032
 ] 

ASF GitHub Bot commented on FLINK-10454:


tillrohrmann opened a new pull request #6799: [FLINK-10454][tests] Start 
MiniCluster with rest port 0
URL: https://github.com/apache/flink/pull/6799
 
 
   ## What is the purpose of the change
   
   Start the MiniCluster used by `ScheduleOrUpdateConsumersTest` and 
`SlotCountExceedingParallelismTest` with a rest port 0 in order to avoid port 
conflicts when these two tests are executed concurrently.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-10454:
---
Labels: pull-request-available test-stability  (was: test-stability)

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] tillrohrmann opened a new pull request #6799: [FLINK-10454][tests] Start MiniCluster with rest port 0

[GitBox]

tillrohrmann opened a new pull request #6799: [FLINK-10454][tests] Start 
MiniCluster with rest port 0
URL: https://github.com/apache/flink/pull/6799
 
 
   ## What is the purpose of the change
   
   Start the MiniCluster used by `ScheduleOrUpdateConsumersTest` and 
`SlotCountExceedingParallelismTest` with a rest port 0 in order to avoid port 
conflicts when these two tests are executed concurrently.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10427) Port JobSubmitTest to new code base

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640029#comment-16640029
 ] 

ASF GitHub Bot commented on FLINK-10427:


asfgit closed pull request #6768:  [FLINK-10427] [tests] Port JobSubmitTest to 
new code base
URL: https://github.com/apache/flink/pull/6768
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
deleted file mode 100644
index ef493b9330a..000
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
+++ /dev/null
@@ -1,244 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.runtime.jobmanager;
-
-import akka.actor.ActorSystem;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.JobManagerOptions;
-import org.apache.flink.runtime.akka.AkkaUtils;
-import org.apache.flink.runtime.akka.ListeningBehaviour;
-import org.apache.flink.runtime.blob.PermanentBlobKey;
-import org.apache.flink.runtime.checkpoint.CheckpointRetentionPolicy;
-import org.apache.flink.runtime.client.JobExecutionException;
-import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
-import 
org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedHaServices;
-import org.apache.flink.runtime.instance.ActorGateway;
-import org.apache.flink.runtime.jobgraph.JobGraph;
-import org.apache.flink.runtime.jobgraph.JobVertex;
-import org.apache.flink.runtime.jobgraph.JobVertexID;
-import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
-import 
org.apache.flink.runtime.jobgraph.tasks.CheckpointCoordinatorConfiguration;
-import org.apache.flink.runtime.jobgraph.tasks.JobCheckpointingSettings;
-import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
-import org.apache.flink.runtime.messages.JobManagerMessages;
-import org.apache.flink.runtime.metrics.NoOpMetricRegistry;
-import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testtasks.NoOpInvokable;
-import org.apache.flink.runtime.util.LeaderRetrievalUtils;
-import org.apache.flink.util.NetUtils;
-
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-import scala.Option;
-import scala.Tuple2;
-import scala.concurrent.Await;
-import scala.concurrent.Future;
-import scala.concurrent.duration.FiniteDuration;
-
-import java.io.IOException;
-import java.util.Collections;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-/**
- * Tests that the JobManager handles Jobs correctly that fail in
- * the initialization during the submit phase.
- */
-public class JobSubmitTest {
-
-   private static final FiniteDuration timeout = new FiniteDuration(6, 
TimeUnit.MILLISECONDS);
-
-   private static ActorSystem jobManagerSystem;
-   private static ActorGateway jmGateway;
-   private static Configuration jmConfig;
-   private static HighAvailabilityServices highAvailabilityServices;
-
-   @BeforeClass
-   public static void setupJobManager() {
-   jmConfig = new Configuration();
-
-   int port = NetUtils.getAvailablePort();
-
-   jmConfig.setString(JobManagerOptions.ADDRESS, "localhost");
-   jmConfig.setInteger(JobManagerOptions.PORT, port);
-
-   scala.Option> listeningAddress = 
scala.Option.apply(new Tuple2("localhost", port));
-   jobManagerSystem = AkkaUtils.createActorSystem(jmConfig, 
listeningAddress);
-
-   highAvailabilityServices = new 
EmbeddedHaServices(TestingUtils.defaultExecutor());
-
-   // only start 

[GitHub] asfgit closed pull request #6768: [FLINK-10427] [tests] Port JobSubmitTest to new code base

[GitBox]

asfgit closed pull request #6768:  [FLINK-10427] [tests] Port JobSubmitTest to 
new code base
URL: https://github.com/apache/flink/pull/6768
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
deleted file mode 100644
index ef493b9330a..000
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobSubmitTest.java
+++ /dev/null
@@ -1,244 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.runtime.jobmanager;
-
-import akka.actor.ActorSystem;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.JobManagerOptions;
-import org.apache.flink.runtime.akka.AkkaUtils;
-import org.apache.flink.runtime.akka.ListeningBehaviour;
-import org.apache.flink.runtime.blob.PermanentBlobKey;
-import org.apache.flink.runtime.checkpoint.CheckpointRetentionPolicy;
-import org.apache.flink.runtime.client.JobExecutionException;
-import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
-import 
org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedHaServices;
-import org.apache.flink.runtime.instance.ActorGateway;
-import org.apache.flink.runtime.jobgraph.JobGraph;
-import org.apache.flink.runtime.jobgraph.JobVertex;
-import org.apache.flink.runtime.jobgraph.JobVertexID;
-import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
-import 
org.apache.flink.runtime.jobgraph.tasks.CheckpointCoordinatorConfiguration;
-import org.apache.flink.runtime.jobgraph.tasks.JobCheckpointingSettings;
-import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
-import org.apache.flink.runtime.messages.JobManagerMessages;
-import org.apache.flink.runtime.metrics.NoOpMetricRegistry;
-import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testtasks.NoOpInvokable;
-import org.apache.flink.runtime.util.LeaderRetrievalUtils;
-import org.apache.flink.util.NetUtils;
-
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-import scala.Option;
-import scala.Tuple2;
-import scala.concurrent.Await;
-import scala.concurrent.Future;
-import scala.concurrent.duration.FiniteDuration;
-
-import java.io.IOException;
-import java.util.Collections;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-/**
- * Tests that the JobManager handles Jobs correctly that fail in
- * the initialization during the submit phase.
- */
-public class JobSubmitTest {
-
-   private static final FiniteDuration timeout = new FiniteDuration(6, 
TimeUnit.MILLISECONDS);
-
-   private static ActorSystem jobManagerSystem;
-   private static ActorGateway jmGateway;
-   private static Configuration jmConfig;
-   private static HighAvailabilityServices highAvailabilityServices;
-
-   @BeforeClass
-   public static void setupJobManager() {
-   jmConfig = new Configuration();
-
-   int port = NetUtils.getAvailablePort();
-
-   jmConfig.setString(JobManagerOptions.ADDRESS, "localhost");
-   jmConfig.setInteger(JobManagerOptions.PORT, port);
-
-   scala.Option> listeningAddress = 
scala.Option.apply(new Tuple2("localhost", port));
-   jobManagerSystem = AkkaUtils.createActorSystem(jmConfig, 
listeningAddress);
-
-   highAvailabilityServices = new 
EmbeddedHaServices(TestingUtils.defaultExecutor());
-
-   // only start JobManager (no ResourceManager)
-   JobManager.startJobManagerActors(
-   jmConfig,
-   jobManagerSystem,
-   TestingUtils.defaultExecutor(),
-   

[jira] [Resolved] (FLINK-10427) Port JobSubmitTest to new code base

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-10427.
---
Resolution: Fixed

Fixed via
deef139e02efc3b728654a202e2ad5247bc93a46
f90f5e5832425f999c5ce564c25284b912ebab18

> Port JobSubmitTest to new code base
> ---
>
> Key: FLINK-10427
> URL: https://issues.apache.org/jira/browse/FLINK-10427
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Port {{JobSubmitTest}} to new code base.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-5405) Add streaming connection to twitter through a internet proxy

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640016#comment-16640016
 ] 

Luis Angel Hernández Acosta commented on FLINK-5405:


Can you be more specific?

> Add streaming connection to twitter through a internet proxy
> 
>
> Key: FLINK-5405
> URL: https://issues.apache.org/jira/browse/FLINK-5405
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Luis
>Priority: Minor
>  Labels: easyfix
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Add streaming connection to twitter through a internet proxy upgrading 
> hbc-core dependency to 2.2.1-SNAPSHOT and changing the code to pass proxy 
> configurations to com.twitter.hbc.ClientBuilder.proxy().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10340) Add Cosh math function supported in Table API and SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640005#comment-16640005
 ] 

ASF GitHub Bot commented on FLINK-10340:


yanghua commented on issue #6700: [FLINK-10340][table] Add Cosh math function 
supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6700#issuecomment-427417452
 
 
   @pnowojski and @xccui Does this PR look good to you?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Cosh math function supported in Table API and SQL
> -
>
> Key: FLINK-10340
> URL: https://issues.apache.org/jira/browse/FLINK-10340
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.6.0
>Reporter: Sergey Tsvetkov
>Assignee: vinoyang
>Priority: Minor
>  Labels: pull-request-available
>
> Implement udf of cosh, just like in oracle
> [https://docs.oracle.com/cd/B28359_01/server.111/b28286/functions031.htm#SQLRF00623]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] yanghua commented on issue #6700: [FLINK-10340][table] Add Cosh math function supported in Table API and SQL

[GitBox]

yanghua commented on issue #6700: [FLINK-10340][table] Add Cosh math function 
supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6700#issuecomment-427417452
 
 
   @pnowojski and @xccui Does this PR look good to you?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-9682) Add setDescription to execution environment and provide description field for the rest api

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640003#comment-16640003
 ] 

ASF GitHub Bot commented on FLINK-9682:
---

yanghua commented on issue #6266: [FLINK-9682] Add setDescription to execution 
environment and provide description field for the rest api
URL: https://github.com/apache/flink/pull/6266#issuecomment-427416654
 
 
   @tillrohrmann and @aljoscha any opinion about this PR broke the API 
compatibility?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add setDescription to execution environment and provide description field for 
> the rest api
> --
>
> Key: FLINK-9682
> URL: https://issues.apache.org/jira/browse/FLINK-9682
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API, Webfrontend
>Affects Versions: 1.5.0
>Reporter: Elias Levy
>Assignee: vinoyang
>Priority: Major
>  Labels: pull-request-available
>
> Currently you can provide a job name to {{execute}} in the execution 
> environment.  In an environment where many version of a job may be executing, 
> such as a development or test environment, identifying which running job is 
> of a specific version via the UI can be difficult unless the version is 
> embedded into the job name given the {{execute}}.  But the job name is uses 
> for other purposes, such as for namespacing metrics.  Thus, it is not ideal 
> to modify the job name, as that could require modifying metric dashboards and 
> monitors each time versions change.
> I propose a new method be added to the execution environment, 
> {{setDescription}}, that would allow a user to pass in an arbitrary 
> description that would be displayed in the dashboard, allowing users to 
> distinguish jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] yanghua commented on issue #6266: [FLINK-9682] Add setDescription to execution environment and provide description field for the rest api

[GitBox]

yanghua commented on issue #6266: [FLINK-9682] Add setDescription to execution 
environment and provide description field for the rest api
URL: https://github.com/apache/flink/pull/6266#issuecomment-427416654
 
 
   @tillrohrmann and @aljoscha any opinion about this PR broke the API 
compatibility?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-10454:
-

Assignee: Till Rohrmann

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10471) State TTL cleanup using RocksDb compaction filter

[Andrey Zagrebin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639954#comment-16639954
 ] 

Andrey Zagrebin commented on FLINK-10471:
-

[RocksDB PR for Flink compaction 
filter|https://github.com/facebook/rocksdb/pull/4463]

> State TTL cleanup using RocksDb compaction filter
> -
>
> Key: FLINK-10471
> URL: https://issues.apache.org/jira/browse/FLINK-10471
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.7.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Fix For: 1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639922#comment-16639922
 ] 

ASF GitHub Bot commented on FLINK-8865:
---

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223034776
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java
 ##
 @@ -233,6 +238,36 @@ public String explainStatement(SessionContext session, 
String statement) throws
}
}
 
+   public org.apache.calcite.schema.Table getTable(SessionContext session, 
String name) throws SqlExecutionException {
 
 Review comment:
   Please implement a better separation of concerns. The executor interface 
might be used for a REST API in the future (see FLIP-24). So this method should 
not expose internal Calcite classes but only information about how to complete 
the SQL query.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add CLI query code completion in SQL Client
> ---
>
> Key: FLINK-8865
> URL: https://issues.apache.org/jira/browse/FLINK-8865
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
>  Labels: pull-request-available
>
> This issue is a subtask of part two "Full Embedded SQL Client" of the 
> implementation plan mentioned in 
> [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client].
> Calcite already offers a code completion functionality. It would be great if 
> we could expose this feature also through the SQL CLI Client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639926#comment-16639926
 ] 

ASF GitHub Bot commented on FLINK-8865:
---

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223033380
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/SqlMultiLineParser.java
 ##
 @@ -20,33 +20,26 @@
 
 import org.jline.reader.EOFError;
 import org.jline.reader.ParsedLine;
-import org.jline.reader.Parser;
 import org.jline.reader.impl.DefaultParser;
 
-import java.util.Collections;
-
 /**
  * Multi-line parser for parsing an arbitrary number of SQL lines until a line 
ends with ';'.
  */
-public class SqlMultiLineParser implements Parser {
-
+//public class SqlMultiLineParser implements Parser {
 
 Review comment:
   Remove line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add CLI query code completion in SQL Client
> ---
>
> Key: FLINK-8865
> URL: https://issues.apache.org/jira/browse/FLINK-8865
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
>  Labels: pull-request-available
>
> This issue is a subtask of part two "Full Embedded SQL Client" of the 
> implementation plan mentioned in 
> [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client].
> Calcite already offers a code completion functionality. It would be great if 
> we could expose this feature also through the SQL CLI Client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639908#comment-16639908
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223030850
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
+   List> tasks = new 
ArrayList<>(stateHandleMap.size());
+
for (Map.Entry entry 
: stateHandleMap.entrySet()) {
StateHandleID stateHandleID = entry.getKey();
StreamStateHandle remoteFileHandle = 
entry.getValue();
-   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+
+   FutureTask task = new FutureTask<>(() -> {
+   try {
+   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+   } catch (IOException e) {
+   LOG.error("Copy State Data 
error, stateHandleID [{}], remoteFileHandle[{}]", stateHandleID.toString(), 
remoteFileHandle);
+   throw e;
+   }
+   return null;
+   });
+   tasks.add(task);
+   executorService.submit(task);
 
 Review comment:
   I suggest to use `CompletableFuture.runAsync(Runnable, Executor)` instead of 
`FutureTask`. It allows to use 
`FutureUtils.waitForAll(CompletableFutures).get()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8302) Support shift_left and shift_right in TableAPI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639935#comment-16639935
 ] 

ASF GitHub Bot commented on FLINK-8302:
---

twalthr commented on issue #6445: [FLINK-8302] [table] Add SHIFT_LEFT and 
SHIFT_RIGHT
URL: https://github.com/apache/flink/pull/6445#issuecomment-427397780
 
 
   @fhueske @pnowojski what is your opinion here? should we do the Spark/Hive 
way for now? We can still support other syntax in the future.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support shift_left and shift_right in TableAPI
> --
>
> Key: FLINK-8302
> URL: https://issues.apache.org/jira/browse/FLINK-8302
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: DuBin
>Priority: Major
>  Labels: features, pull-request-available
> Fix For: 1.7.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Add shift_left and shift_right support in TableAPI, shift_left(input, n) act 
> as input << n, shift_right(input, n) act as input >> n.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] twalthr commented on issue #6445: [FLINK-8302] [table] Add SHIFT_LEFT and SHIFT_RIGHT

[GitBox]

twalthr commented on issue #6445: [FLINK-8302] [table] Add SHIFT_LEFT and 
SHIFT_RIGHT
URL: https://github.com/apache/flink/pull/6445#issuecomment-427397780
 
 
   @fhueske @pnowojski what is your opinion here? should we do the Spark/Hive 
way for now? We can still support other syntax in the future.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639925#comment-16639925
 ] 

ASF GitHub Bot commented on FLINK-8865:
---

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223032399
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
 ##
 @@ -97,10 +97,12 @@ public CliClient(SessionContext context, Executor 
executor) {
.terminal(terminal)
.appName(CliStrings.CLI_NAME)
.parser(new SqlMultiLineParser())
+   .completer(new SqlCompleter(context, executor))
.build();
// this option is disabled for now for correct backslash 
escaping
// a "SELECT '\'" query should return a string with a backslash
lineReader.option(LineReader.Option.DISABLE_EVENT_EXPANSION, 
true);
+   lineReader.setVariable("errors", 1);
 
 Review comment:
   Please add a line comment for what this variable is good for?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add CLI query code completion in SQL Client
> ---
>
> Key: FLINK-8865
> URL: https://issues.apache.org/jira/browse/FLINK-8865
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
>  Labels: pull-request-available
>
> This issue is a subtask of part two "Full Embedded SQL Client" of the 
> implementation plan mentioned in 
> [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client].
> Calcite already offers a code completion functionality. It would be great if 
> we could expose this feature also through the SQL CLI Client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639924#comment-16639924
 ] 

ASF GitHub Bot commented on FLINK-8865:
---

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223035081
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java
 ##
 @@ -233,6 +238,36 @@ public String explainStatement(SessionContext session, 
String statement) throws
}
}
 
+   public org.apache.calcite.schema.Table getTable(SessionContext session, 
String name) throws SqlExecutionException {
+   final TableEnvironment tableEnv = 
getOrCreateExecutionContext(session)
+   .createEnvironmentInstance()
+   .getTableEnvironment();
+
+   Class clazz = Object.class;
+   if (tableEnv instanceof StreamTableEnvironment) {
+   StreamTableEnvironment tmpTableEnv = 
(StreamTableEnvironment) tableEnv;
+   clazz = tmpTableEnv.getClass();
+   } else if (tableEnv instanceof BatchTableEnvironment) {
+   BatchTableEnvironment tmpTableEnv = 
(BatchTableEnvironment) tableEnv;
+   clazz = tmpTableEnv.getClass();
+   }
+
+   for (; clazz != Object.class; clazz = clazz.getSuperclass()) {
+   try {
+   Method[] methods = clazz.getDeclaredMethods();
+
+   Method mt = clazz.getDeclaredMethod("getTable", 
String.class);
 
 Review comment:
   Instead of ugly reflection, we should implement a proper solution in the 
table environment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add CLI query code completion in SQL Client
> ---
>
> Key: FLINK-8865
> URL: https://issues.apache.org/jira/browse/FLINK-8865
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
>  Labels: pull-request-available
>
> This issue is a subtask of part two "Full Embedded SQL Client" of the 
> implementation plan mentioned in 
> [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client].
> Calcite already offers a code completion functionality. It would be great if 
> we could expose this feature also through the SQL CLI Client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8865) Add CLI query code completion in SQL Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639923#comment-16639923
 ] 

ASF GitHub Bot commented on FLINK-8865:
---

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223033140
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/SqlCompleter.java
 ##
 @@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.client.cli;
+
+import org.apache.flink.table.calcite.FlinkRelBuilder;
+import org.apache.flink.table.calcite.FlinkTypeFactory;
+import org.apache.flink.table.calcite.FlinkTypeSystem;
+import org.apache.flink.table.client.gateway.Executor;
+import org.apache.flink.table.client.gateway.SessionContext;
+import org.apache.flink.table.client.gateway.local.LocalExecutor;
+import org.apache.flink.table.plan.cost.DataSetCostFactory;
+import org.apache.flink.table.validate.FunctionCatalog;
+
+import org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.calcite.config.CalciteConnectionProperty;
+import org.apache.calcite.config.Lex;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.advise.SqlAdvisor;
+import org.apache.calcite.sql.advise.SqlAdvisorValidator;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.calcite.sql.validate.SqlMoniker;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.calcite.tools.Frameworks;
+import org.jline.reader.Candidate;
+import org.jline.reader.Completer;
+import org.jline.reader.LineReader;
+import org.jline.reader.ParsedLine;
+import org.jline.utils.AttributedString;
+
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * SQL Completer.
+ */
+public class SqlCompleter implements Completer {
+   private SessionContext context;
+   private Executor executor;
+   private SqlParser.Config parserConfig = SqlParser
 
 Review comment:
   These properties are already defined in a table environment. They should not 
be copied in this class. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add CLI query code completion in SQL Client
> ---
>
> Key: FLINK-8865
> URL: https://issues.apache.org/jira/browse/FLINK-8865
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
>  Labels: pull-request-available
>
> This issue is a subtask of part two "Full Embedded SQL Client" of the 
> implementation plan mentioned in 
> [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client].
> Calcite already offers a code completion functionality. It would be great if 
> we could expose this feature also through the SQL CLI Client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client

[GitBox]

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223033380
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/SqlMultiLineParser.java
 ##
 @@ -20,33 +20,26 @@
 
 import org.jline.reader.EOFError;
 import org.jline.reader.ParsedLine;
-import org.jline.reader.Parser;
 import org.jline.reader.impl.DefaultParser;
 
-import java.util.Collections;
-
 /**
  * Multi-line parser for parsing an arbitrary number of SQL lines until a line 
ends with ';'.
  */
-public class SqlMultiLineParser implements Parser {
-
+//public class SqlMultiLineParser implements Parser {
 
 Review comment:
   Remove line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client

[GitBox]

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223033140
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/SqlCompleter.java
 ##
 @@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.client.cli;
+
+import org.apache.flink.table.calcite.FlinkRelBuilder;
+import org.apache.flink.table.calcite.FlinkTypeFactory;
+import org.apache.flink.table.calcite.FlinkTypeSystem;
+import org.apache.flink.table.client.gateway.Executor;
+import org.apache.flink.table.client.gateway.SessionContext;
+import org.apache.flink.table.client.gateway.local.LocalExecutor;
+import org.apache.flink.table.plan.cost.DataSetCostFactory;
+import org.apache.flink.table.validate.FunctionCatalog;
+
+import org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.calcite.config.CalciteConnectionProperty;
+import org.apache.calcite.config.Lex;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.advise.SqlAdvisor;
+import org.apache.calcite.sql.advise.SqlAdvisorValidator;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.calcite.sql.validate.SqlMoniker;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.calcite.tools.Frameworks;
+import org.jline.reader.Candidate;
+import org.jline.reader.Completer;
+import org.jline.reader.LineReader;
+import org.jline.reader.ParsedLine;
+import org.jline.utils.AttributedString;
+
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * SQL Completer.
+ */
+public class SqlCompleter implements Completer {
+   private SessionContext context;
+   private Executor executor;
+   private SqlParser.Config parserConfig = SqlParser
 
 Review comment:
   These properties are already defined in a table environment. They should not 
be copied in this class. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client

[GitBox]

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223035081
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java
 ##
 @@ -233,6 +238,36 @@ public String explainStatement(SessionContext session, 
String statement) throws
}
}
 
+   public org.apache.calcite.schema.Table getTable(SessionContext session, 
String name) throws SqlExecutionException {
+   final TableEnvironment tableEnv = 
getOrCreateExecutionContext(session)
+   .createEnvironmentInstance()
+   .getTableEnvironment();
+
+   Class clazz = Object.class;
+   if (tableEnv instanceof StreamTableEnvironment) {
+   StreamTableEnvironment tmpTableEnv = 
(StreamTableEnvironment) tableEnv;
+   clazz = tmpTableEnv.getClass();
+   } else if (tableEnv instanceof BatchTableEnvironment) {
+   BatchTableEnvironment tmpTableEnv = 
(BatchTableEnvironment) tableEnv;
+   clazz = tmpTableEnv.getClass();
+   }
+
+   for (; clazz != Object.class; clazz = clazz.getSuperclass()) {
+   try {
+   Method[] methods = clazz.getDeclaredMethods();
+
+   Method mt = clazz.getDeclaredMethod("getTable", 
String.class);
 
 Review comment:
   Instead of ugly reflection, we should implement a proper solution in the 
table environment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client

[GitBox]

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223032399
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
 ##
 @@ -97,10 +97,12 @@ public CliClient(SessionContext context, Executor 
executor) {
.terminal(terminal)
.appName(CliStrings.CLI_NAME)
.parser(new SqlMultiLineParser())
+   .completer(new SqlCompleter(context, executor))
.build();
// this option is disabled for now for correct backslash 
escaping
// a "SELECT '\'" query should return a string with a backslash
lineReader.option(LineReader.Option.DISABLE_EVENT_EXPANSION, 
true);
+   lineReader.setVariable("errors", 1);
 
 Review comment:
   Please add a line comment for what this variable is good for?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] Add CLI query code completion in SQL Client

[GitBox]

twalthr commented on a change in pull request #6791: [FLINK-8865][sql-client] 
Add CLI query code completion in SQL Client
URL: https://github.com/apache/flink/pull/6791#discussion_r223034776
 
 

 ##
 File path: 
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/LocalExecutor.java
 ##
 @@ -233,6 +238,36 @@ public String explainStatement(SessionContext session, 
String statement) throws
}
}
 
+   public org.apache.calcite.schema.Table getTable(SessionContext session, 
String name) throws SqlExecutionException {
 
 Review comment:
   Please implement a better separation of concerns. The executor interface 
might be used for a REST API in the future (see FLIP-24). So this method should 
not expose internal Calcite classes but only information about how to complete 
the SQL query.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639916#comment-16639916
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223029260
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Number of threads should be configurable. For example, the same way as 
incremental checkpointing. At least in RocksDBStateBackend constructor and 
CheckpointingOptions. The default behaviour (e.g. number of threads 1) can be 
as it is now, basically running in current thread. We also can create a follow 
up issue to reconsider rocksdb backend configuration later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223029260
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Number of threads should be configurable. For example, the same way as 
incremental checkpointing. At least in RocksDBStateBackend constructor and 
CheckpointingOptions. The default behaviour (e.g. number of threads 1) can be 
as it is now, basically running in current thread. We also can create a follow 
up issue to reconsider rocksdb backend configuration later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223030850
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
+   List> tasks = new 
ArrayList<>(stateHandleMap.size());
+
for (Map.Entry entry 
: stateHandleMap.entrySet()) {
StateHandleID stateHandleID = entry.getKey();
StreamStateHandle remoteFileHandle = 
entry.getValue();
-   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+
+   FutureTask task = new FutureTask<>(() -> {
+   try {
+   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+   } catch (IOException e) {
+   LOG.error("Copy State Data 
error, stateHandleID [{}], remoteFileHandle[{}]", stateHandleID.toString(), 
remoteFileHandle);
+   throw e;
+   }
+   return null;
+   });
+   tasks.add(task);
+   executorService.submit(task);
 
 Review comment:
   I suggest to use `CompletableFuture.runAsync(Runnable, Executor)` instead of 
`FutureTask`. It allows to use 
`FutureUtils.waitForAll(CompletableFutures).get()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223034163
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackendTest.java
 ##
 @@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.contrib.streaming.state;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.IncrementalKeyedStateHandle;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateHandleID;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.runtime.state.TestLocalRecoveryConfig;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for RocksDBKeyedStateBackend.
+ */
+public class RocksDBKeyedStateBackendTest {
 
 Review comment:
   In addition to failure scenario, I would also then add a test which mocks 
`StreamStateHandle` streams and checks that all streams are read and written 
correctly in parallel.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639912#comment-16639912
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223032334
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Cancelation of `copyStateDataHandleData` futures and 
`executorService.shutdownNow()` should be registered in 
`cancelStreamRegistry.registerCloseable()` and unregistered with 
`unregisterCloseable` before `executorService.shutdown()` in `finally`. Similar 
to streams in `copyStateDataHandleData`. This way we guarantee freeing of all 
resources in case of job abrupt shutdown.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639911#comment-16639911
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223034163
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackendTest.java
 ##
 @@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.contrib.streaming.state;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.IncrementalKeyedStateHandle;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateHandleID;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.runtime.state.TestLocalRecoveryConfig;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for RocksDBKeyedStateBackend.
+ */
+public class RocksDBKeyedStateBackendTest {
 
 Review comment:
   In addition to failure scenario, I would also then add a test which mocks 
`StreamStateHandle` streams and checks that all streams are read and written 
correctly in parallel.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223032334
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Cancelation of `copyStateDataHandleData` futures and 
`executorService.shutdownNow()` should be registered in 
`cancelStreamRegistry.registerCloseable()` and unregistered with 
`unregisterCloseable` before `executorService.shutdown()` in `finally`. Similar 
to streams in `copyStateDataHandleData`. This way we guarantee freeing of all 
resources in case of job abrupt shutdown.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223030317
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
+   List> tasks = new 
ArrayList<>(stateHandleMap.size());
+
for (Map.Entry entry 
: stateHandleMap.entrySet()) {
StateHandleID stateHandleID = entry.getKey();
StreamStateHandle remoteFileHandle = 
entry.getValue();
-   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+
+   FutureTask task = new FutureTask<>(() -> {
 
 Review comment:
   I think it would be more maintainable if we extract these 3 methods:
   - `transferAllStateDataToDirectory`
   - `transferAllDataFromStateHandles`
   - `copyStateDataHandleData`
   into a separate class file, e.g. `RocksDbStateDataTransfer`.
   `RocksDBKeyedStateBackend` already contains a lot.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] azagrebin commented on a change in pull request #6777: [FLINK-10461] [State Backends, Checkpointing] Speed up download files when restore from DFS

[GitBox]

azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223029260
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Number of threads should be configurable. For example, the same way as 
incremental checkpointing. At least in RocksDBStateBackend constructor and 
CheckpointingOptions. The default behaviour (e.g. number of threads 1) can be 
as it is now, basically running in current thread.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639909#comment-16639909
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223030317
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
+   List> tasks = new 
ArrayList<>(stateHandleMap.size());
+
for (Map.Entry entry 
: stateHandleMap.entrySet()) {
StateHandleID stateHandleID = entry.getKey();
StreamStateHandle remoteFileHandle = 
entry.getValue();
-   copyStateDataHandleData(new 
Path(restoreInstancePath, stateHandleID.toString()), remoteFileHandle);
+
+   FutureTask task = new FutureTask<>(() -> {
 
 Review comment:
   I think it would be more maintainable if we extract these 3 methods:
   - `transferAllStateDataToDirectory`
   - `transferAllDataFromStateHandles`
   - `copyStateDataHandleData`
   into a separate class file, e.g. `RocksDbStateDataTransfer`.
   `RocksDBKeyedStateBackend` already contains a lot.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10461) Speed up download file procedure when restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639910#comment-16639910
 ] 

ASF GitHub Bot commented on FLINK-10461:


azagrebin commented on a change in pull request #6777: [FLINK-10461] [State 
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223029260
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
 ##
 @@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map stateHandleMap,
Path restoreInstancePath) throws IOException {
 
+   ExecutorService executorService = 
Executors.newFixedThreadPool(5);
 
 Review comment:
   Number of threads should be configurable. For example, the same way as 
incremental checkpointing. At least in RocksDBStateBackend constructor and 
CheckpointingOptions. The default behaviour (e.g. number of threads 1) can be 
as it is now, basically running in current thread.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Speed up download file procedure when restore 
> --
>
> Key: FLINK-10461
> URL: https://issues.apache.org/jira/browse/FLINK-10461
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
>  Labels: pull-request-available
>
> In the current master branch, the restore will download file from DFS, the 
> download procedure are single-thread, this could speed up by using 
> multi-thread for downloading states from DFS.
>  
> In my company, the states will come to some terabytes, so the restore 
> procedure will become a litter slow, after a bit digging, I find download 
> states from DFS using single thread, this could using multi-thread for speed 
> up.
> I test the time used for download states from DFS with ~2 terabytes states. 
> With single thread it used 640+s, and 130+s when using 5 threads for download.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bbaja42 opened a new pull request #6798: [docs] Fixed typo, please instead of pelase

[GitBox]

bbaja42 opened a new pull request #6798: [docs]  Fixed typo, please instead of 
pelase
URL: https://github.com/apache/flink/pull/6798
 
 
   ## Contribution Checklist
   
 - Make sure that the pull request corresponds to a [JIRA 
issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are 
made for typos in JavaDoc or documentation files, which need no JIRA issue.
 
 - Name the pull request in the form "[FLINK-] [component] Title of the 
pull request", where *FLINK-* should be replaced by the actual issue 
number. Skip *component* if you are unsure about which is the best component.
 Typo fixes that have no associated JIRA issue should be named following 
this pattern: `[hotfix] [docs] Fix typo in event time introduction` or 
`[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
 - Fill out the template below to describe the changes contributed by the 
pull request. That will give reviewers the context they need to do the review.
 
 - Make sure that the change passes the automated tests, i.e., `mvn clean 
verify` passes. You can set up Travis CI to do that following [this 
guide](http://flink.apache.org/contribute-code.html#best-practices).
   
 - Each pull request should address only one issue, not mix up code from 
multiple issues.
 
 - Each commit in the pull request has a meaningful commit message 
(including the JIRA id)
   
 - Once all items of the checklist are addressed, remove the above text and 
this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   *(For example: This pull request makes task deployment go through the blob 
server, rather than through RPC. That way we avoid re-transferring them on each 
deployment (during recovery).)*
   
   
   ## Brief change log
   
   *(for example:)*
 - *The TaskInfo is stored in the blob store on job creation time as a 
persistent artifact*
 - *Deployments RPC transmits only the blob storage reference*
 - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(100MB)*
 - *Extended integration test for recovery after master (JobManager) 
failure*
 - *Added test that validates that TaskInfo is transferred only once across 
recoveries*
 - *Manually verified the change by running a 4 node cluser with 2 
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one 
JobManager and two TaskManagers during the execution, verifying that recovery 
happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / no)
 - The serializers: (yes / no / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / no / 
don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
 - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (FLINK-10496) CommandLineParser arguments interleaving

[eugen yushin (JIRA)]

eugen yushin created FLINK-10496:


 Summary: CommandLineParser arguments interleaving
 Key: FLINK-10496
 URL: https://issues.apache.org/jira/browse/FLINK-10496
 Project: Flink
  Issue Type: Improvement
  Components: Configuration, Core, Java API
Affects Versions: 1.6.1, 1.7.0
Reporter: eugen yushin
Assignee: eugen yushin


*Business case:*
Run Flink job cluster within Docker/k8s. Job takes an argument called 
`--config` which can't be recognized in runtime.

{code:java}
Caused by: java.lang.RuntimeException: No data for required key 'config'
{code}

*Problem statement:*
Command line parser can't recognize job specific arguments when they have the 
same prefix as Flink's ones.

e.g.
[https://github.com/apache/flink/blob/master/flink-container/src/test/java/org/apache/flink/container/entrypoint/StandaloneJobClusterConfigurationParserFactoryTest.java#L52]

the following args results in failure:
{code:java}
final String arg1 = "--config";
final String arg2 = "/path/to/job.yaml";{code}

*Reason*:
Apache CLI parser use string prefix matching to parse options and adds extra 
--configDir to result set instead of adding new --config.
https://github.com/apache/commons-cli/blob/cli-1.3.1/src/main/java/org/apache/commons/cli/DefaultParser.java#L391

*Scope*:
Update commons-cli dependency with version 1.4 which has flag to disable 
partial matching.
https://github.com/apache/commons-cli/commit/bdb4a09ceaceab7e3d214b1beadb93bd9c911342

Update Flink's command line parser to utilize this feature.
https://github.com/apache/flink/blob/6258a4c333ce9dba914621b13eac2f7d91f5cb72/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/parser/CommandLineParser.java#L45



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10496) CommandLineParser arguments interleaving

[eugen yushin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

eugen yushin updated FLINK-10496:
-
Labels: pull-request-available  (was: )

> CommandLineParser arguments interleaving
> 
>
> Key: FLINK-10496
> URL: https://issues.apache.org/jira/browse/FLINK-10496
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration, Core, Docker, Java API
>Affects Versions: 1.6.1, 1.7.0
>Reporter: eugen yushin
>Assignee: eugen yushin
>Priority: Major
>  Labels: pull-request-available
>
> *Business case:*
> Run Flink job cluster within Docker/k8s. Job takes an argument called 
> `--config` which can't be recognized in runtime.
> {code:java}
> Caused by: java.lang.RuntimeException: No data for required key 'config'
> {code}
> *Problem statement:*
> Command line parser can't recognize job specific arguments when they have the 
> same prefix as Flink's ones.
> e.g.
> [https://github.com/apache/flink/blob/master/flink-container/src/test/java/org/apache/flink/container/entrypoint/StandaloneJobClusterConfigurationParserFactoryTest.java#L52]
> the following args results in failure:
> {code:java}
> final String arg1 = "--config";
> final String arg2 = "/path/to/job.yaml";{code}
> *Reason*:
> Apache CLI parser use string prefix matching to parse options and adds extra 
> --configDir to result set instead of adding new --config.
> https://github.com/apache/commons-cli/blob/cli-1.3.1/src/main/java/org/apache/commons/cli/DefaultParser.java#L391
> *Scope*:
> Update commons-cli dependency with version 1.4 which has flag to disable 
> partial matching.
> https://github.com/apache/commons-cli/commit/bdb4a09ceaceab7e3d214b1beadb93bd9c911342
> Update Flink's command line parser to utilize this feature.
> https://github.com/apache/flink/blob/6258a4c333ce9dba914621b13eac2f7d91f5cb72/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/parser/CommandLineParser.java#L45



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10496) CommandLineParser arguments interleaving

[eugen yushin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

eugen yushin updated FLINK-10496:
-
Component/s: Docker

> CommandLineParser arguments interleaving
> 
>
> Key: FLINK-10496
> URL: https://issues.apache.org/jira/browse/FLINK-10496
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration, Core, Docker, Java API
>Affects Versions: 1.6.1, 1.7.0
>Reporter: eugen yushin
>Assignee: eugen yushin
>Priority: Major
>  Labels: pull-request-available
>
> *Business case:*
> Run Flink job cluster within Docker/k8s. Job takes an argument called 
> `--config` which can't be recognized in runtime.
> {code:java}
> Caused by: java.lang.RuntimeException: No data for required key 'config'
> {code}
> *Problem statement:*
> Command line parser can't recognize job specific arguments when they have the 
> same prefix as Flink's ones.
> e.g.
> [https://github.com/apache/flink/blob/master/flink-container/src/test/java/org/apache/flink/container/entrypoint/StandaloneJobClusterConfigurationParserFactoryTest.java#L52]
> the following args results in failure:
> {code:java}
> final String arg1 = "--config";
> final String arg2 = "/path/to/job.yaml";{code}
> *Reason*:
> Apache CLI parser use string prefix matching to parse options and adds extra 
> --configDir to result set instead of adding new --config.
> https://github.com/apache/commons-cli/blob/cli-1.3.1/src/main/java/org/apache/commons/cli/DefaultParser.java#L391
> *Scope*:
> Update commons-cli dependency with version 1.4 which has flag to disable 
> partial matching.
> https://github.com/apache/commons-cli/commit/bdb4a09ceaceab7e3d214b1beadb93bd9c911342
> Update Flink's command line parser to utilize this feature.
> https://github.com/apache/flink/blob/6258a4c333ce9dba914621b13eac2f7d91f5cb72/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/parser/CommandLineParser.java#L45



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9679) Implement AvroSerializationSchema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639835#comment-16639835
 ] 

ASF GitHub Bot commented on FLINK-9679:
---

WheresAlice commented on a change in pull request #6259: [FLINK-9679] Implement 
AvroSerializationSchema
URL: https://github.com/apache/flink/pull/6259#discussion_r223005167
 
 

 ##
 File path: flink-formats/flink-avro-confluent-registry/pom.xml
 ##
 @@ -29,6 +29,10 @@ under the License.
 
flink-avro-confluent-registry
 
+   
+   
4.1.0
 
 Review comment:
   Typo - "registry" not "rigestry"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Implement AvroSerializationSchema
> -
>
> Key: FLINK-9679
> URL: https://issues.apache.org/jira/browse/FLINK-9679
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Yazdan Shirvany
>Assignee: Yazdan Shirvany
>Priority: Major
>  Labels: pull-request-available
>
> Implement AvroSerializationSchema using Confluent Schema Registry



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10470) Add method to check if pattern can produce empty matches

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639833#comment-16639833
 ] 

ASF GitHub Bot commented on FLINK-10470:


dawidwys closed pull request #6781: [FLINK-10470] Add method to check if 
pattern can produce empty matches
URL: https://github.com/apache/flink/pull/6781
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
 
b/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
index 8f49f68fe41..ed2ff2e7e59 100644
--- 
a/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
+++ 
b/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
@@ -40,8 +40,13 @@
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.Stack;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * Compiler class containing methods to compile a {@link Pattern} into a 
{@link NFA} or a
@@ -74,6 +79,44 @@
}
}
 
+   /**
+* Verifies if the provided pattern can possibly generate empty match. 
Example of patterns that can possibly
+* generate empty matches are: A*, A?, A* B? etc.
+*
+* @param pattern pattern to check
+* @return true if empty match could potentially match the pattern, 
false otherwise
+*/
+   public static boolean canProduceEmptyMatches(final Pattern 
pattern) {
+   NFAFactoryCompiler compiler = new 
NFAFactoryCompiler<>(checkNotNull(pattern));
+   compiler.compileFactory();
+   State startState = 
compiler.getStates().stream().filter(State::isStart).findFirst().orElseThrow(
+   () -> new IllegalStateException("Compiler produced no 
start state. It is a bug. File a jira."));
+
+   Set> visitedStates = new HashSet<>();
+   final Stack> statesToCheck = new Stack<>();
+   statesToCheck.push(startState);
+   while (!statesToCheck.isEmpty()) {
+   final State currentState = statesToCheck.pop();
+   if (visitedStates.contains(currentState)) {
+   continue;
+   } else {
+   visitedStates.add(currentState);
+   }
+
+   for (StateTransition transition : 
currentState.getStateTransitions()) {
+   if (transition.getAction() == 
StateTransitionAction.PROCEED) {
+   if 
(transition.getTargetState().isFinal()) {
+   return true;
+   } else {
+   
statesToCheck.push(transition.getTargetState());
+   }
+   }
+   }
+   }
+
+   return false;
+   }
+
/**
 * Converts a {@link Pattern} into graph of {@link State}. It enables 
sharing of
 * compilation state across methods.
diff --git 
a/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
 
b/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
index f39b17483cd..821da031c6d 100644
--- 
a/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
+++ 
b/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
@@ -44,7 +44,9 @@
 import java.util.Set;
 
 import static org.apache.flink.cep.utils.NFAUtils.compile;
+import static org.hamcrest.Matchers.is;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 
 /**
@@ -226,4 +228,14 @@ public boolean filter(Event value) throws Exception {
return transitions;
}
 
+   @Test
+   public void testCheckingEmptyMatches() {
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").optional()), 
is(true));
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").oneOrMore().optional()),
 is(true));
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").oneOrMore().optional().next("b").optional()),
 is(true));
+
+   

[GitHub] WheresAlice commented on a change in pull request #6259: [FLINK-9679] Implement AvroSerializationSchema

[GitBox]

WheresAlice commented on a change in pull request #6259: [FLINK-9679] Implement 
AvroSerializationSchema
URL: https://github.com/apache/flink/pull/6259#discussion_r223005167
 
 

 ##
 File path: flink-formats/flink-avro-confluent-registry/pom.xml
 ##
 @@ -29,6 +29,10 @@ under the License.
 
flink-avro-confluent-registry
 
+   
+   
4.1.0
 
 Review comment:
   Typo - "registry" not "rigestry"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (FLINK-10470) Add method to check if pattern can produce empty matches

[Dawid Wysakowicz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz resolved FLINK-10470.
--
   Resolution: Fixed
Fix Version/s: 1.7.0

> Add method to check if pattern can produce empty matches
> 
>
> Key: FLINK-10470
> URL: https://issues.apache.org/jira/browse/FLINK-10470
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> There is couple of inconsistencies how CEP library handles greedy and 
> reluctant operators at the beginning at end of pattern. This results in 
> subtle problems how empty matches should be generated for patterns like e.g. 
> A? or A*?, where one is greedy and the other one is reluctant. In order to 
> provide first version of MATCH_RECOGNIZE function we should have a 
> possibility to disable patterns which can produce empty matches.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] dawidwys closed pull request #6781: [FLINK-10470] Add method to check if pattern can produce empty matches

[GitBox]

dawidwys closed pull request #6781: [FLINK-10470] Add method to check if 
pattern can produce empty matches
URL: https://github.com/apache/flink/pull/6781
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
 
b/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
index 8f49f68fe41..ed2ff2e7e59 100644
--- 
a/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
+++ 
b/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
@@ -40,8 +40,13 @@
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.Stack;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * Compiler class containing methods to compile a {@link Pattern} into a 
{@link NFA} or a
@@ -74,6 +79,44 @@
}
}
 
+   /**
+* Verifies if the provided pattern can possibly generate empty match. 
Example of patterns that can possibly
+* generate empty matches are: A*, A?, A* B? etc.
+*
+* @param pattern pattern to check
+* @return true if empty match could potentially match the pattern, 
false otherwise
+*/
+   public static boolean canProduceEmptyMatches(final Pattern 
pattern) {
+   NFAFactoryCompiler compiler = new 
NFAFactoryCompiler<>(checkNotNull(pattern));
+   compiler.compileFactory();
+   State startState = 
compiler.getStates().stream().filter(State::isStart).findFirst().orElseThrow(
+   () -> new IllegalStateException("Compiler produced no 
start state. It is a bug. File a jira."));
+
+   Set> visitedStates = new HashSet<>();
+   final Stack> statesToCheck = new Stack<>();
+   statesToCheck.push(startState);
+   while (!statesToCheck.isEmpty()) {
+   final State currentState = statesToCheck.pop();
+   if (visitedStates.contains(currentState)) {
+   continue;
+   } else {
+   visitedStates.add(currentState);
+   }
+
+   for (StateTransition transition : 
currentState.getStateTransitions()) {
+   if (transition.getAction() == 
StateTransitionAction.PROCEED) {
+   if 
(transition.getTargetState().isFinal()) {
+   return true;
+   } else {
+   
statesToCheck.push(transition.getTargetState());
+   }
+   }
+   }
+   }
+
+   return false;
+   }
+
/**
 * Converts a {@link Pattern} into graph of {@link State}. It enables 
sharing of
 * compilation state across methods.
diff --git 
a/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
 
b/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
index f39b17483cd..821da031c6d 100644
--- 
a/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
+++ 
b/flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/nfa/compiler/NFACompilerTest.java
@@ -44,7 +44,9 @@
 import java.util.Set;
 
 import static org.apache.flink.cep.utils.NFAUtils.compile;
+import static org.hamcrest.Matchers.is;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 
 /**
@@ -226,4 +228,14 @@ public boolean filter(Event value) throws Exception {
return transitions;
}
 
+   @Test
+   public void testCheckingEmptyMatches() {
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").optional()), 
is(true));
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").oneOrMore().optional()),
 is(true));
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").oneOrMore().optional().next("b").optional()),
 is(true));
+
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a")), is(false));
+   
assertThat(NFACompiler.canProduceEmptyMatches(Pattern.begin("a").oneOrMore()), 
is(false));
+   

[jira] [Commented] (FLINK-10088) error when trying to run flink-sql-client on windows

[xueyu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639764#comment-16639764
 ] 

xueyu commented on FLINK-10088:
---

hi, [~anubhavtarar], did you run sql-client.sh using the command 
" sh ./bin/sql-client.sh embedded"? I tried using this command on Windows 
gitbash, it report the same error as you report. But maybe you could try 
"./bin/sql-client.sh embedded", it looks that could avoid this error...

> error when trying to run flink-sql-client on windows
> 
>
> Key: FLINK-10088
> URL: https://issues.apache.org/jira/browse/FLINK-10088
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Affects Versions: 1.5.2
>Reporter: anubhav tarar
>Priority: Major
>
> When trying to run flink on windows it is giving me error
>  
> ./config.sh: line 32: syntax error near unexpected token `<'
> ./config.sh: line 32: `    done < <(find "$FLINK_LIB_DIR" ! -type d -name 
> '*.jar' -print0 | sort -z)'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10495) Add HYPOT math function supported in Table API and SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639700#comment-16639700
 ] 

ASF GitHub Bot commented on FLINK-10495:


aai95 opened a new pull request #6797: [FLINK-10495] [table] Add HYPOT math 
function supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6797
 
 
   ## What is the purpose of the change
   
   *This pull request adds `Hypot` math function supported in Table API and 
SQL.*
   
   ## Brief change log
   
 - *Added `Hypot` math function supported in Table API and SQL.*
   
   ## Verifying this change
   *This change added test `ScalarFunctionsTest#testHypot`.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**yes** / no)
 - If yes, how is the feature documented? (not applicable / **docs** / 
JavaDocs / not documented)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add HYPOT math function supported in Table API and SQL
> --
>
> Key: FLINK-10495
> URL: https://issues.apache.org/jira/browse/FLINK-10495
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Aleksei Izmalkin
>Priority: Minor
>  Labels: pull-request-available
>
> Implement the function to calculate the sqrt(_x_^2^ +_y_^2^).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] aai95 opened a new pull request #6797: [FLINK-10495] [table] Add HYPOT math function supported in Table API and SQL

[GitBox]

aai95 opened a new pull request #6797: [FLINK-10495] [table] Add HYPOT math 
function supported in Table API and SQL
URL: https://github.com/apache/flink/pull/6797
 
 
   ## What is the purpose of the change
   
   *This pull request adds `Hypot` math function supported in Table API and 
SQL.*
   
   ## Brief change log
   
 - *Added `Hypot` math function supported in Table API and SQL.*
   
   ## Verifying this change
   *This change added test `ScalarFunctionsTest#testHypot`.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**yes** / no)
 - If yes, how is the feature documented? (not applicable / **docs** / 
JavaDocs / not documented)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-9940) File source continuous monitoring mode: S3 files sometimes missed

[Huyen Levan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639672#comment-16639672
 ] 

Huyen Levan commented on FLINK-9940:


[~aljoscha] after having created a PR, do I need to do anything else? I could 
not find any guideline for this.

Thanks!

> File source continuous monitoring mode: S3 files sometimes missed
> -
>
> Key: FLINK-9940
> URL: https://issues.apache.org/jira/browse/FLINK-9940
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.5.1
> Environment: Flink 1.5, EMRFS
>Reporter: Huyen Levan
>Assignee: Huyen Levan
>Priority: Major
>  Labels: EMRFS, Flink, S3, pull-request-available
> Fix For: 1.7.0
>
>
> When using StreamExecutionEnvironment.readFile() with 
> FileProcessingMode.PROCESS_CONTINUOUSLY mode to monitor an S3 prefix, if 
> there is a high amount of new/modified files at the same time, the directory 
> monitoring process might miss some files. The number of missing files depends 
> on the monitoring interval.
> Cause: Flink tracks which files it has read by remembering the modification 
> time of the file that was added (or modified) last. So when there are 
> multiple files having a same last-modified timestamp.
> Suggested solution (thanks to [[Fabian 
> Hueske|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=user_nodes=25]|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=user_nodes=25]):
>  a hybrid approach that keeps the names of all files that have a mod 
> timestamp that is larger than the max mod time minus an offset. 
> _org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10205) Batch Job: InputSplit Fault tolerant for DataSourceTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639654#comment-16639654
 ] 

ASF GitHub Bot commented on FLINK-10205:


StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: 
InputSplit Fault tolerant for DataSource…
URL: https://github.com/apache/flink/pull/6684#issuecomment-427323755
 
 
   @isunjin Ok, I think I was just connecting a different type of problem with 
the word "corruption", but the case makes sense because the assumption so far 
was probably that there are only global failover and that the 
`InputSplitAssigner` is reset. I think in this context the implementation is 
ok, I still not the biggest fan of pulling the `InputSplit` concept into 
`Execution` but at the same time I see that currently all alternatives that 
came to my mind have their own set of problems. Guess it would be ok for me to 
merge the PR, but I would like to ask @tillrohrmann for a second opinion.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Batch Job: InputSplit Fault tolerant for DataSourceTask
> ---
>
> Key: FLINK-10205
> URL: https://issues.apache.org/jira/browse/FLINK-10205
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Affects Versions: 1.6.1, 1.7.0, 1.6.2
>Reporter: JIN SUN
>Assignee: JIN SUN
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Today DataSource Task pull InputSplits from JobManager to achieve better 
> performance, however, when a DataSourceTask failed and rerun, it will not get 
> the same splits as its previous version. this will introduce inconsistent 
> result or even data corruption.
> Furthermore,  if there are two executions run at the same time (in batch 
> scenario), this two executions should process same splits.
> we need to fix the issue to make the inputs of a DataSourceTask 
> deterministic. The propose is save all splits into ExecutionVertex and 
> DataSourceTask will pull split from there.
>  document:
> [https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource…

[GitBox]

StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: 
InputSplit Fault tolerant for DataSource…
URL: https://github.com/apache/flink/pull/6684#issuecomment-427323755
 
 
   @isunjin Ok, I think I was just connecting a different type of problem with 
the word "corruption", but the case makes sense because the assumption so far 
was probably that there are only global failover and that the 
`InputSplitAssigner` is reset. I think in this context the implementation is 
ok, I still not the biggest fan of pulling the `InputSplit` concept into 
`Execution` but at the same time I see that currently all alternatives that 
came to my mind have their own set of problems. Guess it would be ok for me to 
merge the PR, but I would like to ask @tillrohrmann for a second opinion.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-5405) Add streaming connection to twitter through a internet proxy

[Mariia Borovikova (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639628#comment-16639628
 ] 

Mariia Borovikova commented on FLINK-5405:
--

Can I work on this ticket?

> Add streaming connection to twitter through a internet proxy
> 
>
> Key: FLINK-5405
> URL: https://issues.apache.org/jira/browse/FLINK-5405
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Luis
>Priority: Minor
>  Labels: easyfix
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Add streaming connection to twitter through a internet proxy upgrading 
> hbc-core dependency to 2.2.1-SNAPSHOT and changing the code to pass proxy 
> configurations to com.twitter.hbc.ClientBuilder.proxy().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9328) RocksDBStateBackend might use PlaceholderStreamStateHandle to restor due to StateBackendTestBase class not register snapshots in some UTs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639603#comment-16639603
 ] 

ASF GitHub Bot commented on FLINK-9328:
---

StefanRRichter closed pull request #5984: [FLINK-9328][state] Fix 
RocksDBStateBackend restore problem due to StateBackendTestBase not registering 
snapshots
URL: https://github.com/apache/flink/pull/5984
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
index 2634268c947..5b7421217a1 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
@@ -1248,6 +1248,7 @@ public void testValueState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ValueStateDescriptor kvId = new 
ValueStateDescriptor<>("id", String.class);
 
@@ -1586,6 +1587,7 @@ public void testListState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ListStateDescriptor kvId = new 
ListStateDescriptor<>("id", String.class);
 
@@ -2035,6 +2037,7 @@ public void testReducingState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ReducingStateDescriptor kvId = new 
ReducingStateDescriptor<>("id", new AppendingReduce(), String.class);
 
@@ -2653,6 +2656,7 @@ public void testFoldingState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
FoldingStateDescriptor kvId = new 
FoldingStateDescriptor<>("id",
"Fold-Initial:",
@@ -2757,6 +2761,7 @@ public void testMapState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(StringSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
MapStateDescriptor kvId = new 
MapStateDescriptor<>("id", Integer.class, String.class);
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> RocksDBStateBackend might use PlaceholderStreamStateHandle to restor due to 
> StateBackendTestBase class not register snapshots in some UTs
> -
>
> Key: FLINK-9328
> URL: https://issues.apache.org/jira/browse/FLINK-9328
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.4.2
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.5.5
>
>
> Currently, StateBackendTestBase class does not register snapshots to 
> SharedStateRegistry in testValueState, testListState, testReducingState, 
> testFoldingState and testMapState UTs, 

[jira] [Commented] (FLINK-9328) RocksDBStateBackend might use PlaceholderStreamStateHandle to restor due to StateBackendTestBase class not register snapshots in some UTs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639602#comment-16639602
 ] 

ASF GitHub Bot commented on FLINK-9328:
---

StefanRRichter commented on issue #5984: [FLINK-9328][state] Fix 
RocksDBStateBackend restore problem due to StateBackendTestBase not registering 
snapshots
URL: https://github.com/apache/flink/pull/5984#issuecomment-427313984
 
 
   Outdated. This was already fixed in master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> RocksDBStateBackend might use PlaceholderStreamStateHandle to restor due to 
> StateBackendTestBase class not register snapshots in some UTs
> -
>
> Key: FLINK-9328
> URL: https://issues.apache.org/jira/browse/FLINK-9328
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.4.2
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.5.5
>
>
> Currently, StateBackendTestBase class does not register snapshots to 
> SharedStateRegistry in testValueState, testListState, testReducingState, 
> testFoldingState and testMapState UTs, which may cause RocksDBStateBackend to 
> restore from PlaceholderStreamStateHandle during the 2nd restore procedure if 
> one specific sst file both existed in the 1st snapshot and the 2nd snapshot 
> handle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] StefanRRichter closed pull request #5984: [FLINK-9328][state] Fix RocksDBStateBackend restore problem due to StateBackendTestBase not registering snapshots

[GitBox]

StefanRRichter closed pull request #5984: [FLINK-9328][state] Fix 
RocksDBStateBackend restore problem due to StateBackendTestBase not registering 
snapshots
URL: https://github.com/apache/flink/pull/5984
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
index 2634268c947..5b7421217a1 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java
@@ -1248,6 +1248,7 @@ public void testValueState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ValueStateDescriptor kvId = new 
ValueStateDescriptor<>("id", String.class);
 
@@ -1586,6 +1587,7 @@ public void testListState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ListStateDescriptor kvId = new 
ListStateDescriptor<>("id", String.class);
 
@@ -2035,6 +2037,7 @@ public void testReducingState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
ReducingStateDescriptor kvId = new 
ReducingStateDescriptor<>("id", new AppendingReduce(), String.class);
 
@@ -2653,6 +2656,7 @@ public void testFoldingState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(IntSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
FoldingStateDescriptor kvId = new 
FoldingStateDescriptor<>("id",
"Fold-Initial:",
@@ -2757,6 +2761,7 @@ public void testMapState() throws Exception {
CheckpointStreamFactory streamFactory = createStreamFactory();
SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
AbstractKeyedStateBackend backend = 
createKeyedBackend(StringSerializer.INSTANCE);
+   SharedStateRegistry sharedStateRegistry = new 
SharedStateRegistry();
 
MapStateDescriptor kvId = new 
MapStateDescriptor<>("id", Integer.class, String.class);
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] StefanRRichter commented on issue #5984: [FLINK-9328][state] Fix RocksDBStateBackend restore problem due to StateBackendTestBase not registering snapshots

[GitBox]

StefanRRichter commented on issue #5984: [FLINK-9328][state] Fix 
RocksDBStateBackend restore problem due to StateBackendTestBase not registering 
snapshots
URL: https://github.com/apache/flink/pull/5984#issuecomment-427313984
 
 
   Outdated. This was already fixed in master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-9752) Add an S3 RecoverableWriter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639594#comment-16639594
 ] 

ASF GitHub Bot commented on FLINK-9752:
---

kl0u commented on issue #6795: [FLINK-9752][s3-fs-connector] Add s3 recoverable 
writer.
URL: https://github.com/apache/flink/pull/6795#issuecomment-427311243
 
 
   R @igalshilman @pnowojski 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add an S3 RecoverableWriter
> ---
>
> Key: FLINK-9752
> URL: https://issues.apache.org/jira/browse/FLINK-9752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Reporter: Stephan Ewen
>Assignee: Kostas Kloudas
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> S3 offers persistence only when uploads are complete. That means at the end 
> of simple uploads and uploads of parts of a MultiPartUpload.
> We should implement a RecoverableWriter for S3 that does a MultiPartUpload 
> with a Part per checkpoint.
> Recovering the reader needs the MultiPartUploadID and the list of ETags of 
> previous parts.
> We need additional staging of data in Flink state to work around the fact that
>  - Parts in a MultiPartUpload must be at least 5MB
>  - Part sizes must be known up front. (Note that data can still be streamed 
> in the upload)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kl0u commented on issue #6795: [FLINK-9752][s3-fs-connector] Add s3 recoverable writer.

[GitBox]

kl0u commented on issue #6795: [FLINK-9752][s3-fs-connector] Add s3 recoverable 
writer.
URL: https://github.com/apache/flink/pull/6795#issuecomment-427311243
 
 
   R @igalshilman @pnowojski 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (FLINK-10279) Make jython limitations more obvious in documentation

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels resolved FLINK-10279.

Resolution: Fixed

Fixed via 982fa402a0a9c65356c5e89b25569ed7a8ca89a5

> Make jython limitations more obvious in documentation
> -
>
> Key: FLINK-10279
> URL: https://issues.apache.org/jira/browse/FLINK-10279
> Project: Flink
>  Issue Type: Task
>  Components: Documentation, Python API
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: pull-request-available
>
> The "Python Programming Guide (Streaming) Beta" at 
> [https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/python.html]
>  does not sufficiently highlight limitations of the API. It should probably 
> have a prominent disclaimer right at the top stating that this actually isn't 
> a "Python" API but Jython, which likely means that the user looking for a 
> solution to run native Python code won't be able to use many important 
> libraries, which is often the reason to look for Python support in first 
> place.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10279) Make jython limitations more obvious in documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639582#comment-16639582
 ] 

ASF GitHub Bot commented on FLINK-10279:


mxm closed pull request #6761: [FLINK-10279] [documentation] Make jython 
limitations more obvious in documentation.
URL: https://github.com/apache/flink/pull/6761
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/dev/stream/python.md b/docs/dev/stream/python.md
index af9c393b3ea..e7c2fb267ba 100644
--- a/docs/dev/stream/python.md
+++ b/docs/dev/stream/python.md
@@ -24,6 +24,10 @@ specific language governing permissions and limitations
 under the License.
 -->
 
+Attention *This API is based on Jython,
+which is not a full Python replacement and may restrict the libraries you are 
able
+to use with your application (see below for more information).*
+
 Analysis streaming programs in Flink are regular programs that implement 
transformations on
 streaming data sets (e.g., filtering, mapping, joining, grouping). The 
streaming data sets are initially
 created from certain sources (e.g., by reading files, or from collections).
@@ -50,7 +54,12 @@ existing Java streaming APIs.
 There are two main constraints for using Jython:
 
 * The latest Python supported version is 2.7
-* It is not straightforward to use Python C extensions
+* It is not straightforward to use Python C extensions, which may prevent use 
of some libraries
+
+(For more information please see 
.)
+
+One possible alternative for streaming that allows for native Python execution 
would be the [Apache Beam
+portability framework](https://beam.apache.org/contribute/portability/) with 
the Flink runner.
 
 Streaming Program Example
 -


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make jython limitations more obvious in documentation
> -
>
> Key: FLINK-10279
> URL: https://issues.apache.org/jira/browse/FLINK-10279
> Project: Flink
>  Issue Type: Task
>  Components: Documentation, Python API
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: pull-request-available
>
> The "Python Programming Guide (Streaming) Beta" at 
> [https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/python.html]
>  does not sufficiently highlight limitations of the API. It should probably 
> have a prominent disclaimer right at the top stating that this actually isn't 
> a "Python" API but Jython, which likely means that the user looking for a 
> solution to run native Python code won't be able to use many important 
> libraries, which is often the reason to look for Python support in first 
> place.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] mxm closed pull request #6761: [FLINK-10279] [documentation] Make jython limitations more obvious in documentation.

[GitBox]

mxm closed pull request #6761: [FLINK-10279] [documentation] Make jython 
limitations more obvious in documentation.
URL: https://github.com/apache/flink/pull/6761
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/dev/stream/python.md b/docs/dev/stream/python.md
index af9c393b3ea..e7c2fb267ba 100644
--- a/docs/dev/stream/python.md
+++ b/docs/dev/stream/python.md
@@ -24,6 +24,10 @@ specific language governing permissions and limitations
 under the License.
 -->
 
+Attention *This API is based on Jython,
+which is not a full Python replacement and may restrict the libraries you are 
able
+to use with your application (see below for more information).*
+
 Analysis streaming programs in Flink are regular programs that implement 
transformations on
 streaming data sets (e.g., filtering, mapping, joining, grouping). The 
streaming data sets are initially
 created from certain sources (e.g., by reading files, or from collections).
@@ -50,7 +54,12 @@ existing Java streaming APIs.
 There are two main constraints for using Jython:
 
 * The latest Python supported version is 2.7
-* It is not straightforward to use Python C extensions
+* It is not straightforward to use Python C extensions, which may prevent use 
of some libraries
+
+(For more information please see 
.)
+
+One possible alternative for streaming that allows for native Python execution 
would be the [Apache Beam
+portability framework](https://beam.apache.org/contribute/portability/) with 
the Flink runner.
 
 Streaming Program Example
 -


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-10421:
-

Assignee: Till Rohrmann

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639534#comment-16639534
 ] 

Till Rohrmann commented on FLINK-10421:
---

The problem seems to be related to this string {{2018-09-25 12:55:31,783 INFO  
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
  - Error when creating PropertyDescriptor for public final void 
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
 Ignoring this property.}}. 

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639529#comment-16639529
 ] 

Till Rohrmann commented on FLINK-10421:
---

Another instance: https://api.travis-ci.org/v3/job/437354816/log.txt

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[tison (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639508#comment-16639508
 ] 

tison commented on FLINK-10454:
---

https://travis-ci.org/apache/flink/jobs/434894763
https://api.travis-ci.org/v3/job/434894763/log.txt

https://travis-ci.org/apache/flink/jobs/434894758
https://api.travis-ci.org/v3/job/434894758/log.txt

These are on a pull request branch, but since it can be reproduced locally and 
the pr has nothing to do with it I file this issue.

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10427) Port JobSubmitTest to new code base

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639513#comment-16639513
 ] 

ASF GitHub Bot commented on FLINK-10427:


TisonKun commented on issue #6768:  [FLINK-10427] [tests] Port JobSubmitTest to 
new code base
URL: https://github.com/apache/flink/pull/6768#issuecomment-427297515
 
 
   Thanks for your review @tillrohrmann !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Port JobSubmitTest to new code base
> ---
>
> Key: FLINK-10427
> URL: https://issues.apache.org/jira/browse/FLINK-10427
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Assignee: tison
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Port {{JobSubmitTest}} to new code base.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-10421:
--
Fix Version/s: 1.7.0

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10421) Shaded Hadoop S3A end-to-end test failed on Travis

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-10421:
--
Priority: Critical  (was: Major)

> Shaded Hadoop S3A end-to-end test failed on Travis
> --
>
> Key: FLINK-10421
> URL: https://issues.apache.org/jira/browse/FLINK-10421
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> https://api.travis-ci.org/v3/job/432916761/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10075) HTTP connections to a secured REST endpoint flood the log

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-10075:
---
Labels: pull-request-available  (was: )

> HTTP connections to a secured REST endpoint flood the log
> -
>
> Key: FLINK-10075
> URL: https://issues.apache.org/jira/browse/FLINK-10075
> Project: Flink
>  Issue Type: Improvement
>  Components: REST
>Affects Versions: 1.5.2, 1.6.0, 1.7.0
>Reporter: Stephan Ewen
>Assignee: Andrey Zagrebin
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> When connecting with a browser (or other client tool) to a secured REST 
> endpoint, the decoder throws many exceptions indicating that the received 
> data is not an SSL record.
> This massively floods the log, drowning out everything else (see below).
> h2. Proposed Solution
> If a {{NotSslRecordException}} is caught, Netty should send a response HTTP 
> 301 with a new location of https://host:port/
> The response would need to bypass the SSL handler because it must come in 
> plain text.
> h2. Fallback Solution
> If the proper solution cannot work, we should reduce the log level for that 
> particular exception to *TRACE*.
> h2. Sample Log Output
> Log message that is written per each request (there are many per web UI page)
> {code}
> 2018-08-06 19:07:57,734 WARN  
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint- Unhandled 
> exception
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException: 
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException: 
> not an SSL/TLS record: 474554
> 202f7061727469616c732f6f766572766965772e68746d6c20485454502f312e310d0a486f73743a206c6f63616c686f73743a383038310d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a4163636570743a
> 20746578742f68746d6c0d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f363429204170706c655765624b69742f3533372e333620284b48544d4c2c206c696b6520
> 4765636b6f29204368726f6d652f34372e302e323532362e313131205361666172692f3533372e33360d0a526566657265723a20687474703a2f2f6c6f63616c686f73743a383038312f0d0a4163636570742d456e636f64
> 696e673a20677a69702c206465666c6174652c20736463680d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e382c64653b713d302e360d0a49662d4d6f6469666965642d53696e63653a204d
> 6f6e2c2030362041756720323031382031353a34343a313720474d540d0a0d0a
> at 
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
> at 
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] TisonKun commented on issue #6768: [FLINK-10427] [tests] Port JobSubmitTest to new code base

[GitBox]

TisonKun commented on issue #6768:  [FLINK-10427] [tests] Port JobSubmitTest to 
new code base
URL: https://github.com/apache/flink/pull/6768#issuecomment-427297515
 
 
   Thanks for your review @tillrohrmann !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[tison (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639508#comment-16639508
 ] 

tison edited comment on FLINK-10454 at 10/5/18 9:07 AM:


https://travis-ci.org/apache/flink/jobs/434894763
https://api.travis-ci.org/v3/job/434894763/log.txt

https://travis-ci.org/apache/flink/jobs/434894758
https://api.travis-ci.org/v3/job/434894758/log.txt

These are on a pull request branch, but since it can be reproduced locally(on 
master branch) and the pr has nothing to do with it I file this issue.


was (Author: tison):
https://travis-ci.org/apache/flink/jobs/434894763
https://api.travis-ci.org/v3/job/434894763/log.txt

https://travis-ci.org/apache/flink/jobs/434894758
https://api.travis-ci.org/v3/job/434894758/log.txt

These are on a pull request branch, but since it can be reproduced locally and 
the pr has nothing to do with it I file this issue.

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10075) HTTP connections to a secured REST endpoint flood the log

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639500#comment-16639500
 ] 

ASF GitHub Bot commented on FLINK-10075:


azagrebin opened a new pull request #6796: [FLINK-10075] Redirect non-ssl 
requests to https url if ssl is enabled
URL: https://github.com/apache/flink/pull/6796
 
 
   ## What is the purpose of the change
   
   This PR introduces a wrapper around SSL handler in rest server endpoint. 
   The wrapper checks whether the request is encrypted and processes normally 
only encrypted requests.
   The non-encrypted requests are redirected to advertised host with https 
protocol.
   The path is added to the redirect url if the request is http and the path is 
available.
   
   ## Brief change log
   
 - add RedirectingSslHandler wrapper for SslHandler with redirecting in 
newNonSslHandler()
 - replace SslHandler in RestServerEndpoint with RedirectingSslHandler
 - add unit test  checking the redirection 
RestServerEndpointITCase.testNonSslRedirectForEnabledSsl()
   
   ## Verifying this change
   
   unit tests, try access Flink UI with http url and ssl enabled in flink.conf
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> HTTP connections to a secured REST endpoint flood the log
> -
>
> Key: FLINK-10075
> URL: https://issues.apache.org/jira/browse/FLINK-10075
> Project: Flink
>  Issue Type: Improvement
>  Components: REST
>Affects Versions: 1.5.2, 1.6.0, 1.7.0
>Reporter: Stephan Ewen
>Assignee: Andrey Zagrebin
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> When connecting with a browser (or other client tool) to a secured REST 
> endpoint, the decoder throws many exceptions indicating that the received 
> data is not an SSL record.
> This massively floods the log, drowning out everything else (see below).
> h2. Proposed Solution
> If a {{NotSslRecordException}} is caught, Netty should send a response HTTP 
> 301 with a new location of https://host:port/
> The response would need to bypass the SSL handler because it must come in 
> plain text.
> h2. Fallback Solution
> If the proper solution cannot work, we should reduce the log level for that 
> particular exception to *TRACE*.
> h2. Sample Log Output
> Log message that is written per each request (there are many per web UI page)
> {code}
> 2018-08-06 19:07:57,734 WARN  
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint- Unhandled 
> exception
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException: 
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException: 
> not an SSL/TLS record: 474554
> 202f7061727469616c732f6f766572766965772e68746d6c20485454502f312e310d0a486f73743a206c6f63616c686f73743a383038310d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a4163636570743a
> 20746578742f68746d6c0d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f363429204170706c655765624b69742f3533372e333620284b48544d4c2c206c696b6520
> 4765636b6f29204368726f6d652f34372e302e323532362e313131205361666172692f3533372e33360d0a526566657265723a20687474703a2f2f6c6f63616c686f73743a383038312f0d0a4163636570742d456e636f64
> 696e673a20677a69702c206465666c6174652c20736463680d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e382c64653b713d302e360d0a49662d4d6f6469666965642d53696e63653a204d
> 6f6e2c2030362041756720323031382031353a34343a313720474d540d0a0d0a
> at 
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
> at 
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at 
> 

[GitHub] azagrebin opened a new pull request #6796: [FLINK-10075] Redirect non-ssl requests to https url if ssl is enabled

[GitBox]

azagrebin opened a new pull request #6796: [FLINK-10075] Redirect non-ssl 
requests to https url if ssl is enabled
URL: https://github.com/apache/flink/pull/6796
 
 
   ## What is the purpose of the change
   
   This PR introduces a wrapper around SSL handler in rest server endpoint. 
   The wrapper checks whether the request is encrypted and processes normally 
only encrypted requests.
   The non-encrypted requests are redirected to advertised host with https 
protocol.
   The path is added to the redirect url if the request is http and the path is 
available.
   
   ## Brief change log
   
 - add RedirectingSslHandler wrapper for SslHandler with redirecting in 
newNonSslHandler()
 - replace SslHandler in RestServerEndpoint with RedirectingSslHandler
 - add unit test  checking the redirection 
RestServerEndpointITCase.testNonSslRedirectForEnabledSsl()
   
   ## Verifying this change
   
   unit tests, try access Flink UI with http url and ssl enabled in flink.conf
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10454) Travis fails on ScheduleOrUpdateConsumersTest

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639491#comment-16639491
 ] 

Till Rohrmann commented on FLINK-10454:
---

[~Tison] could you link the maven logs where you encountered the problem? They 
help us by stating the commit revision.

> Travis fails on ScheduleOrUpdateConsumersTest
> -
>
> Key: FLINK-10454
> URL: https://issues.apache.org/jira/browse/FLINK-10454
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.7.0
>Reporter: tison
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.7.0
>
>
> Can even be reproduced locally. Maybe a duplicate but as a reminder.
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest 
> Time elapsed: 4.514 sec <<< ERROR! java.net.BindException: Address already in 
> use at sun.nio.ch.Net.bind0(Native Method) at 
> sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
>  at 
> org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
>  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10289) Classify Exceptions to different category for apply different failover strategy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639445#comment-16639445
 ] 

ASF GitHub Bot commented on FLINK-10289:


StefanRRichter commented on issue #6739: [FLINK-10289] [JobManager] Classify 
Exceptions to different category for apply different failover strategy
URL: https://github.com/apache/flink/pull/6739#issuecomment-427281153
 
 
   @isunjin I think this looks good now. I wonder if we should rename the type 
`Other` to something like `RecoverableFailure`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Classify Exceptions to different category for apply different failover 
> strategy
> ---
>
> Key: FLINK-10289
> URL: https://issues.apache.org/jira/browse/FLINK-10289
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: JIN SUN
>Assignee: JIN SUN
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> We need to classify exceptions and treat them with different strategies. To 
> do this, we propose to introduce the following Throwable Types, and the 
> corresponding exceptions:
>  * NonRecoverable
>  ** We shouldn’t retry if an exception was classified as NonRecoverable
>  ** For example, NoResouceAvailiableException is a NonRecoverable Exception
>  ** Introduce a new Exception UserCodeException to wrap all exceptions that 
> throw from user code
>  * PartitionDataMissingError
>  ** In certain scenarios producer data was transferred in blocking mode or 
> data was saved in persistent store. If the partition was missing, we need to 
> revoke/rerun the produce task to regenerate the data.
>  ** Introduce a new exception PartitionDataMissingException to wrap all those 
> kinds of issues.
>  * EnvironmentError
>  ** It happened due to hardware, or software issues that were related to 
> specific environments. The assumption is that a task will succeed if we run 
> it in a different environment, and other task run in this bad environment 
> will very likely fail. If multiple task failures in the same machine due to 
> EnvironmentError, we need to consider adding the bad machine to blacklist, 
> and avoiding schedule task on it.
>  ** Introduce a new exception EnvironmentException to wrap all those kind of 
> issues.
>  * Recoverable
>  ** We assume other issues are recoverable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] StefanRRichter commented on issue #6739: [FLINK-10289] [JobManager] Classify Exceptions to different category for apply different failover strategy

[GitBox]

StefanRRichter commented on issue #6739: [FLINK-10289] [JobManager] Classify 
Exceptions to different category for apply different failover strategy
URL: https://github.com/apache/flink/pull/6739#issuecomment-427281153
 
 
   @isunjin I think this looks good now. I wonder if we should rename the type 
`Other` to something like `RecoverableFailure`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10487) fix invalid Flink SQL example

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639402#comment-16639402
 ] 

ASF GitHub Bot commented on FLINK-10487:


fhueske commented on issue #6790: [FLINK-10487] [table] fix invalid Flink SQL 
example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#issuecomment-427271829
 
 
   Thanks for the update @bowenli86!
   
   +1 to merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> fix invalid Flink SQL example
> -
>
> Key: FLINK-10487
> URL: https://issues.apache.org/jira/browse/FLINK-10487
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation, Table API  SQL
>Affects Versions: 1.6.0, 1.6.1, 1.7.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> On https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html 
> , the example is :
> {code:java}
> StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
> // ingest a DataStream from an external source
> DataStream> ds = env.addSource(...);
> // SQL query with an inlined (unregistered) table
> Table table = tableEnv.toTable(ds, "user, product, amount");
> Table result = tableEnv.sqlQuery(
>   "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
> ...
> {code}
> But there's no API of {{toTable()}} in StreamTableEnvironment 
> cc [~fhueske]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9752) Add an S3 RecoverableWriter

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-9752:
--
Labels: pull-request-available  (was: )

> Add an S3 RecoverableWriter
> ---
>
> Key: FLINK-9752
> URL: https://issues.apache.org/jira/browse/FLINK-9752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Reporter: Stephan Ewen
>Assignee: Kostas Kloudas
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> S3 offers persistence only when uploads are complete. That means at the end 
> of simple uploads and uploads of parts of a MultiPartUpload.
> We should implement a RecoverableWriter for S3 that does a MultiPartUpload 
> with a Part per checkpoint.
> Recovering the reader needs the MultiPartUploadID and the list of ETags of 
> previous parts.
> We need additional staging of data in Flink state to work around the fact that
>  - Parts in a MultiPartUpload must be at least 5MB
>  - Part sizes must be known up front. (Note that data can still be streamed 
> in the upload)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] kl0u opened a new pull request #6795: [FLINK-9752][s3-fs-connector] Add s3 recoverable writer.

[GitBox]

kl0u opened a new pull request #6795: [FLINK-9752][s3-fs-connector] Add s3 
recoverable writer.
URL: https://github.com/apache/flink/pull/6795
 
 
   ## What is the purpose of the change
   
   Adds the recoverable writer for S3. The new recoverable writer is only 
available for **Hadoop S3**
   (**not Presto** for now) and uses the MultiPart feature to upload part files.
   
   The user is supposed to call `fs.createRecoverableWriter()`, which will give 
back an `S3RecoverableWriter`. This allows to: `open(Path)` which give an 
`S3RecoverableFsDataOutputStream` or call `recover()` a previous such stream 
from a checkpoint.
   
   The main functionality is implemented by the 
`S3RecoverableFsDataOutputStream`. This uses:
   1) a `RefCountedFSOutputStream` - a file stream backed by a local tmp file 
which is reference counted so when there are no references it gets deleted - to 
write a part of the multi-part upload.
   2) a `RecoverableMultiPartUpload` to take snapshots of in-flight Multi-Part 
Uploads (MPU) and upload already ready parts.
   3) From the stream, the user can also get a `Committer` which allows him to 
complete the MPU, i.e. "publish" the data. 
   
   The whole process is a two-phase commit, with files being staged for commit, 
and then committed as a unit.
   
   **Checkpointing / Recovery**
   
   As the user writes data to the stream, when the part reaches a minimum size, 
it gets uploaded to S3, and a new part-file is opened. An uploaded part is 
identified by its `PartETag` which is further used when "committing" the MPU. 
So the list of `PartETag` 's associated with the MPU are stored in state. 
   
   When `persist`  is called, the "current" part file which has not yet reached 
the minimum size is uploaded to S3 as an independent object (not as part of the 
MPU), and its handle is stored in state. Apart from that, we keep writing to 
the same part file.
   
   Upon recovery, we retrieve the set of valid `PartEtags` from the state and 
we download the "in-progress" part file, which was uploaded as an independent 
object, and we resume writing from there. Any uploaded `PartEtags` between the 
last successful checkpoint and the failure are simply discarded. This 
alleviates any need for active `truncating`.
   
   ## Brief change log
   
   This PR consists of mainly new code so the changelog is not much help. The 
reviewer can find above a description of what the code does.
   
   ## Verifying this change
   
   This change added Unit Tests and can be verified as follows:
   * `RefCountedBufferingFileStreamTest`
   * `RefCountedFileTest`
   * `IncompletePartPrefixTest`
   * `S3RecoverableFsDataOutputStreamTest`
   
   And semi end-to-end test against actually S3:
   * `HadoopS3RecoverableWriterTest`
   * `HadoopS3RecoverableWriterExceptionTest`
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / n**o** / don't know)
 - The S3 file system connector: (**yes** / no / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**yes** / no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ **not documented**) yet
   
   R @pnowojski 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-9752) Add an S3 RecoverableWriter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639420#comment-16639420
 ] 

ASF GitHub Bot commented on FLINK-9752:
---

kl0u opened a new pull request #6795: [FLINK-9752][s3-fs-connector] Add s3 
recoverable writer.
URL: https://github.com/apache/flink/pull/6795
 
 
   ## What is the purpose of the change
   
   Adds the recoverable writer for S3. The new recoverable writer is only 
available for **Hadoop S3**
   (**not Presto** for now) and uses the MultiPart feature to upload part files.
   
   The user is supposed to call `fs.createRecoverableWriter()`, which will give 
back an `S3RecoverableWriter`. This allows to: `open(Path)` which give an 
`S3RecoverableFsDataOutputStream` or call `recover()` a previous such stream 
from a checkpoint.
   
   The main functionality is implemented by the 
`S3RecoverableFsDataOutputStream`. This uses:
   1) a `RefCountedFSOutputStream` - a file stream backed by a local tmp file 
which is reference counted so when there are no references it gets deleted - to 
write a part of the multi-part upload.
   2) a `RecoverableMultiPartUpload` to take snapshots of in-flight Multi-Part 
Uploads (MPU) and upload already ready parts.
   3) From the stream, the user can also get a `Committer` which allows him to 
complete the MPU, i.e. "publish" the data. 
   
   The whole process is a two-phase commit, with files being staged for commit, 
and then committed as a unit.
   
   **Checkpointing / Recovery**
   
   As the user writes data to the stream, when the part reaches a minimum size, 
it gets uploaded to S3, and a new part-file is opened. An uploaded part is 
identified by its `PartETag` which is further used when "committing" the MPU. 
So the list of `PartETag` 's associated with the MPU are stored in state. 
   
   When `persist`  is called, the "current" part file which has not yet reached 
the minimum size is uploaded to S3 as an independent object (not as part of the 
MPU), and its handle is stored in state. Apart from that, we keep writing to 
the same part file.
   
   Upon recovery, we retrieve the set of valid `PartEtags` from the state and 
we download the "in-progress" part file, which was uploaded as an independent 
object, and we resume writing from there. Any uploaded `PartEtags` between the 
last successful checkpoint and the failure are simply discarded. This 
alleviates any need for active `truncating`.
   
   ## Brief change log
   
   This PR consists of mainly new code so the changelog is not much help. The 
reviewer can find above a description of what the code does.
   
   ## Verifying this change
   
   This change added Unit Tests and can be verified as follows:
   * `RefCountedBufferingFileStreamTest`
   * `RefCountedFileTest`
   * `IncompletePartPrefixTest`
   * `S3RecoverableFsDataOutputStreamTest`
   
   And semi end-to-end test against actually S3:
   * `HadoopS3RecoverableWriterTest`
   * `HadoopS3RecoverableWriterExceptionTest`
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / n**o** / don't know)
 - The S3 file system connector: (**yes** / no / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**yes** / no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ **not documented**) yet
   
   R @pnowojski 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add an S3 RecoverableWriter
> ---
>
> Key: FLINK-9752
> URL: https://issues.apache.org/jira/browse/FLINK-9752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Reporter: Stephan Ewen
>Assignee: Kostas Kloudas
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2
>
>
> S3 offers persistence only when uploads are complete. That means at the end 
> of simple uploads and uploads of parts of a MultiPartUpload.
> We should implement a RecoverableWriter for S3 that does a MultiPartUpload 
> with a Part per checkpoint.
> Recovering the reader needs the MultiPartUploadID and the 

[jira] [Commented] (FLINK-10495) Add HYPOT math function supported in Table API and SQL

[Hequn Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639411#comment-16639411
 ] 

Hequn Cheng commented on FLINK-10495:
-

[~Aleksei_Izmalkin] Done. Thanks for your contributions.

> Add HYPOT math function supported in Table API and SQL
> --
>
> Key: FLINK-10495
> URL: https://issues.apache.org/jira/browse/FLINK-10495
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Aleksei Izmalkin
>Priority: Minor
>  Labels: pull-request-available
>
> Implement the function to calculate the sqrt(_x_^2^ +_y_^2^).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10495) Add HYPOT math function supported in Table API and SQL

[Aleksei Izmalkin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639376#comment-16639376
 ] 

Aleksei Izmalkin commented on FLINK-10495:
--

I will work on this issue.
Hello [~yanghua], please mark this issue as a sub-task of FLINK-6810.
I still can't select this type from the drop-down list...

> Add HYPOT math function supported in Table API and SQL
> --
>
> Key: FLINK-10495
> URL: https://issues.apache.org/jira/browse/FLINK-10495
> Project: Flink
>  Issue Type: Task
>  Components: Table API  SQL
>Reporter: Aleksei Izmalkin
>Priority: Minor
>  Labels: pull-request-available
>
> Implement the function to calculate the sqrt(_x_^2^ +_y_^2^).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10495) Add HYPOT math function supported in Table API and SQL

[Hequn Cheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hequn Cheng updated FLINK-10495:

Issue Type: Sub-task  (was: Task)
Parent: FLINK-6810

> Add HYPOT math function supported in Table API and SQL
> --
>
> Key: FLINK-10495
> URL: https://issues.apache.org/jira/browse/FLINK-10495
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Aleksei Izmalkin
>Priority: Minor
>  Labels: pull-request-available
>
> Implement the function to calculate the sqrt(_x_^2^ +_y_^2^).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] fhueske commented on issue #6790: [FLINK-10487] [table] fix invalid Flink SQL example and add runnable SQL example for Java API

[GitBox]

fhueske commented on issue #6790: [FLINK-10487] [table] fix invalid Flink SQL 
example and add runnable SQL example for Java API
URL: https://github.com/apache/flink/pull/6790#issuecomment-427271829
 
 
   Thanks for the update @bowenli86!
   
   +1 to merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services