[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427909#comment-16427909
 ] 

ASF GitHub Bot commented on FLINK-8335:
---

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5488
  
I put this upgrade dependency tree here for a reference.

```[INFO] -< org.apache.flink:flink-hbase_2.11 
>--
[INFO] Building flink-hbase 1.6-SNAPSHOT
[INFO] [ jar 
]-
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ 
flink-hbase_2.11 ---
[INFO] org.apache.flink:flink-hbase_2.11:jar:1.6-SNAPSHOT
[INFO] +- org.apache.flink:flink-core:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-annotations:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-shaded-asm:jar:5.0.4-2.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:provided
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:provided
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:provided
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:provided
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] +- org.apache.flink:flink-java:jar:1.6-SNAPSHOT:provided
[INFO] |  \- org.apache.commons:commons-math3:jar:3.5:provided
[INFO] +- org.apache.flink:flink-scala_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:provided
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.12:provided
[INFO] |  \- org.scala-lang:scala-compiler:jar:2.11.12:provided
[INFO] | +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.5:provided
[INFO] | \- 
org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:provided
[INFO] +- 
org.apache.flink:flink-streaming-scala_2.11:jar:1.6-SNAPSHOT:provided
[INFO] +- org.apache.flink:flink-table_2.11:jar:1.6-SNAPSHOT:provided
[INFO] +- org.apache.flink:flink-shaded-hadoop2:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |  +- xmlenc:xmlenc:jar:0.52:provided
[INFO] |  +- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:provided
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:provided
[INFO] |  +- commons-el:commons-el:jar:1.0:provided
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-configuration:commons-configuration:jar:1.7:provided
[INFO] |  +- commons-digester:commons-digester:jar:1.8.1:provided
[INFO] |  +- com.jcraft:jsch:jar:0.1.42:provided
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.10:compile
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.3:provided
[INFO] |  +- commons-daemon:commons-daemon:jar:1.0.13:provided
[INFO] |  +- com.sun.jersey:jersey-client:jar:1.9:provided
[INFO] |  +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] |  +- javax.xml.stream:stax-api:jar:1.0-2:provided
[INFO] |  \- javax.activation:activation:jar:1.1:provided
[INFO] +- 
org.apache.flink:flink-streaming-java_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-runtime_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- 
org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- org.apache.flink:flink-hadoop-fs:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- 
org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-2.0:provided
[INFO] |  |  +- org.apache.flink:flink-shaded-jackson:jar:2.7.9-3.0:provided
[INFO] |  |  +- org.javassist:javassist:jar:3.18.2-GA:provided
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.11:jar:2.4.20:provided
[INFO] |  |  |  +- com.typesafe:config:jar:1.3.0:provided
[INFO] |  |  |  \- 
org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0:provided
[INFO] |  |  +- com.typesafe.akka:akka-stream_2.11:jar:2.4.20:provided
[INFO] |  |  |  +- org.reactivestreams:reactive-streams:jar:1.0.0:provided
[INFO] |  |  |  \- com.typesafe:ssl-config-core_2.11:jar:0.2.1:provided
[INFO] |  |  +- com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20:provided
[INFO] |  |  +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20:provided

[GitHub] flink issue #5488: [FLINK-8335] [hbase] Upgrade hbase connector dependency t...

[zhangminglei]

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5488
  
I put this upgrade dependency tree here for a reference.

```[INFO] -< org.apache.flink:flink-hbase_2.11 
>--
[INFO] Building flink-hbase 1.6-SNAPSHOT
[INFO] [ jar 
]-
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ 
flink-hbase_2.11 ---
[INFO] org.apache.flink:flink-hbase_2.11:jar:1.6-SNAPSHOT
[INFO] +- org.apache.flink:flink-core:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-annotations:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-shaded-asm:jar:5.0.4-2.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:provided
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:provided
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:provided
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:provided
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] +- org.apache.flink:flink-java:jar:1.6-SNAPSHOT:provided
[INFO] |  \- org.apache.commons:commons-math3:jar:3.5:provided
[INFO] +- org.apache.flink:flink-scala_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:provided
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.12:provided
[INFO] |  \- org.scala-lang:scala-compiler:jar:2.11.12:provided
[INFO] | +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.5:provided
[INFO] | \- 
org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:provided
[INFO] +- 
org.apache.flink:flink-streaming-scala_2.11:jar:1.6-SNAPSHOT:provided
[INFO] +- org.apache.flink:flink-table_2.11:jar:1.6-SNAPSHOT:provided
[INFO] +- org.apache.flink:flink-shaded-hadoop2:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |  +- xmlenc:xmlenc:jar:0.52:provided
[INFO] |  +- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:provided
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:provided
[INFO] |  +- commons-el:commons-el:jar:1.0:provided
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-configuration:commons-configuration:jar:1.7:provided
[INFO] |  +- commons-digester:commons-digester:jar:1.8.1:provided
[INFO] |  +- com.jcraft:jsch:jar:0.1.42:provided
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.10:compile
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.3:provided
[INFO] |  +- commons-daemon:commons-daemon:jar:1.0.13:provided
[INFO] |  +- com.sun.jersey:jersey-client:jar:1.9:provided
[INFO] |  +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] |  +- javax.xml.stream:stax-api:jar:1.0-2:provided
[INFO] |  \- javax.activation:activation:jar:1.1:provided
[INFO] +- 
org.apache.flink:flink-streaming-java_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  +- org.apache.flink:flink-runtime_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- 
org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- org.apache.flink:flink-hadoop-fs:jar:1.6-SNAPSHOT:provided
[INFO] |  |  +- 
org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-2.0:provided
[INFO] |  |  +- org.apache.flink:flink-shaded-jackson:jar:2.7.9-3.0:provided
[INFO] |  |  +- org.javassist:javassist:jar:3.18.2-GA:provided
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.11:jar:2.4.20:provided
[INFO] |  |  |  +- com.typesafe:config:jar:1.3.0:provided
[INFO] |  |  |  \- 
org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0:provided
[INFO] |  |  +- com.typesafe.akka:akka-stream_2.11:jar:2.4.20:provided
[INFO] |  |  |  +- org.reactivestreams:reactive-streams:jar:1.0.0:provided
[INFO] |  |  |  \- com.typesafe:ssl-config-core_2.11:jar:0.2.1:provided
[INFO] |  |  +- com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20:provided
[INFO] |  |  +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20:provided
[INFO] |  |  +- org.clapper:grizzled-slf4j_2.11:jar:1.0.2:provided
[INFO] |  |  +- com.github.scopt:scopt_2.11:jar:3.5.0:provided
[INFO] |  |  \- com.twitter:chill_2.11:jar:0.7.4:provided
[INFO] |  | \- 

[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427873#comment-16427873
 ] 

ASF GitHub Bot commented on FLINK-9068:
---

GitHub user shankarganesh1234 opened a pull request:

https://github.com/apache/flink/pull/5824

[FLINK-9068][docs] Fix for removing extra '' tag

## What is the purpose of the change
Removing an unclosed paragraph tag

## Brief change log
Closed the paragraph tags appropriately

## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:
none 

## Documentation
none


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shankarganesh1234/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5824


commit 2f13b6507b2a6bb5e39de8e0509e838c549f285f
Author: shankarganesh1234 
Date:   2018-04-06T02:42:13Z

[FLINK-9068][docs] Fix for removing extra '' tag

commit 29c2756c5cb80a444a5c5b14631bfc98f6d86681
Author: shankarganesh1234 
Date:   2018-04-06T02:56:44Z

[FLINK-9068][docs] Fix for removing extra '' tag




> Website documentation issue - html tag visible on screen
> 
>
> Key: FLINK-9068
> URL: https://issues.apache.org/jira/browse/FLINK-9068
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: SHANKAR GANESH
>Assignee: SHANKAR GANESH
>Priority: Minor
> Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png
>
>
> In the documentation at the following url
> [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning]
> In the section which explains the 'Reduce' operator (*Reduce*
> KeyedStream → DataStream), an html tag () is visible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5824: [FLINK-9068][docs] Fix for removing extra '' t...

[shankarganesh1234]

GitHub user shankarganesh1234 opened a pull request:

https://github.com/apache/flink/pull/5824

[FLINK-9068][docs] Fix for removing extra '' tag

## What is the purpose of the change
Removing an unclosed paragraph tag

## Brief change log
Closed the paragraph tags appropriately

## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:
none 

## Documentation
none


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shankarganesh1234/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5824


commit 2f13b6507b2a6bb5e39de8e0509e838c549f285f
Author: shankarganesh1234 
Date:   2018-04-06T02:42:13Z

[FLINK-9068][docs] Fix for removing extra '' tag

commit 29c2756c5cb80a444a5c5b14631bfc98f6d86681
Author: shankarganesh1234 
Date:   2018-04-06T02:56:44Z

[FLINK-9068][docs] Fix for removing extra '' tag




---

[jira] [Updated] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-8335:

Summary: Upgrade hbase connector dependency to 1.4.3  (was: Upgrade hbase 
connector dependency to 1.4.1)

> Upgrade hbase connector dependency to 1.4.3
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.1

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427834#comment-16427834
 ] 

mingleizhang commented on FLINK-8335:
-

Thanks [~yuzhih...@gmail.com]. I will apply 1.4.3 this to here today soon.

> Upgrade hbase connector dependency to 1.4.1
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8335) Upgrade hbase connector dependency to 1.4.1

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang reassigned FLINK-8335:
---

Assignee: mingleizhang

> Upgrade hbase connector dependency to 1.4.1
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427664#comment-16427664
 ] 

Ted Yu commented on FLINK-8037:
---

Please rebase PR.

> Missing cast in integer arithmetic in 
> TransactionalIdsGenerator#generateIdsToAbort
> --
>
> Key: FLINK-8037
> URL: https://issues.apache.org/jira/browse/FLINK-8037
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Greg Hogan
>Priority: Minor
>  Labels: kafka-connect
>
> {code}
>   public Set generateIdsToAbort() {
> Set idsToAbort = new HashSet<>();
> for (int i = 0; i < safeScaleDownFactor; i++) {
>   idsToAbort.addAll(generateIdsToUse(i * poolSize * 
> totalNumberOfSubtasks));
> {code}
> The operands are integers where generateIdsToUse() expects long parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-8335) Upgrade hbase connector dependency to 1.4.1

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427636#comment-16427636
 ] 

Ted Yu edited comment on FLINK-8335 at 4/5/18 10:05 PM:


Actually 1.4.3 has been released.


was (Author: yuzhih...@gmail.com):
+1 on upgrading to 1.4.2

> Upgrade hbase connector dependency to 1.4.1
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.1

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427636#comment-16427636
 ] 

Ted Yu commented on FLINK-8335:
---

+1 on upgrading to 1.4.2

> Upgrade hbase connector dependency to 1.4.1
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9048) LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers sometimes fails

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-9048:
--
Description: 
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
  Time elapsed: 41.681 sec  <<< FAILURE!
java.lang.AssertionError: Thread 
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
cluster, but not shut down
  at org.junit.Assert.fail(Assert.java:88)
  at 
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}

  was:
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :

{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
  Time elapsed: 41.681 sec  <<< FAILURE!
java.lang.AssertionError: Thread 
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
cluster, but not shut down
  at org.junit.Assert.fail(Assert.java:88)
  at 
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}


> LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers 
> sometimes fails
> -
>
> Key: FLINK-9048
> URL: https://issues.apache.org/jira/browse/FLINK-9048
> Project: Flink
>  Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
> {code}
> testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
>   Time elapsed: 41.681 sec  <<< FAILURE!
> java.lang.AssertionError: Thread 
> Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
> cluster, but not shut down
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-9135) Remove AggregateReduceFunctionsRule once CALCITE-2216 is fixed

[Shuyi Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shuyi Chen reassigned FLINK-9135:
-

Assignee: Shuyi Chen

> Remove AggregateReduceFunctionsRule once CALCITE-2216 is fixed
> --
>
> Key: FLINK-9135
> URL: https://issues.apache.org/jira/browse/FLINK-9135
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Fabian Hueske
>Assignee: Shuyi Chen
>Priority: Major
>
> We had to copy and slightly modify {{AggregateReduceFunctionsRule}} from 
> Calcite to fix FLINK-8903.
> We proposed the changes to Calcite as CALCITE-2216. Once this issue is fixed 
> and we updated to Calcite dependency to a version that includes the fix, we 
> can remove our custom rule.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Narayanan Arunachalam (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427293#comment-16427293
 ] 

Narayanan Arunachalam commented on FLINK-9138:
--

[~glaksh100] are you considering the PR for 1.4?

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Lakshmi Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427291#comment-16427291
 ] 

Lakshmi Rao commented on FLINK-9138:


Agreed that event time is not applicable here. Yes, that's right, we'll have a 
setter for the the rollIntervalThreshold. I can get a PR started then.

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Narayanan Arunachalam (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427287#comment-16427287
 ] 

Narayanan Arunachalam commented on FLINK-9138:
--

This looks good. I suppose the event time is not applicable here. We will need 
a setter to pass the value for the config 'rollIntervalThreshold'.

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Lakshmi Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427254#comment-16427254
 ] 

Lakshmi Rao commented on FLINK-9138:


[~narayaruna]  [~fhueske] do you have thoughts on how to implement this? I have 
a working version of it, that I've been running on a stream for the last couple 
of days and has been writing to my filesystem of choice every X minutes. The 
basic idea is to track a bucketCreationTime  and add this snippet to the 
[shouldRoll|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L452]
 method:


{code:java}
else {
long createTime = bucketState.bucketCreationTime;
long currentProcessingTime = processingTimeService.getCurrentProcessingTime();
if(currentProcessingTime - createTime > rollIntervalThreshold) {
shouldRoll = true;
LOG.debug(  "BucketingSinkWithSuffix {} starting new bucket because bucket 
is older than rollover interval {} ",
subtaskIndex,
rollIntervalThreshold
);
}
{code}

 I can create a PR out of it, if this makes sense.

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Narayanan Arunachalam (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427227#comment-16427227
 ] 

Narayanan Arunachalam commented on FLINK-9138:
--

Thanks [~glaksh100]

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427224#comment-16427224
 ] 

Fabian Hueske commented on FLINK-9138:
--

Thank you [~glaksh100]!

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Lakshmi Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427220#comment-16427220
 ] 

Lakshmi Rao commented on FLINK-9138:


Hi [~fhueske], I've marked FLINK-9118 as resolved (and duplicated by this 
ticket).

Thanks

Lakshmi

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-9118) Support a time based rollover of part file in Bucketing Sink

[Lakshmi Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lakshmi Rao resolved FLINK-9118.

Resolution: Duplicate

> Support a time based rollover of part file in Bucketing Sink
> 
>
> Key: FLINK-9118
> URL: https://issues.apache.org/jira/browse/FLINK-9118
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Reporter: Lakshmi Rao
>Priority: Major
>
> In the current implementation, the BucketingSink rolls over a part file based 
> on a _batchSize_ 
> ([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]).
>   Can we also support a roll over based on a constant time interval? This is 
> not the same as the _inactiveBucketCheckInterval_ as this bucket is not truly 
> inactive, it's still being written to but just has to be flushed every X 
> minutes, where X is a user-specified time interval. 
> The change would involve tracking a _bucketCreationTime_ in the BucketState 
> (much like the _lastWrittenToTime_) whenever a new part file is opened and 
> would include a condition to check _currentProcessingTime_ - 
> _bucketCreationTime_ > X in the _shouldRoll_ method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427208#comment-16427208
 ] 

Fabian Hueske commented on FLINK-9138:
--

Hi [~narayaruna] and [~glaksh100], can you check if you are requesting the same 
(or sufficiently similar) feature and close one of the JIRAs as duplicate?
Thank you, Fabian

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[Lakshmi Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427186#comment-16427186
 ] 

Lakshmi Rao commented on FLINK-9138:


I had a similar request here: https://issues.apache.org/jira/browse/FLINK-9118

> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9055) WebUI shows job as Running although not enough resources are available

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427104#comment-16427104
 ] 

Fabian Hueske commented on FLINK-9055:
--

TBH, I'm not familiar with the details of the job submission and life cycle 
logic and cannot give valuable feedback on your approach :-(

> WebUI shows job as Running although not enough resources are available
> --
>
> Key: FLINK-9055
> URL: https://issues.apache.org/jira/browse/FLINK-9055
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Webfrontend
>Affects Versions: 1.5.0
> Environment: * FLIP-6 enabled
>  * Local Flink instance with fixed number of TMs
>  * Job parallelism exceeds available slots
>Reporter: Fabian Hueske
>Assignee: Sihua Zhou
>Priority: Major
>
> The WebUI shows a (batch) job as "Running" although not enough resources have 
> been allocated to actually run the job with the requested parallelism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9120) Task Manager Fault Tolerance issue

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-9120.

Resolution: Invalid

> Task Manager Fault Tolerance issue
> --
>
> Key: FLINK-9120
> URL: https://issues.apache.org/jira/browse/FLINK-9120
> Project: Flink
>  Issue Type: Bug
>  Components: Cluster Management, Configuration, Core
>Affects Versions: 1.4.2
>Reporter: dhiraj prajapati
>Priority: Critical
> Attachments: flink-dhiraj.prajapati-client-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-client-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-jobmanager-5-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-jobmanager-5-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-taskmanager-5-ip-10-14-25-116.log, 
> flink-dhiraj.prajapati-taskmanager-5-ip-10-14-25-116.log
>
>
> HI, 
> I have set up a flink 1.4 cluster with 1 job manager and two task managers. 
> The configs taskmanager.numberOfTaskSlots and parallelism.default were set 
> to 2 on each node. I submitted a job to this cluster and it runs fine. To 
> test fault tolerance, I killed one task manager. I was expecting the job to 
> run fine because one of the 2 task managers was still up and running. 
> However, the job failed. Am I missing something? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9120) Task Manager Fault Tolerance issue

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427042#comment-16427042
 ] 

Till Rohrmann commented on FLINK-9120:
--

I think we can close the issue because the behaviour is correct to fail a job 
after all of the restarts have been depleted.

> Task Manager Fault Tolerance issue
> --
>
> Key: FLINK-9120
> URL: https://issues.apache.org/jira/browse/FLINK-9120
> Project: Flink
>  Issue Type: Bug
>  Components: Cluster Management, Configuration, Core
>Affects Versions: 1.4.2
>Reporter: dhiraj prajapati
>Priority: Critical
> Attachments: flink-dhiraj.prajapati-client-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-client-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-jobmanager-5-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-jobmanager-5-ip-10-14-25-115.log, 
> flink-dhiraj.prajapati-taskmanager-5-ip-10-14-25-116.log, 
> flink-dhiraj.prajapati-taskmanager-5-ip-10-14-25-116.log
>
>
> HI, 
> I have set up a flink 1.4 cluster with 1 job manager and two task managers. 
> The configs taskmanager.numberOfTaskSlots and parallelism.default were set 
> to 2 on each node. I submitted a job to this cluster and it runs fine. To 
> test fault tolerance, I killed one task manager. I was expecting the job to 
> run fine because one of the 2 task managers was still up and running. 
> However, the job failed. Am I missing something? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9055) WebUI shows job as Running although not enough resources are available

[Sihua Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427032#comment-16427032
 ] 

Sihua Zhou commented on FLINK-9055:
---

Hi [~fhueske], thanks for your reply, I was thinking that changing the 
lifecycle states maybe the most natural way to solve this problem, but that way 
means we need a really big change for the current code, so as you pointed out 
we should avoid changing the lifecycle states, I agree it. And before 
implementing the code, I'd like to have a bit discussion on this issue, cause I 
don't know what would be the best solution for this issue either... My current 
work-around is when we {{requestJobStatus}}, we won't always just return the 
{{executionGraph.getState()}}, when {{executionGraph.getState() == RUNNING}} we 
need do a double-check to check the {{Executions}} that belong to the 
{{ExecutionGraph}}. What do you think of this work-around?

> WebUI shows job as Running although not enough resources are available
> --
>
> Key: FLINK-9055
> URL: https://issues.apache.org/jira/browse/FLINK-9055
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Webfrontend
>Affects Versions: 1.5.0
> Environment: * FLIP-6 enabled
>  * Local Flink instance with fixed number of TMs
>  * Job parallelism exceeds available slots
>Reporter: Fabian Hueske
>Assignee: Sihua Zhou
>Priority: Major
>
> The WebUI shows a (batch) job as "Running" although not enough resources have 
> been allocated to actually run the job with the requested parallelism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9008) End-to-end test: Quickstarts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427021#comment-16427021
 ] 

ASF GitHub Bot commented on FLINK-9008:
---

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/5823

[FLINK-9008] [e2e] Implements quickstarts end to end test

## What is the purpose of the change
Add an end-to-end test which verifies Flink's quickstarts.

## Brief change log
1. Create a new Flink project using the quickstarts archetype []
2. Add a new Flink dependency to the pom.xml (Flink elasticsearch connector)
3. Add a scripts verify that program


## Verifying this change
Run test_quickstarts.sh verify this change

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes / no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no )
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not documented)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9008

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5823


commit c7b86bb18358715c747e0fa12946b3cfaff86a4d
Author: zhangminglei 
Date:   2018-04-05T14:20:17Z

[FLINK-9008] [e2e] Implements quickstarts end to end test




> End-to-end test: Quickstarts
> 
>
> Key: FLINK-9008
> URL: https://issues.apache.org/jira/browse/FLINK-9008
> Project: Flink
>  Issue Type: Sub-task
>  Components: Quickstarts, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>Priority: Critical
> Fix For: 1.5.0
>
>
> We could add an end-to-end test which verifies Flink's quickstarts. It should 
> do the following:
> # create a new Flink project using the quickstarts archetype 
> # add a new Flink dependency to the {{pom.xml}} (e.g. Flink connector or 
> library) 
> # run {{mvn clean package -Pbuild-jar}}
> # verify that no core dependencies are contained in the jar file
> # Run the program



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427022#comment-16427022
 ] 

ASF GitHub Bot commented on FLINK-8699:
---

Github user sihuazhou commented on the issue:

https://github.com/apache/flink/pull/5705
  
Hi, @StefanRRichter thanks a lot.


> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be 
> changed when writeKVStateMetaData() is invoking ... This can lead to 
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
>   // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
>   stateBackend.kvStateInformation.entrySet()) {
> }
>   //...
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5705: [FLINK-8699][FLINK-8968][state]Fix native resource leak c...

[sihuazhou]

Github user sihuazhou commented on the issue:

https://github.com/apache/flink/pull/5705
  
Hi, @StefanRRichter thanks a lot.


---

[GitHub] flink pull request #5823: [FLINK-9008] [e2e] Implements quickstarts end to e...

[zhangminglei]

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/5823

[FLINK-9008] [e2e] Implements quickstarts end to end test

## What is the purpose of the change
Add an end-to-end test which verifies Flink's quickstarts.

## Brief change log
1. Create a new Flink project using the quickstarts archetype []
2. Add a new Flink dependency to the pom.xml (Flink elasticsearch connector)
3. Add a scripts verify that program


## Verifying this change
Run test_quickstarts.sh verify this change

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes / no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no )
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not documented)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9008

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5823


commit c7b86bb18358715c747e0fa12946b3cfaff86a4d
Author: zhangminglei 
Date:   2018-04-05T14:20:17Z

[FLINK-9008] [e2e] Implements quickstarts end to end test




---

[jira] [Commented] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427012#comment-16427012
 ] 

ASF GitHub Bot commented on FLINK-8699:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/5705
  
Thanks for the fixes. LGTM  Will merge this. Sorry that this took so long.


> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be 
> changed when writeKVStateMetaData() is invoking ... This can lead to 
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
>   // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
>   stateBackend.kvStateInformation.entrySet()) {
> }
>   //...
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5705: [FLINK-8699][FLINK-8968][state]Fix native resource leak c...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/5705
  
Thanks for the fixes. LGTM 👍 Will merge this. Sorry that this took so 
long.


---

[jira] [Commented] (FLINK-9141) Calling getSideOutput() and split() on one DataStream causes NPE

[Julio Biason (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426937#comment-16426937
 ] 

Julio Biason commented on FLINK-9141:
-

Just a note: We got a problem with split/side outputs too, but instead of NPE, 
we got an error message:

 
{code:java}
java.lang.UnsupportedOperationException: Cannot use split/select with side 
outputs.

  
{code}
... and the "map" trick solved this too.

 

> Calling getSideOutput() and split() on one DataStream causes NPE
> 
>
> Key: FLINK-9141
> URL: https://issues.apache.org/jira/browse/FLINK-9141
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Critical
>
> Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
> {{NullPointerException}} to be thrown at runtime.
> As a work-around one can add a no-op map function before the split() call.
> Exception:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Reproducer:
> {code}
> private static final OutputTag tag = new OutputTag("tag") {};
> public static void main(String[] args) throws Exception {
>   StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
>   DataStream dataStream1 = env.fromElements("foo");
>   SingleOutputStreamOperator processedStream = dataStream1
>   .process(new ProcessFunction() {
>   @Override
>   public void processElement(String value, Context ctx, 
> Collector out) {
>   }
>   });
>   processedStream.getSideOutput(tag)
>   .print();
>   processedStream
>   .map(record -> record)
>   .split(Collections::singletonList)
>   .select("bar")
>   .print();
>   env.execute();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9142) Lower the minimum number of buffers for incoming channels to 1

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-9142:
--

 Summary: Lower the minimum number of buffers for incoming channels 
to 1
 Key: FLINK-9142
 URL: https://issues.apache.org/jira/browse/FLINK-9142
 Project: Flink
  Issue Type: Sub-task
  Components: Network
Affects Versions: 1.5.0, 1.6.0
Reporter: Nico Kruber
 Fix For: 1.5.0


Even if we make the floating buffers optional, we still require 
{{taskmanager.network.memory.buffers-per-channel}} number of (exclusive) 
buffers per incoming channel with credit-based flow control while without, the 
minimum was 1 and only the maximum number of buffers was influenced by this 
parameter.

{{taskmanager.network.memory.buffers-per-channel}} is set to {{2}} by default 
with the argumentation that this way we will have one buffer available for 
netty to process while a worker thread is processing/deserializing the other 
buffer. While this seems reasonable, it does increase our minimum requirements. 
Instead, we could probably live with {{1}} exclusive buffer and up to 
{{gate.getNumberOfInputChannels() * (networkBuffersPerChannel - 1) + 
extraNetworkBuffersPerGate}} floating buffers. That way we will have the same 
memory footprint as before with only slightly changed behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9087) Return value of broadcastEvent should be closed in StreamTask#performCheckpoint

[Nico Kruber (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber updated FLINK-9087:
---
Component/s: Network

> Return value of broadcastEvent should be closed in 
> StreamTask#performCheckpoint
> ---
>
> Key: FLINK-9087
> URL: https://issues.apache.org/jira/browse/FLINK-9087
> Project: Flink
>  Issue Type: Bug
>  Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> for (StreamRecordWriter> 
> streamRecordWriter : streamRecordWriters) {
>   try {
> streamRecordWriter.broadcastEvent(message);
> {code}
> The BufferConsumer returned by broadcastEvent() should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9087) Return value of broadcastEvent should be closed in StreamTask#performCheckpoint

[Nico Kruber (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber updated FLINK-9087:
---
Affects Version/s: 1.6.0
   1.5.0

> Return value of broadcastEvent should be closed in 
> StreamTask#performCheckpoint
> ---
>
> Key: FLINK-9087
> URL: https://issues.apache.org/jira/browse/FLINK-9087
> Project: Flink
>  Issue Type: Bug
>  Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> for (StreamRecordWriter> 
> streamRecordWriter : streamRecordWriters) {
>   try {
> streamRecordWriter.broadcastEvent(message);
> {code}
> The BufferConsumer returned by broadcastEvent() should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5672: [FLINK-8872][flip6] fix yarn detached mode command parsin...

[NicoK]

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5672
  
ok, after fixing one typo, this should pass Travis now


---

[jira] [Commented] (FLINK-8872) Yarn detached mode via -yd does not detach

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426911#comment-16426911
 ] 

ASF GitHub Bot commented on FLINK-8872:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5672
  
ok, after fixing one typo, this should pass Travis now


> Yarn detached mode via -yd does not detach
> --
>
> Key: FLINK-8872
> URL: https://issues.apache.org/jira/browse/FLINK-8872
> Project: Flink
>  Issue Type: Bug
>  Components: Client, YARN
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Running yarn per-job cluster in detached mode currently does not work and 
> waits for the job to finish.
> Example:
> {code}
> ./bin/flink run -m yarn-cluster -yn 10 -yjm 768 -ytm 3072 -ys 2 -yd -p 20 -c 
> org.apache.flink.streaming.examples.wordcount.WordCount 
> ./examples/streaming/WordCount.jar --input
> {code}
> Output in case of an infinite program would then end with something like this:
> {code}
> 2018-03-05 13:41:23,311 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Waiting for 
> the cluster to be allocated
> 2018-03-05 13:41:23,313 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Deploying 
> cluster, current state ACCEPTED
> 2018-03-05 13:41:28,342 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - YARN 
> application has been deployed successfully.
> 2018-03-05 13:41:28,343 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - The Flink 
> YARN client has been started in detached mode. In order to stop Flink on 
> YARN, use the following command or a YARN web interface to stop it:
> yarn application -kill application_1519984124671_0006
> Please also note that the temporary files of the YARN session in the home 
> directoy will not be removed.
> Starting execution of program
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9087) Return value of broadcastEvent should be closed in StreamTask#performCheckpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426882#comment-16426882
 ] 

ASF GitHub Bot commented on FLINK-9087:
---

Github user NicoK commented on a diff in the pull request:

https://github.com/apache/flink/pull/5802#discussion_r179451610
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
 ---
@@ -164,7 +164,7 @@ public BufferConsumer broadcastEvent(AbstractEvent 
event) throws IOException {
if (flushAlways) {
flushAll();
}
-   return eventBufferConsumer;
--- End diff --

You don't need to close the `eventBufferConsumer` since the 
try-with-resources should already do that. Returning the closed value, however, 
is kind of strange and only needed in one test. Let's make the method return 
`void` as you suggested and adapt the test.


> Return value of broadcastEvent should be closed in 
> StreamTask#performCheckpoint
> ---
>
> Key: FLINK-9087
> URL: https://issues.apache.org/jira/browse/FLINK-9087
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> for (StreamRecordWriter> 
> streamRecordWriter : streamRecordWriters) {
>   try {
> streamRecordWriter.broadcastEvent(message);
> {code}
> The BufferConsumer returned by broadcastEvent() should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9087) Return value of broadcastEvent should be closed in StreamTask#performCheckpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426883#comment-16426883
 ] 

ASF GitHub Bot commented on FLINK-9087:
---

Github user NicoK commented on a diff in the pull request:

https://github.com/apache/flink/pull/5802#discussion_r179453256
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
 ---
@@ -309,8 +309,6 @@ public void testBroadcastEventBufferReferenceCounting() 
throws Exception {
for (int i = 0; i < queues.length; i++) {
assertTrue(parseBuffer(queues[i].remove(), 
i).isEvent());
}
-
-   assertTrue(bufferConsumer.isRecycled());
}
--- End diff --

yes, this should not be dropped since this line is the actual test - how 
about this instead:
```
// Verify added to all queues
assertEquals(1, queues[0].size());
assertEquals(1, queues[1].size());

// get references to buffer consumers (copies from the original 
event buffer consumer)
BufferConsumer bufferConsumer1 = queues[0].getFirst();
BufferConsumer bufferConsumer2 = queues[1].getFirst();

// process all collected events (recycles the buffer)
assertTrue(parseBuffer(queues[0].remove(), 0).isEvent());
assertTrue(parseBuffer(queues[1].remove(), 1).isEvent());

assertTrue(bufferConsumer1.isRecycled());
assertTrue(bufferConsumer2.isRecycled());
```


> Return value of broadcastEvent should be closed in 
> StreamTask#performCheckpoint
> ---
>
> Key: FLINK-9087
> URL: https://issues.apache.org/jira/browse/FLINK-9087
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> for (StreamRecordWriter> 
> streamRecordWriter : streamRecordWriters) {
>   try {
> streamRecordWriter.broadcastEvent(message);
> {code}
> The BufferConsumer returned by broadcastEvent() should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

[NicoK]

Github user NicoK commented on a diff in the pull request:

https://github.com/apache/flink/pull/5802#discussion_r179453256
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
 ---
@@ -309,8 +309,6 @@ public void testBroadcastEventBufferReferenceCounting() 
throws Exception {
for (int i = 0; i < queues.length; i++) {
assertTrue(parseBuffer(queues[i].remove(), 
i).isEvent());
}
-
-   assertTrue(bufferConsumer.isRecycled());
}
--- End diff --

yes, this should not be dropped since this line is the actual test - how 
about this instead:
```
// Verify added to all queues
assertEquals(1, queues[0].size());
assertEquals(1, queues[1].size());

// get references to buffer consumers (copies from the original 
event buffer consumer)
BufferConsumer bufferConsumer1 = queues[0].getFirst();
BufferConsumer bufferConsumer2 = queues[1].getFirst();

// process all collected events (recycles the buffer)
assertTrue(parseBuffer(queues[0].remove(), 0).isEvent());
assertTrue(parseBuffer(queues[1].remove(), 1).isEvent());

assertTrue(bufferConsumer1.isRecycled());
assertTrue(bufferConsumer2.isRecycled());
```


---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

[NicoK]

Github user NicoK commented on a diff in the pull request:

https://github.com/apache/flink/pull/5802#discussion_r179451610
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
 ---
@@ -164,7 +164,7 @@ public BufferConsumer broadcastEvent(AbstractEvent 
event) throws IOException {
if (flushAlways) {
flushAll();
}
-   return eventBufferConsumer;
--- End diff --

You don't need to close the `eventBufferConsumer` since the 
try-with-resources should already do that. Returning the closed value, however, 
is kind of strange and only needed in one test. Let's make the method return 
`void` as you suggested and adapt the test.


---

[jira] [Commented] (FLINK-8468) Make the connector to take advantage of AMQP features (routing key, exchange and message properties)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426834#comment-16426834
 ] 

ASF GitHub Bot commented on FLINK-8468:
---

Github user pduveau commented on the issue:

https://github.com/apache/flink/pull/5410
  
Hi, Is there anyone to check this PR ? We have interest in getting an 
official version including this. 


> Make the connector to take advantage of AMQP features (routing key, exchange 
> and message properties)
> 
>
> Key: FLINK-8468
> URL: https://issues.apache.org/jira/browse/FLINK-8468
> Project: Flink
>  Issue Type: Improvement
>  Components: RabbitMQ Connector
>Affects Versions: 1.4.0
>Reporter: Ph.Duveau
>Priority: Major
>
> Make the connector to take advantage of AMQP features by adding a constructor 
> and an interface to implement



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5410: [FLINK-8468] [RabbitMQ Connector] Take advantage of AMQP ...

[pduveau]

Github user pduveau commented on the issue:

https://github.com/apache/flink/pull/5410
  
Hi, Is there anyone to check this PR ? We have interest in getting an 
official version including this. 


---

[jira] [Commented] (FLINK-8556) Add proxy feature to Kinesis Connector to acces its endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426830#comment-16426830
 ] 

ASF GitHub Bot commented on FLINK-8556:
---

Github user pduveau commented on the issue:

https://github.com/apache/flink/pull/5411
  
Hi @tzulitai, Were you able to get time to check this PR ? We have interest 
in an official version including this.


> Add proxy feature to Kinesis Connector to acces its endpoint
> 
>
> Key: FLINK-8556
> URL: https://issues.apache.org/jira/browse/FLINK-8556
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.4.0
>Reporter: Ph.Duveau
>Priority: Major
>  Labels: features
>
> The connector can not be configured to use a proxy to access Kinesis 
> endpoint. This feature is required on EC2 instances which can access internet 
> only through a proxy. VPC Kinesis endpoints are currently available in few 
> AWS' regions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5411: [FLINK-8556] [Kinesis Connector] Add proxy feature to the...

[pduveau]

Github user pduveau commented on the issue:

https://github.com/apache/flink/pull/5411
  
Hi @tzulitai, Were you able to get time to check this PR ? We have interest 
in an official version including this.


---

[jira] [Assigned] (FLINK-8742) Move ConfigDocsGenerator annotation to flink-annotations

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-8742:
---

Assignee: Chesnay Schepler  (was: Tuo Wang)

> Move ConfigDocsGenerator annotation to flink-annotations
> 
>
> Key: FLINK-8742
> URL: https://issues.apache.org/jira/browse/FLINK-8742
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
>  Labels: easy-fix, starter
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} added annotations to modify the behavior of the 
> generator output. ({{ConfigGroup}} and {{ConfigGroups}}).
> Currently these annotations are part of {{flink-core}} and residein the same 
> package as the various `ConfigOption` classes.
> I propose moving them to flink-annotations for consolidation. We may also 
> place them in a new module in {{flink-docs}}, but given that we already have 
> a module exclusively for annotations this seems overkill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5822: [FLINK-8743][docs] Allow overriding documented def...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/5822

 [FLINK-8743][docs] Allow overriding documented default 

Based on #5821.

## What is the purpose of the change

This PR adds an annotation to override the documented default.

## Verifying this change

This change added tests and can be verified as follows:

* `ConfigOptionsDocGeneratorTest#testOverrideDefault`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 8743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5822.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5822


commit 4c6527430ca5d3ac9f9c1a18cdcc403b3f47f93e
Author: zentol 
Date:   2018-04-05T10:17:27Z

[FLINK-8742][docs] Move docs generator annotations to flink-annotations

commit 10064482cc0d0e2dbddfc4d3988c4a17f78700fa
Author: zentol 
Date:   2018-04-05T10:42:42Z

[FLINK-8743][docs] Allow overriding documented default




---

[jira] [Commented] (FLINK-8743) Add annotation to override documented default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426744#comment-16426744
 ] 

ASF GitHub Bot commented on FLINK-8743:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/5822

 [FLINK-8743][docs] Allow overriding documented default 

Based on #5821.

## What is the purpose of the change

This PR adds an annotation to override the documented default.

## Verifying this change

This change added tests and can be verified as follows:

* `ConfigOptionsDocGeneratorTest#testOverrideDefault`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 8743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5822.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5822


commit 4c6527430ca5d3ac9f9c1a18cdcc403b3f47f93e
Author: zentol 
Date:   2018-04-05T10:17:27Z

[FLINK-8742][docs] Move docs generator annotations to flink-annotations

commit 10064482cc0d0e2dbddfc4d3988c4a17f78700fa
Author: zentol 
Date:   2018-04-05T10:42:42Z

[FLINK-8743][docs] Allow overriding documented default




> Add annotation to override documented default
> -
>
> Key: FLINK-8743
> URL: https://issues.apache.org/jira/browse/FLINK-8743
> Project: Flink
>  Issue Type: New Feature
>  Components: Configuration, Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
>
> The default value for some {{ConfigOptions}} is difficult to document as it 
> isn't static. This mostly affects options that use 
> {{System.getProperty("java.io.tmpdir")}}, as for example 
> {{CoreOptions#TMP_DIRS}}.
> To deal with this the generator has a special branch for selected options 
> that overrides this default: 
> {code}
> if (option == WebOptions.TMP_DIR || option.key().equals("python.dc.tmp.dir") 
> || option == CoreOptions.TMP_DIRS) {
>   defaultValue = null;
> }
> {code}
> (let's ignore for now that it just wipes the default and isn't setting it to 
> "System.getProperty("java.io.tmpdir")")
> This is pretty much hidden deep in the implementation of the generator. It 
> would be better if we had a dedicated annotation 
> {{@OverrideDocumentedDefault(String override)}} that options could be 
> annotated with.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8742) Move ConfigDocsGenerator annotation to flink-annotations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426739#comment-16426739
 ] 

ASF GitHub Bot commented on FLINK-8742:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/5821

[FLINK-8742][docs] Move docs generator annotations to flink-annotations

## What is the purpose of the change

This PR moves the docs generator annotations to flink-docs, for 
consolidation purposes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 8742

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5821.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5821


commit 4c6527430ca5d3ac9f9c1a18cdcc403b3f47f93e
Author: zentol 
Date:   2018-04-05T10:17:27Z

[FLINK-8742][docs] Move docs generator annotations to flink-annotations




> Move ConfigDocsGenerator annotation to flink-annotations
> 
>
> Key: FLINK-8742
> URL: https://issues.apache.org/jira/browse/FLINK-8742
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Tuo Wang
>Priority: Trivial
>  Labels: easy-fix, starter
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} added annotations to modify the behavior of the 
> generator output. ({{ConfigGroup}} and {{ConfigGroups}}).
> Currently these annotations are part of {{flink-core}} and residein the same 
> package as the various `ConfigOption` classes.
> I propose moving them to flink-annotations for consolidation. We may also 
> place them in a new module in {{flink-docs}}, but given that we already have 
> a module exclusively for annotations this seems overkill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5821: [FLINK-8742][docs] Move docs generator annotations...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/5821

[FLINK-8742][docs] Move docs generator annotations to flink-annotations

## What is the purpose of the change

This PR moves the docs generator annotations to flink-docs, for 
consolidation purposes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 8742

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5821.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5821


commit 4c6527430ca5d3ac9f9c1a18cdcc403b3f47f93e
Author: zentol 
Date:   2018-04-05T10:17:27Z

[FLINK-8742][docs] Move docs generator annotations to flink-annotations




---

[GitHub] flink pull request #5818: [hotfix][table][docs]sql() is deprecated.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5818


---

[jira] [Commented] (FLINK-8835) Fix TaskManager config keys

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426664#comment-16426664
 ] 

ASF GitHub Bot commented on FLINK-8835:
---

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5808
  
Thanks @zentol for review.  


> Fix TaskManager config keys
> ---
>
> Key: FLINK-8835
> URL: https://issues.apache.org/jira/browse/FLINK-8835
> Project: Flink
>  Issue Type: Bug
>  Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>Priority: Blocker
>  Labels: easy-fix
> Fix For: 1.5.0
>
>
> Many new config keys in the TaskManager don't follow the proper naming 
> scheme. We need to clear those up before the release. I would also suggest to 
> keep the key names short, because that makes it easier for users.
> When doing this cleanup pass over the config keys, I would suggest to also 
> make some of the existing keys more hierarchical harmonize them with the 
> common scheme in Flink.
> h1. New Keys
> * {{taskmanager.network.credit-based-flow-control.enabled}} to 
> {{taskmanager.network.credit-model}}.
> h1. Existing Keys
> * {{taskmanager.debug.memory.startLogThread}} => 
> {{taskmanager.debug.memory.log}}
> * {{taskmanager.debug.memory.logIntervalMs}} => 
> {{taskmanager.debug.memory.log-interval}}
> * {{taskmanager.initial-registration-pause}} => 
> {{taskmanager.registration.initial-backoff}}
> * {{taskmanager.max-registration-pause}} => 
> {{taskmanager.registration.max-backoff}}
> * {{taskmanager.refused-registration-pause}} 
> {{taskmanager.registration.refused-backoff}}
> * {{taskmanager.maxRegistrationDuration}} ==> * 
> {{taskmanager.registration.timeout}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5808: [FLINK-8835] [taskmanager] Fix TaskManager config keys

[zhangminglei]

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5808
  
Thanks @zentol for review. 👍 


---

[jira] [Commented] (FLINK-8961) Port JobRetrievalITCase to flip6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426651#comment-16426651
 ] 

ASF GitHub Bot commented on FLINK-8961:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5730
  
@tillrohrmann I've rebased and update the PR.

I've added a `getClientConfiguration()` to the `MiniClusterResource`, and 
converted the MiniCluster used in the test to the `MiniClusterResource`.

I have not removed the locks though. There is indeed no difference 
_currently_ between fetching the result for running and finished jobs, but we 
also want to make sure it stays this way in the future.


> Port JobRetrievalITCase to flip6
> 
>
> Key: FLINK-8961
> URL: https://issues.apache.org/jira/browse/FLINK-8961
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5730: [FLINK-8961][tests] Port JobRetrievalITCase to flip6

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5730
  
@tillrohrmann I've rebased and update the PR.

I've added a `getClientConfiguration()` to the `MiniClusterResource`, and 
converted the MiniCluster used in the test to the `MiniClusterResource`.

I have not removed the locks though. There is indeed no difference 
_currently_ between fetching the result for running and finished jobs, but we 
also want to make sure it stays this way in the future.


---

[GitHub] flink issue #5818: [hotfix][table][docs]sql() is deprecated.

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5818
  
Thanks for the fix @mayyamus! 
Merging


---

[jira] [Updated] (FLINK-9141) Calling getSideOutput() and split() on one DataStream causes NPE

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-9141:

Description: 
Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
{{NullPointerException}} to be thrown at runtime.

As a work-around one can add a no-op map function before the split() call.

Exception:
{code}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproducer:
{code}
private static final OutputTag tag = new OutputTag("tag") {};

public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();

DataStream dataStream1 = env.fromElements("foo");

SingleOutputStreamOperator processedStream = dataStream1
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx, 
Collector out) {
}
});

processedStream.getSideOutput(tag)
.print();

processedStream
.map(record -> record)
.split(Collections::singletonList)
.select("bar")
.print();

env.execute();
}
{code}

  was:
Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
{{NullPointerException}} to be thrown.

As a work-around one can add a no-op map function before the split() call.

Exception:
{code}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproducer:
{code}
private static final OutputTag tag = new OutputTag("tag") {};

public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();

DataStream dataStream1 = env.fromElements("foo");

SingleOutputStreamOperator processedStream = dataStream1
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx, 
Collector out) {
}
});

processedStream.getSideOutput(tag)
.print();

processedStream
.map(record -> record)
.split(Collections::singletonList)
.select("bar")
.print();

env.execute();
}
{code}


> Calling getSideOutput() and split() on one DataStream causes NPE
> 
>
> Key: FLINK-9141
> URL: https://issues.apache.org/jira/browse/FLINK-9141
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Critical
>
> Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
> {{NullPointerException}} to be thrown at runtime.
> As a work-around one can add a no-op map function before the split() call.
> Exception:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Reproducer:
> {code}
> private static final OutputTag tag = new OutputTag("tag") {};
> public static void main(String[] args) throws Exception {
>   StreamExecutionEnvironment env = 
> 

[jira] [Updated] (FLINK-9141) Calling getSideOutput() and split() on one DataStream causes NPE

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-9141:

Description: 
Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
{{NullPointerException}} to be thrown.

As a work-around one can add a no-op map function before the split() call.

Exception:
{code}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproducer:
{code}
private static final OutputTag tag = new OutputTag("tag") {};

public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();

DataStream dataStream1 = env.fromElements("foo");

SingleOutputStreamOperator processedStream = dataStream1
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx, 
Collector out) {
}
});

processedStream.getSideOutput(tag)
.print();

processedStream
.map(record -> record)
.split(Collections::singletonList)
.select("bar")
.print();

env.execute();
}
{code}

  was:
Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
{{NullPointerException}} to be thrown.

As a temporary work-around one can add a no-op map function before the split() 
call.

Exception:
{code}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproducer:
{code}
private static final OutputTag tag = new OutputTag("tag") {};

public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();

DataStream dataStream1 = env.fromElements("foo");

SingleOutputStreamOperator processedStream = dataStream1
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx, 
Collector out) {
}
});

processedStream.getSideOutput(tag)
.print();

processedStream
.map(record -> record)
.split(Collections::singletonList)
.select("bar")
.print();

env.execute();
}
{code}


> Calling getSideOutput() and split() on one DataStream causes NPE
> 
>
> Key: FLINK-9141
> URL: https://issues.apache.org/jira/browse/FLINK-9141
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Critical
>
> Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
> {{NullPointerException}} to be thrown.
> As a work-around one can add a no-op map function before the split() call.
> Exception:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Reproducer:
> {code}
> private static final OutputTag tag = new OutputTag("tag") {};
> public static void main(String[] args) throws Exception {
>   StreamExecutionEnvironment env = 
> 

[jira] [Created] (FLINK-9141) Calling getSideOutput() and split() on one DataStream causes NPE

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-9141:
---

 Summary: Calling getSideOutput() and split() on one DataStream 
causes NPE
 Key: FLINK-9141
 URL: https://issues.apache.org/jira/browse/FLINK-9141
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Affects Versions: 1.4.0
Reporter: Chesnay Schepler


Calling both {{getSideOutput()}} and {{split()}} on one DataStream causes a 
{{NullPointerException}} to be thrown.

As a temporary work-around one can add a no-op map function before the split() 
call.

Exception:
{code}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.streaming.api.collector.selector.DirectedOutput.(DirectedOutput.java:79)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:326)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:128)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:274)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproducer:
{code}
private static final OutputTag tag = new OutputTag("tag") {};

public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();

DataStream dataStream1 = env.fromElements("foo");

SingleOutputStreamOperator processedStream = dataStream1
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx, 
Collector out) {
}
});

processedStream.getSideOutput(tag)
.print();

processedStream
.map(record -> record)
.split(Collections::singletonList)
.select("bar")
.print();

env.execute();
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5820: [hotfix] removed unused scala imports

[bowenli86]

GitHub user bowenli86 opened a pull request:

https://github.com/apache/flink/pull/5820

[hotfix] removed unused scala imports

## What is the purpose of the change

removed unused scala imports

## Brief change log

removed unused scala imports

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:

none

## Documentation

none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bowenli86/flink hotfix_2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5820.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5820


commit f6940a0bf0fd5ff487a63ffad29e4dc9cd7a970c
Author: Bowen Li 
Date:   2018-04-05T07:52:37Z

[hotfix] removed unused scala imports




---

[jira] [Commented] (FLINK-9056) Job submission fails with AskTimeoutException if not enough slots are available

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426596#comment-16426596
 ] 

Fabian Hueske commented on FLINK-9056:
--

I assume that the following happens: 

The client job submission call blocks while the JobManager tries to start the 
job. However, the job will never start because there are not enough slots to 
achieve the requested parallelism. At some point the Akka request times out and 
throws the reported exception.

I think it is OK to throw an exception, however it would be better if the 
exception would indicate why the job could not be started. If the situation is 
as I assumed, this won't be trivial, because right now the client times out and 
for a proper error message, the JM would need to propagate an exception.

> Job submission fails with AskTimeoutException if not enough slots are 
> available
> ---
>
> Key: FLINK-9056
> URL: https://issues.apache.org/jira/browse/FLINK-9056
> Project: Flink
>  Issue Type: Improvement
>  Components: Job-Submission
>Affects Versions: 1.5.0
> Environment: * FLIP-6 enabled
>  * Local Flink instance with fixed number of TMs
>  * Job parallelism exceeds available slots
>Reporter: Fabian Hueske
>Assignee: yuqi
>Priority: Major
>
> The error message if a job submission fails due to lack of available slots is 
> not helpful:
> {code:java}
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/8f0fabba-4021-45b6-a1f7-b8afd6627640#-574617182|#-574617182]]
>  after [30 ms]. Sender[null] sent message of type 
> "org.apache.flink.runtime.rpc.messages.LocalRpcInvocation".
>      at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>      at 
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>      at 
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>      at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>      at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>      at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9140) simplify scalastyle configurations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426590#comment-16426590
 ] 

ASF GitHub Bot commented on FLINK-9140:
---

GitHub user bowenli86 opened a pull request:

https://github.com/apache/flink/pull/5819

[FLINK-9140] [Build System] [scalastyle] simplify scalastyle configurations

## What is the purpose of the change

Simplifying `` to ``

## Brief change log

Simplifying `` to ``

## Verifying this change

This change is already covered by existing tests

## Does this pull request potentially affect one of the following parts:

none

## Documentation

none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bowenli86/flink FLINK-9140

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5819.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5819


commit 8bcddf43805190665237325b1f2efd17a5b9f47f
Author: Bowen Li 
Date:   2018-04-05T07:42:50Z

[FLINK-9140] simplify scalastyle configurations




> simplify scalastyle configurations
> --
>
> Key: FLINK-9140
> URL: https://issues.apache.org/jira/browse/FLINK-9140
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.5.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Minor
> Fix For: 1.5.0, 1.6.0
>
>
> Simplifying {{}} to {{}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5819: [FLINK-9140] [Build System] [scalastyle] simplify ...

[bowenli86]

GitHub user bowenli86 opened a pull request:

https://github.com/apache/flink/pull/5819

[FLINK-9140] [Build System] [scalastyle] simplify scalastyle configurations

## What is the purpose of the change

Simplifying `` to ``

## Brief change log

Simplifying `` to ``

## Verifying this change

This change is already covered by existing tests

## Does this pull request potentially affect one of the following parts:

none

## Documentation

none

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bowenli86/flink FLINK-9140

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5819.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5819


commit 8bcddf43805190665237325b1f2efd17a5b9f47f
Author: Bowen Li 
Date:   2018-04-05T07:42:50Z

[FLINK-9140] simplify scalastyle configurations




---

[jira] [Created] (FLINK-9140) simplify scalastyle configurations

[Bowen Li (JIRA)]

Bowen Li created FLINK-9140:
---

 Summary: simplify scalastyle configurations
 Key: FLINK-9140
 URL: https://issues.apache.org/jira/browse/FLINK-9140
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Affects Versions: 1.5.0
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.5.0, 1.6.0


Simplifying {{}} to {{}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9056) Job submission fails with AskTimeoutException if not enough slots are available

[yuqi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426548#comment-16426548
 ] 

yuqi commented on FLINK-9056:
-

[~fhueske] Thank you for your reply, I will first reproduce it and try to 
figure out the cause.

> Job submission fails with AskTimeoutException if not enough slots are 
> available
> ---
>
> Key: FLINK-9056
> URL: https://issues.apache.org/jira/browse/FLINK-9056
> Project: Flink
>  Issue Type: Improvement
>  Components: Job-Submission
>Affects Versions: 1.5.0
> Environment: * FLIP-6 enabled
>  * Local Flink instance with fixed number of TMs
>  * Job parallelism exceeds available slots
>Reporter: Fabian Hueske
>Assignee: yuqi
>Priority: Major
>
> The error message if a job submission fails due to lack of available slots is 
> not helpful:
> {code:java}
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/8f0fabba-4021-45b6-a1f7-b8afd6627640#-574617182|#-574617182]]
>  after [30 ms]. Sender[null] sent message of type 
> "org.apache.flink.runtime.rpc.messages.LocalRpcInvocation".
>      at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>      at 
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>      at 
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>      at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>      at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>      at 
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>      at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8514) move flink-connector-wikiedits to Apache Bahir

[Bowen Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li closed FLINK-8514.
---
Resolution: Won't Fix

> move flink-connector-wikiedits to Apache Bahir
> --
>
> Key: FLINK-8514
> URL: https://issues.apache.org/jira/browse/FLINK-8514
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>
> I propose moving flink-connector-wikiedits to Apache Bahir given its low 
> popularity



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8188) Clean up flink-contrib

[Bowen Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li resolved FLINK-8188.
-
Resolution: Fixed

> Clean up flink-contrib
> --
>
> Key: FLINK-8188
> URL: https://issues.apache.org/jira/browse/FLINK-8188
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>
> This is the umbrella ticket for cleaning up flink-contrib. 
> We argue that flink-contrib should be removed and all its submodules should 
> be migrated to other top-level modules for the following reasons: 
> 1) Apache Flink the whole project itself is a result of contributions from 
> many developers, there's no reason to highlight some contributions in a 
> dedicated module named 'contrib'
> 2) flink-contrib is already too crowded and noisy. It contains lots of sub 
> modules with different purposes which confuse developers and users, and they 
> lack a proper project hierarchy
> 3) This will save us quite some build time
> More details in discussions at FLINK-8175 and FLINK-8167



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-7566) if there's only one checkpointing metadata file in , `flink run -s ` should successfully resume from that metadata file

[Bowen Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li closed FLINK-7566.
---
Resolution: Won't Fix

> if there's only one checkpointing metadata file in , `flink run -s 
> ` should successfully resume from that metadata file 
> --
>
> Key: FLINK-7566
> URL: https://issues.apache.org/jira/browse/FLINK-7566
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.2
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>
> Currently, if we want to start a Flink job from a checkpointing file, we have 
> to run `flink run -s /checkpoint_metadata-x` by explicitly 
> specifying the checkpoint metadata file name 'checkpoint_metadata-x'. 
> Since metadata file name always changes, it's not easy to programmatically 
> restart a failed Flink job. The error from jobmanager.log looks like:
> {code:java}
> 2017-08-30 07:25:04,907 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job  
> (22defcf962ff2ac2e7fe99354f5ab168) switched from state FAILING to FAILED.
> org.apache.flink.runtime.execution.SuppressRestartsException: Unrecoverable 
> failure. This suppresses job restarts. Please check the stack trace for the 
> root cause.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1396)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1372)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.io.IOException: Cannot find meta data file in directory 
> s3:///checkpoints. Please try to load the savepoint directly from the 
> meta data file instead of the directory.
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepointWithHandle(SavepointStore.java:262)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:69)
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1140)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1386)
>   ... 10 more
> {code}
> What I want is like this: users should be able to start a Flink job by 
> running `flink run -s ` if there's only one checkpointing metadata file 
> in . If there's none or more than 1 metadata file, the command can fail 
> like it is right now. This way, we can programmatically restart a failed 
> Flink job by hardcoding .
> To achieve that, I think there're two appraches we can do:
> 1) modify {{CheckpointCoordinator.restoreSavepoint}} to check how many 
> metadata files are in 
> 2) add another commandline option like '-sd' / '--savepointdirectory' to 
> explicitly load a dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8837) add @Experimental annotation and properly annotate some classes

[Bowen Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li updated FLINK-8837:

Affects Version/s: 1.5.0

>  add @Experimental annotation and properly annotate some classes
> 
>
> Key: FLINK-8837
> URL: https://issues.apache.org/jira/browse/FLINK-8837
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.5.0
>Reporter: Stephan Ewen
>Assignee: Bowen Li
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The class {{DataStreamUtils}} came from 'flink-contrib' and now accidentally 
> moved to the fully supported API packages. It should be in package 
> 'experimental' to properly communicate that it is not guaranteed to be API 
> stable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9127) Filesystem State Backend logged incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426530#comment-16426530
 ] 

ASF GitHub Bot commented on FLINK-9127:
---

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/5810
  
The original logging is correct - filesystem state backend is actually 
memory state backend + filesystem checkpointing. No need to change the logging. 


> Filesystem State Backend logged incorrectly
> ---
>
> Key: FLINK-9127
> URL: https://issues.apache.org/jira/browse/FLINK-9127
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.3.2, 1.4.2
>Reporter: Scott Kidder
>Priority: Trivial
>
> When using a filesystem backend, the 
> '[StateBackendLoader|https://github.com/apache/flink/blob/1f9c2d9740ffea2b59b8f5f3da287a0dc890ddbf/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L123]'
>  class produces a log message stating: "State backend is set to heap memory". 
> Example:
> {{2018-04-04 00:45:49,591 INFO  
> org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend 
> is set to heap memory (checkpoints to filesystem 
> "hdfs://hdfs:8020/flink/checkpoints")}}
> It looks like this resulted from some copy-pasta of the previous 
> case-statement that matches on the memory backend. This bug is also present 
> in earlier releases (1.3.2, 1.4.0) of Flink in the 'AbstractStateBackend' 
> class.
> This log statement should be corrected to indicate that a filesystem backend 
> is in use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5810: [FLINK-9127] [Core] Filesystem State Backend logged incor...

[bowenli86]

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/5810
  
The original logging is correct - filesystem state backend is actually 
memory state backend + filesystem checkpointing. No need to change the logging. 


---