[GitHub] flink pull request: [FLINK-3025] [kafka consumer] Bump transitive ...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/1365

[FLINK-3025] [kafka consumer] Bump transitive ZkClient dependency to 0.7 
for bugfixes

Gives us a fix against deadlocks in the ZkClient which have caused the 
Kafka Consumer to freeze.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink zk_version_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1365.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1365


commit 97715cd9e6ad88a1b7d1178f13e1e12fd0544996
Author: Stephan Ewen 
Date:   2015-11-17T11:03:09Z

[FLINK-3025] [kafka consumer] Bump transitive ZkClient dependency to 0.7 
for bugfixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Bumped the docker container to the latest vers...

[thedrow]

GitHub user thedrow opened a pull request:

https://github.com/apache/flink/pull/1366

Bumped the docker container to the latest version



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thedrow/flink patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1366.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1366


commit af16ac80eeff168abf11e8ea351ceef94eeea34f
Author: Omer Katz 
Date:   2015-11-17T12:35:42Z

Bumped the docker container to the latest version.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3026) Publish the flink docker container to the docker registry

[Omer Katz (JIRA)]

Omer Katz created FLINK-3026:


 Summary: Publish the flink docker container to the docker registry
 Key: FLINK-3026
 URL: https://issues.apache.org/jira/browse/FLINK-3026
 Project: Flink
  Issue Type: Task
Reporter: Omer Katz


There's a dockerfile that can be used to build a docker container already in 
the repository. It'd be awesome to just be able to pull it instead of building 
it ourselves.
The dockerfile can be found at 
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
It also doesn't point to the latest version of Flink which I fixed in 
https://github.com/apache/flink/pull/1366



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2508) Confusing sharing of StreamExecutionEnvironment

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2508.
-
   Resolution: Fixed
 Assignee: (was: Márton Balassi)
Fix Version/s: (was: 1.0.0)
   0.10.0

Was fixed as part of the streaming API rework prior to 0.10

> Confusing sharing of StreamExecutionEnvironment
> ---
>
> Key: FLINK-2508
> URL: https://issues.apache.org/jira/browse/FLINK-2508
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
> Fix For: 0.10.0
>
>
> In the {{StreamExecutionEnvironment}}, the environment is once created and 
> then shared with a static variable to all successive calls to 
> {{getExecutionEnvironment()}}. But it can be overridden by calls to 
> {{createLocalEnvironment()}} and {{createRemoteEnvironment()}}.
> This seems a bit un-intuitive, and probably creates confusion when 
> dispatching multiple streaming jobs from within the same JVM.
> Why is it even necessary to cache the "current" execution environment?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2508) Confusing sharing of StreamExecutionEnvironment

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2508.
---

> Confusing sharing of StreamExecutionEnvironment
> ---
>
> Key: FLINK-2508
> URL: https://issues.apache.org/jira/browse/FLINK-2508
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
> Fix For: 0.10.0
>
>
> In the {{StreamExecutionEnvironment}}, the environment is once created and 
> then shared with a static variable to all successive calls to 
> {{getExecutionEnvironment()}}. But it can be overridden by calls to 
> {{createLocalEnvironment()}} and {{createRemoteEnvironment()}}.
> This seems a bit un-intuitive, and probably creates confusion when 
> dispatching multiple streaming jobs from within the same JVM.
> Why is it even necessary to cache the "current" execution environment?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3024) TimestampExtractor Does not Work When returning Long.MIN_VALUE

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-3024.
---
   Resolution: Fixed
Fix Version/s: 0.10.1
   1.0.0

Fixed in 
https://github.com/apache/flink/commit/bdb306a1152aa9938bde2cc536db540eb2ca40d0

> TimestampExtractor Does not Work When returning Long.MIN_VALUE
> --
>
> Key: FLINK-3024
> URL: https://issues.apache.org/jira/browse/FLINK-3024
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0, 0.10.1
>
>
> The problem is that {{ExtractTimestampsOperator}} wrongly updates the 
> {{currentWatermark}} in the {{trigger}} method. This will interfere with 
> values returned from {{extractWatermark}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2942) Dangling operators in web UI's program visualization (non-deterministic)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008645#comment-15008645
 ] 

ASF GitHub Bot commented on FLINK-2942:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1346#issuecomment-157373231
  
Will merge this for 1.0 and 0.10.0 ...


> Dangling operators in web UI's program visualization (non-deterministic)
> 
>
> Key: FLINK-2942
> URL: https://issues.apache.org/jira/browse/FLINK-2942
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0
> Environment: OSX, Firefox and Chrome
>Reporter: Fabian Hueske
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2015-10-29 at 17.11.19.png, Screen Shot 
> 2015-10-29 at 20.51.46.png, Screen Shot 2015-10-29 at 20.52.13.png, Screen 
> Shot 2015-11-09 at 14.48.03.png
>
>
> When visualizing a program with three {{MapPartition}} operators that branch 
> off from an {{OuterJoin}} operator, two of the three {{MapPartition}} 
> operators are not connected to the {{OuterJoin}} operator and appear to have 
> no input.
> The problem is present in FireFox as well as in Chrome. I'll attach a 
> screenshot.
> The problem and be reproduced by executing the "Cascading for the impatient" 
> [TFIDF example 
> program|https://github.com/Cascading/Impatient/tree/master/part5] using the 
> [Cascading Flink Connector|https://github.com/dataArtisans/cascading-flink].
> Update: It appears that the problem is non-deterministic. I ran the same job 
> again (same setup) and the previously missing connections were visualized. 
> However, the UI showed only one input for a binary operator (OuterJoin). 
> Running the job a third time resulted in a graph layout which was again 
> different from both runs before. However, two of the {{MapPartition}} 
> operators had not inputs just as in the first run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3025:
-

 Summary: Flink Kafka consumer may get stuck due to Kafka/Zookeeper 
client bug
 Key: FLINK-3025
 URL: https://issues.apache.org/jira/browse/FLINK-3025
 Project: Flink
  Issue Type: Bug
  Components: Streaming Connectors
Affects Versions: 0.10.0, 1.0.0
Reporter: Gyula Fora
Priority: Critical


In some cases the Flink kafka consumer might fail due to 
https://issues.apache.org/jira/browse/KAFKA-824.

Subsequently it can happen that the sources gets stuck in a Zookeeper client 
call (zookeeper bug).

A proposed fix would be bumping the zookeeper dependency to a version that 
includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008491#comment-15008491
 ] 

ASF GitHub Bot commented on FLINK-3025:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/1365

[FLINK-3025] [kafka consumer] Bump transitive ZkClient dependency to 0.7 
for bugfixes

Gives us a fix against deadlocks in the ZkClient which have caused the 
Kafka Consumer to freeze.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink zk_version_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1365.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1365


commit 97715cd9e6ad88a1b7d1178f13e1e12fd0544996
Author: Stephan Ewen 
Date:   2015-11-17T11:03:09Z

[FLINK-3025] [kafka consumer] Bump transitive ZkClient dependency to 0.7 
for bugfixes




> Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug
> 
>
> Key: FLINK-3025
> URL: https://issues.apache.org/jira/browse/FLINK-3025
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Critical
>
> In some cases the Flink kafka consumer might fail due to 
> https://issues.apache.org/jira/browse/KAFKA-824.
> Subsequently it can happen that the sources gets stuck in a Zookeeper client 
> call (zookeeper bug).
> A proposed fix would be bumping the zookeeper dependency to a version that 
> includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2680) Create a dedicated aligned-event time window operator

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2680.
---

> Create a dedicated aligned-event time window operator
> -
>
> Key: FLINK-2680
> URL: https://issues.apache.org/jira/browse/FLINK-2680
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Aligned event time windows (where every key triggers and evicts in the same 
> way) are the most common case, and should have a dedicated high-performance 
> implementation that supports:
>   - pre-aggregated state
>   - non aggregated state



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2680) Create a dedicated aligned-event time window operator

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2680.
-
Resolution: Won't Fix

Rather, we should unify the processing and event time windows and make only 
distinctions on aligned/unaligned triggering behavior.

> Create a dedicated aligned-event time window operator
> -
>
> Key: FLINK-2680
> URL: https://issues.apache.org/jira/browse/FLINK-2680
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Aligned event time windows (where every key triggers and evicts in the same 
> way) are the most common case, and should have a dedicated high-performance 
> implementation that supports:
>   - pre-aggregated state
>   - non aggregated state



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3021) Job submission times out due to classloading issue on JobManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008632#comment-15008632
 ] 

ASF GitHub Bot commented on FLINK-3021:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1368

[FLINK-3021] Fix class loading issue for streaming sources

Streaming sources were directly assigned their InputFormat in the 
StreamingJobGraphGenerator. As a consequence, the input formats were directly 
serialized/deserialized by Akka when the JobGraph was sent to the JobManager. 
In cases where the user provided a custom input format or an input format with 
custom types, this could lead to a ClassDefNotFoundException, because the 
system class loader instead of the user code class loader is used by Akka for 
the deserialization.

The problem was fixed by wrapping the InputFormat into a 
UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's 
configuration. By instantiating stream sources as InputFormatVertices, the 
corresponding InputFormat is retrieved from the Configuration in the 
initializeOnMaster method call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixInputFormatClassloading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1368.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1368


commit d841654e392d5114cd2ce62bc47fb68ed8198a52
Author: Till Rohrmann 
Date:   2015-11-17T13:23:31Z

[FLINK-3021] Fix class loading issue for streaming sources

Streaming sources were directly assigned their InputFormat in the 
StreamingJobGraphGenerator. As a consequence, the input formats were directly 
serialized/deserialized by Akka when the JobGraph was sent to the JobManager. 
In cases where the user provided a custom input format or an input format with 
custom types, this could lead to a ClassDefNotFoundException, because the 
system class loader instead of the user code class loader is used by Akka for 
the deserialization.

The problem was fixed by wrapping the InputFormat into a 
UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's 
configuration. By instantiating stream sources as InputFormatVertices, the 
corresponding InputFormat is retrieved from the Configuration in the 
initializeOnMaster method call.




> Job submission times out due to classloading issue on JobManager
> 
>
> Key: FLINK-3021
> URL: https://issues.apache.org/jira/browse/FLINK-3021
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>Priority: Critical
>
> A user reported the following issue when submitting a very simple job using 
> the {{DataStream}} API:
> {code}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: 
> Communication with JobManager failed: Job submission to the JobManager timed 
> out.
>   at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:141)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:368)
>   ... 13 more
> Caused by: 
> org.apache.flink.runtime.client.JobClientActorSubmissionTimeoutException: Job 
> submission to the JobManager timed out.
>   at 
> org.apache.flink.runtime.client.JobClientActor.handleMessage(JobClientActor.java:255)
>   at 
> org.apache.flink.runtime.akka.FlinkUntypedActor.handleLeaderSessionID(FlinkUntypedActor.java:88)
>   at 
> org.apache.flink.runtime.akka.FlinkUntypedActor.onReceive(FlinkUntypedActor.java:68)
>   at 
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> {code}
> The problem is that akka can not deserialize the job submit message on the 
> JobManager. From the logs, the issue becomes apparent:
> {code}
> 22:14:12,964 DEBUG akka.serialization.Serialization(akka://flink) 
>- Using serializer[akka.serialization.JavaSerializer] for message 
> [akka.actor.ActorIdentity]
> 22:14:12,995 DEBUG akka.serialization.Serialization(akka://flink) 
>- Using serializer[akka.serialization.JavaSerializer] for message 
> [java.lang.Integer]
> 22:14:13,007 DEBUG org.apache.flink.runtime.blob.BlobServerConnection 
>- Received PUT request for content addressable BLOB
> 22:14:13,134 ERROR akka.remote.EndpointWriter 
>- AssociationError [akka.tcp://flink@127.0.0.1:6123] <- 
> 

[jira] [Resolved] (FLINK-2670) Unstable CombineTaskTest

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2670.
-
   Resolution: Fixed
Fix Version/s: (was: 1.0.0)
   0.10.0

Fixed a while back in 361947d6c5490f0c6d04ffec709a353995aad373


> Unstable CombineTaskTest
> 
>
> Key: FLINK-2670
> URL: https://issues.apache.org/jira/browse/FLINK-2670
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Stephan Ewen
>Priority: Critical
>  Labels: test-stability
> Fix For: 0.10.0
>
>
> Fails with
> {noformat}
> ==
> Maven produced no output for 300 seconds.
> ==
> {noformat}
> https://travis-ci.org/apache/flink/jobs/80344487



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2670) Unstable CombineTaskTest

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2670.
---

> Unstable CombineTaskTest
> 
>
> Key: FLINK-2670
> URL: https://issues.apache.org/jira/browse/FLINK-2670
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Stephan Ewen
>Priority: Critical
>  Labels: test-stability
> Fix For: 0.10.0
>
>
> Fails with
> {noformat}
> ==
> Maven produced no output for 300 seconds.
> ==
> {noformat}
> https://travis-ci.org/apache/flink/jobs/80344487



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3021] Fix class loading issue for strea...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1368

[FLINK-3021] Fix class loading issue for streaming sources

Streaming sources were directly assigned their InputFormat in the 
StreamingJobGraphGenerator. As a consequence, the input formats were directly 
serialized/deserialized by Akka when the JobGraph was sent to the JobManager. 
In cases where the user provided a custom input format or an input format with 
custom types, this could lead to a ClassDefNotFoundException, because the 
system class loader instead of the user code class loader is used by Akka for 
the deserialization.

The problem was fixed by wrapping the InputFormat into a 
UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's 
configuration. By instantiating stream sources as InputFormatVertices, the 
corresponding InputFormat is retrieved from the Configuration in the 
initializeOnMaster method call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixInputFormatClassloading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1368.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1368


commit d841654e392d5114cd2ce62bc47fb68ed8198a52
Author: Till Rohrmann 
Date:   2015-11-17T13:23:31Z

[FLINK-3021] Fix class loading issue for streaming sources

Streaming sources were directly assigned their InputFormat in the 
StreamingJobGraphGenerator. As a consequence, the input formats were directly 
serialized/deserialized by Akka when the JobGraph was sent to the JobManager. 
In cases where the user provided a custom input format or an input format with 
custom types, this could lead to a ClassDefNotFoundException, because the 
system class loader instead of the user code class loader is used by Akka for 
the deserialization.

The problem was fixed by wrapping the InputFormat into a 
UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's 
configuration. By instantiating stream sources as InputFormatVertices, the 
corresponding InputFormat is retrieved from the Configuration in the 
initializeOnMaster method call.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2967] Enhance TaskManager network detec...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1361#discussion_r45052205
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/net/ConnectionUtils.java 
---
@@ -180,7 +180,41 @@ public static InetAddress 
findConnectingAddress(InetSocketAddress targetAddress,
}
}
 
+   /**
+* This utility method tries to connect to the JobManager using the 
InetAddress returned by
+* InetAddress.getLocalHost(). The purpose of the utility is to have a 
final try connecting to
+* the target address using the LocalHost before using the address 
returned.
+* We do a second try because the JM might have been unavailable during 
the first check.
+*
+* @param preliminaryResult The address detected by the heuristic
+* @return either the preliminaryResult or the address returned by 
InetAddress.getLocalHost() (if
+*  we are able to connect to targetAddress from 
there)
+*/
+   private static InetAddress tryLocalHostBeforeReturning(InetAddress 
preliminaryResult, SocketAddress targetAddress, boolean logging) throws 
IOException {
+   InetAddress localhostName = InetAddress.getLocalHost();
+   if(tryToConnect(localhostName, targetAddress, 
AddressDetectionState.LOCAL_HOST.getTimeout(), logging)) {
--- End diff --

I think we can use a bit higher timeout here. 200ms is probably mostly 
enough, but why make it so short here? If it succeeds fast, it does not add 
delay, and it it does not succeed we went through the other interfaces already 
anyways and better spend another second to make sure we get it right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2967) TM address detection might not always detect the right interface on slow networks / overloaded JMs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008556#comment-15008556
 ] 

ASF GitHub Bot commented on FLINK-2967:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1361#discussion_r45052205
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/net/ConnectionUtils.java 
---
@@ -180,7 +180,41 @@ public static InetAddress 
findConnectingAddress(InetSocketAddress targetAddress,
}
}
 
+   /**
+* This utility method tries to connect to the JobManager using the 
InetAddress returned by
+* InetAddress.getLocalHost(). The purpose of the utility is to have a 
final try connecting to
+* the target address using the LocalHost before using the address 
returned.
+* We do a second try because the JM might have been unavailable during 
the first check.
+*
+* @param preliminaryResult The address detected by the heuristic
+* @return either the preliminaryResult or the address returned by 
InetAddress.getLocalHost() (if
+*  we are able to connect to targetAddress from 
there)
+*/
+   private static InetAddress tryLocalHostBeforeReturning(InetAddress 
preliminaryResult, SocketAddress targetAddress, boolean logging) throws 
IOException {
+   InetAddress localhostName = InetAddress.getLocalHost();
+   if(tryToConnect(localhostName, targetAddress, 
AddressDetectionState.LOCAL_HOST.getTimeout(), logging)) {
--- End diff --

I think we can use a bit higher timeout here. 200ms is probably mostly 
enough, but why make it so short here? If it succeeds fast, it does not add 
delay, and it it does not succeed we went through the other interfaces already 
anyways and better spend another second to make sure we get it right.


> TM address detection might not always detect the right interface on slow 
> networks / overloaded JMs
> --
>
> Key: FLINK-2967
> URL: https://issues.apache.org/jira/browse/FLINK-2967
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.9, 0.10.0, 1.0.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I'm talking to a user which is facing the following issue:
> Some of the TaskManagers select the wrong IP address out of the available 
> network interfaces.
> The first address we try to connect to is the one returned by 
> {{InetAddress.getLocalHost()}}. This address is the right IP address to use, 
> but the JobManager is not able to respond within the timeout (50ms) to that 
> connection request.
> So the TM tries the next address, which is not publicly reachable. However, 
> the TM can connect to the JM from there. Netty will later fail to connect to 
> the TM from the other TMs.
> There are two solutions for this issue:
> - Allow users to configure a higher timeout for the first address detection 
> strategy. In most cases, the address returned by 
> {{InetAddress.getLocalHost()}} is correct. By setting a high timeout, users 
> with slow networks / overloaded JMs can make sure the TM picks this address
> - add an Akka message which we send from the TM to the JM, and the JM tries 
> to connect to the TM. If that succeeds, we know that the TM is reachable from 
> the outside.
> The problem is that we have to start a separate actor system on the 
> TaskManager first. We have to do this because might use a wrong ip address 
> for the TM (so we might end up starting actor systems until we found an 
> externally reachable ip)
> I'm first going to implement the first approach. If that solution works well 
> for my user, I'll contribute this to 0.10 / 1.0.
> If not, I'll implement the second approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2967) TM address detection might not always detect the right interface on slow networks / overloaded JMs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008557#comment-15008557
 ] 

ASF GitHub Bot commented on FLINK-2967:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1361#discussion_r45052228
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/net/ConnectionUtils.java 
---
@@ -180,7 +180,41 @@ public static InetAddress 
findConnectingAddress(InetSocketAddress targetAddress,
}
}
 
+   /**
+* This utility method tries to connect to the JobManager using the 
InetAddress returned by
+* InetAddress.getLocalHost(). The purpose of the utility is to have a 
final try connecting to
+* the target address using the LocalHost before using the address 
returned.
+* We do a second try because the JM might have been unavailable during 
the first check.
+*
+* @param preliminaryResult The address detected by the heuristic
+* @return either the preliminaryResult or the address returned by 
InetAddress.getLocalHost() (if
+*  we are able to connect to targetAddress from 
there)
+*/
+   private static InetAddress tryLocalHostBeforeReturning(InetAddress 
preliminaryResult, SocketAddress targetAddress, boolean logging) throws 
IOException {
+   InetAddress localhostName = InetAddress.getLocalHost();
+   if(tryToConnect(localhostName, targetAddress, 
AddressDetectionState.LOCAL_HOST.getTimeout(), logging)) {
--- End diff --

Also: code style, space


> TM address detection might not always detect the right interface on slow 
> networks / overloaded JMs
> --
>
> Key: FLINK-2967
> URL: https://issues.apache.org/jira/browse/FLINK-2967
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.9, 0.10.0, 1.0.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I'm talking to a user which is facing the following issue:
> Some of the TaskManagers select the wrong IP address out of the available 
> network interfaces.
> The first address we try to connect to is the one returned by 
> {{InetAddress.getLocalHost()}}. This address is the right IP address to use, 
> but the JobManager is not able to respond within the timeout (50ms) to that 
> connection request.
> So the TM tries the next address, which is not publicly reachable. However, 
> the TM can connect to the JM from there. Netty will later fail to connect to 
> the TM from the other TMs.
> There are two solutions for this issue:
> - Allow users to configure a higher timeout for the first address detection 
> strategy. In most cases, the address returned by 
> {{InetAddress.getLocalHost()}} is correct. By setting a high timeout, users 
> with slow networks / overloaded JMs can make sure the TM picks this address
> - add an Akka message which we send from the TM to the JM, and the JM tries 
> to connect to the TM. If that succeeds, we know that the TM is reachable from 
> the outside.
> The problem is that we have to start a separate actor system on the 
> TaskManager first. We have to do this because might use a wrong ip address 
> for the TM (so we might end up starting actor systems until we found an 
> externally reachable ip)
> I'm first going to implement the first approach. If that solution works well 
> for my user, I'll contribute this to 0.10 / 1.0.
> If not, I'll implement the second approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2967] Enhance TaskManager network detec...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1361#discussion_r45052228
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/net/ConnectionUtils.java 
---
@@ -180,7 +180,41 @@ public static InetAddress 
findConnectingAddress(InetSocketAddress targetAddress,
}
}
 
+   /**
+* This utility method tries to connect to the JobManager using the 
InetAddress returned by
+* InetAddress.getLocalHost(). The purpose of the utility is to have a 
final try connecting to
+* the target address using the LocalHost before using the address 
returned.
+* We do a second try because the JM might have been unavailable during 
the first check.
+*
+* @param preliminaryResult The address detected by the heuristic
+* @return either the preliminaryResult or the address returned by 
InetAddress.getLocalHost() (if
+*  we are able to connect to targetAddress from 
there)
+*/
+   private static InetAddress tryLocalHostBeforeReturning(InetAddress 
preliminaryResult, SocketAddress targetAddress, boolean logging) throws 
IOException {
+   InetAddress localhostName = InetAddress.getLocalHost();
+   if(tryToConnect(localhostName, targetAddress, 
AddressDetectionState.LOCAL_HOST.getTimeout(), logging)) {
--- End diff --

Also: code style, space


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Fixed FLINK-2942

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1346#issuecomment-157373231
  
Will merge this for 1.0 and 0.10.0 ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-297) Redesign GUI client-server model

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi resolved FLINK-297.
---
Resolution: Fixed

> Redesign GUI client-server model
> 
>
> Key: FLINK-297
> URL: https://issues.apache.org/jira/browse/FLINK-297
> Project: Flink
>  Issue Type: Improvement
>Reporter: GitHub Import
>  Labels: github-import
> Fix For: pre-apache
>
>
> Factor out job manager status information as REST service running inside the 
> same process. Implement visualization server as a separate web application 
> that runs on the client-side and renders data fetched from from the job 
> manager RESTful API.
>  Imported from GitHub 
> Url: https://github.com/stratosphere/stratosphere/issues/297
> Created by: [aalexandrov|https://github.com/aalexandrov]
> Labels: enhancement, gui, 
> Created at: Tue Nov 26 14:54:53 CET 2013
> State: open



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Change the parameter 'numSample' in DataSetUti...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1362#issuecomment-157342698
  
Looks good, will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008568#comment-15008568
 ] 

ASF GitHub Bot commented on FLINK-3025:
---

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1365#issuecomment-157354372
  
:+1: 


> Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug
> 
>
> Key: FLINK-3025
> URL: https://issues.apache.org/jira/browse/FLINK-3025
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Critical
>
> In some cases the Flink kafka consumer might fail due to 
> https://issues.apache.org/jira/browse/KAFKA-824.
> Subsequently it can happen that the sources gets stuck in a Zookeeper client 
> call (zookeeper bug).
> A proposed fix would be bumping the zookeeper dependency to a version that 
> includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3025] [kafka consumer] Bump transitive ...

[gyfora]

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1365#issuecomment-157354372
  
:+1: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2913] Close of ObjectOutputStream shoul...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1353#issuecomment-157345512
  
How about using a try-with-resources statement here?
I would like to use this whenever we touch code that needs to ensure 
closing a resource...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2913) Close of ObjectOutputStream should be enclosed in finally block in FsStateBackend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008524#comment-15008524
 ] 

ASF GitHub Bot commented on FLINK-2913:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1353#issuecomment-157345512
  
How about using a try-with-resources statement here?
I would like to use this whenever we touch code that needs to ensure 
closing a resource...


> Close of ObjectOutputStream should be enclosed in finally block in 
> FsStateBackend
> -
>
> Key: FLINK-2913
> URL: https://issues.apache.org/jira/browse/FLINK-2913
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
>   ObjectOutputStream os = new ObjectOutputStream(outStream);
>   os.writeObject(state);
>   os.close();
> {code}
> If IOException is thrown out of writeObject(), the close() call would be 
> skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3025] [kafka consumer] Bump transitive ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1365#issuecomment-157362141
  
Tests pass, will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3027) Cancel button in Flink job dashboard does not work

[Krzysztof Zarzycki (JIRA)]

Krzysztof Zarzycki created FLINK-3027:
-

 Summary: Cancel button in Flink job dashboard does not work
 Key: FLINK-3027
 URL: https://issues.apache.org/jira/browse/FLINK-3027
 Project: Flink
  Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Krzysztof Zarzycki


I  tried to cancel Flink job through Flink dashboard, but the button seems not 
working : The string "Cancel" changes to "Cancelling..." But application 
doesn't cancel. After refresh, the button goes back to "Cancel".

Cancelling by command line works fine though. I use: 
{code}
bin/flink cancel 
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2504) ExternalSortLargeRecordsITCase.testSortWithLongAndShortRecordsMixed failed spuriously

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008620#comment-15008620
 ] 

Stephan Ewen commented on FLINK-2504:
-

I have over the days spent quite a bit of time with the code trying to 
reproduce this.

Would like to close this as "Cannot Reproduce". Might really be a spurious I/O 
error / bit flip on Travis...

> ExternalSortLargeRecordsITCase.testSortWithLongAndShortRecordsMixed failed 
> spuriously
> -
>
> Key: FLINK-2504
> URL: https://issues.apache.org/jira/browse/FLINK-2504
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> The test 
> {{ExternalSortLargeRecordsITCase.testSortWithLongAndShortRecordsMixed}} 
> failed in one of my Travis builds: 
> https://travis-ci.org/tillrohrmann/flink/jobs/74881883



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2914] Add missing break Statement in ZK...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1359#issuecomment-157343610
  
Looks correct (and affects only logging).

Will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2351) Deprecate config builders in InputFormats and Output formats

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008589#comment-15008589
 ] 

Stephan Ewen commented on FLINK-2351:
-

Any news on this one? Would be great to include this with 1.0

> Deprecate config builders in InputFormats and Output formats
> 
>
> Key: FLINK-2351
> URL: https://issues.apache.org/jira/browse/FLINK-2351
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.0.0
>
>
> Old APIs used to pass functions as classes and parameters as configs. To 
> support that, all input- and output formats had config builders.
> As the record API is deprecated and about to be removed, these builders are 
> no longer needed and should be deprecated and removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008587#comment-15008587
 ] 

ASF GitHub Bot commented on FLINK-3025:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1365#issuecomment-157362141
  
Tests pass, will merge this...


> Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug
> 
>
> Key: FLINK-3025
> URL: https://issues.apache.org/jira/browse/FLINK-3025
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Critical
>
> In some cases the Flink kafka consumer might fail due to 
> https://issues.apache.org/jira/browse/KAFKA-824.
> Subsequently it can happen that the sources gets stuck in a Zookeeper client 
> call (zookeeper bug).
> A proposed fix would be bumping the zookeeper dependency to a version that 
> includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2676) Add abstraction for keyed window state

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2676.
-
Resolution: Won't Fix

> Add abstraction for keyed window state
> --
>
> Key: FLINK-2676
> URL: https://issues.apache.org/jira/browse/FLINK-2676
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> This abstraction should help to seamlessly switch between window state kept as
>   - Heap-resident maps
>   - Managed memory spillable maps
>   - key/value state backend state
> I would approach this abstraction once we implemented a few window operators 
> and see what operations we need, such as
>   - Drop time-regions across all keys
>   - Append to state for key
>   - Update/replace state by key
>   - Iterate over unions of multiple state time regions
>   - snapshot time regions completely / incrementally
>   - (possibly more)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2676) Add abstraction for keyed window state

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008622#comment-15008622
 ] 

Stephan Ewen commented on FLINK-2676:
-

Approach changed a bit. Will file a more detailed followup at some point.

> Add abstraction for keyed window state
> --
>
> Key: FLINK-2676
> URL: https://issues.apache.org/jira/browse/FLINK-2676
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> This abstraction should help to seamlessly switch between window state kept as
>   - Heap-resident maps
>   - Managed memory spillable maps
>   - key/value state backend state
> I would approach this abstraction once we implemented a few window operators 
> and see what operations we need, such as
>   - Drop time-regions across all keys
>   - Append to state for key
>   - Update/replace state by key
>   - Iterate over unions of multiple state time regions
>   - snapshot time regions completely / incrementally
>   - (possibly more)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2676) Add abstraction for keyed window state

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2676.
---

> Add abstraction for keyed window state
> --
>
> Key: FLINK-2676
> URL: https://issues.apache.org/jira/browse/FLINK-2676
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> This abstraction should help to seamlessly switch between window state kept as
>   - Heap-resident maps
>   - Managed memory spillable maps
>   - key/value state backend state
> I would approach this abstraction once we implemented a few window operators 
> and see what operations we need, such as
>   - Drop time-regions across all keys
>   - Append to state for key
>   - Update/replace state by key
>   - Iterate over unions of multiple state time regions
>   - snapshot time regions completely / incrementally
>   - (possibly more)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-2750) FileStateHandleTest fails when building for Hadoop 2.6.0

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen reassigned FLINK-2750:
---

Assignee: Stephan Ewen

> FileStateHandleTest fails when building for Hadoop 2.6.0
> 
>
> Key: FLINK-2750
> URL: https://issues.apache.org/jira/browse/FLINK-2750
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Fabian Hueske
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> The {{FileStateHandleTest}} fails when building the 0.10.0-milestone-1-rc1 
> with {{mvn clean install -Dhadoop.version=2.6.0}} with the following exception
> {code}
> java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3028) Cannot cancel restarting job via web frontend

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-3028:
--

 Summary: Cannot cancel restarting job via web frontend
 Key: FLINK-3028
 URL: https://issues.apache.org/jira/browse/FLINK-3028
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Affects Versions: 0.10.0
Reporter: Ufuk Celebi


During restart the cancel button is not shown and hence a job cannot be 
canceled via the web frontend during restarts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Vasia Kalavri (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008883#comment-15008883
 ] 

Vasia Kalavri commented on FLINK-3002:
--

Alright, specifically creating a {{Left}} means I can reuse when the input is 
also a {{Left}}.
I think things are clear now. I'll update and open a PR.

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2914) Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008516#comment-15008516
 ] 

ASF GitHub Bot commented on FLINK-2914:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1359#issuecomment-157343610
  
Looks correct (and affects only logging).

Will merge this...


> Missing break in 
> ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
> -
>
> Key: FLINK-2914
> URL: https://issues.apache.org/jira/browse/FLINK-2914
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
>
> {code}
> case CONNECTION_SUSPENDED:
>   LOG.warn("ZooKeeper connection SUSPENDED. Changes to the submitted 
> job " +
>   "graphs are not monitored (temporarily).");
> case CONNECTION_LOST:
>   LOG.warn("ZooKeeper connection LOST. Changes to the submitted job " 
> +
>   "graphs are not monitored (permanently).");
>   break;
> case CONNECTION_RECONNECTED:
>   LOG.info("ZooKeeper connection RECONNECTED. Changes to the 
> submitted job " +
>   "graphs are monitored again.");
> case INITIALIZED:
>   LOG.info("SubmittedJobGraphsPathCacheListener initialized");
>   break;
> {code}
> For CONNECTION_SUSPENDED and CONNECTION_RECONNECTED, the break statement is 
> missing.
> This would result in unrelated event logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2879) Links in documentation are broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008531#comment-15008531
 ] 

ASF GitHub Bot commented on FLINK-2879:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1348#issuecomment-157346537
  
Looks good, merging this!

BTW: It also affects the docs of the 0.10 release, so I will cherry pick it 
into the 0.10.x branch as well.


> Links in documentation are broken
> -
>
> Key: FLINK-2879
> URL: https://issues.apache.org/jira/browse/FLINK-2879
> Project: Flink
>  Issue Type: Bug
>  Components: website
>Reporter: Nikolaas Steenbergen
>Assignee: Andra Lungu
>Priority: Minor
>
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html
> the image of system components link to wrong locations:
> e.g.:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/internals/general_arch.html
> instead of:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2879] [docs] Fixed broken links on the ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1348#issuecomment-157346537
  
Looks good, merging this!

BTW: It also affects the docs of the 0.10 release, so I will cherry pick it 
into the 0.10.x branch as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2351) Deprecate config builders in InputFormats and Output formats

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008604#comment-15008604
 ] 

Chesnay Schepler commented on FLINK-2351:
-

i planned on removing the stuff when the Record API is removed.

> Deprecate config builders in InputFormats and Output formats
> 
>
> Key: FLINK-2351
> URL: https://issues.apache.org/jira/browse/FLINK-2351
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.0.0
>
>
> Old APIs used to pass functions as classes and parameters as configs. To 
> support that, all input- and output formats had config builders.
> As the record API is deprecated and about to be removed, these builders are 
> no longer needed and should be deprecated and removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3018) Job state discarded from web interface for restarting jobs

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008778#comment-15008778
 ] 

Gyula Fora commented on FLINK-3018:
---

You are right but it is still confusing for jobs that fail sometimes. 
I agree that current throughput and latency would mean much more but that 
information is not currently available. Throughput could be estimated from time 
and processed records (although it would be better to see it somewhere).

> Job state discarded from web interface for restarting jobs
> --
>
> Key: FLINK-3018
> URL: https://issues.apache.org/jira/browse/FLINK-3018
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Gyula Fora
>Priority: Minor
>
> When a streaming job goes into the restarting status and recovers, the web ui 
> information is totally lost (number of records/bytes processed etc).
> This is very misleading and should not happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3029) JobGraph is not connected correctly in webfrontend for streaming job

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3029.
---

> JobGraph is not connected correctly in webfrontend for streaming job
> 
>
> Key: FLINK-3029
> URL: https://issues.apache.org/jira/browse/FLINK-3029
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Minor
>
> I have a streaming job consisting of a Source with parallelism 4  and I have 
> 3 different flatmaps with parallelism 24 each. 
> In the web interface the source is only shown to be connected to the first 
> flatmap, and the remaining 2 flatmaps appear as though if they were sources 
> with no input stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2914) Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2914.
---

> Missing break in 
> ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
> -
>
> Key: FLINK-2914
> URL: https://issues.apache.org/jira/browse/FLINK-2914
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.0.0
>
>
> {code}
> case CONNECTION_SUSPENDED:
>   LOG.warn("ZooKeeper connection SUSPENDED. Changes to the submitted 
> job " +
>   "graphs are not monitored (temporarily).");
> case CONNECTION_LOST:
>   LOG.warn("ZooKeeper connection LOST. Changes to the submitted job " 
> +
>   "graphs are not monitored (permanently).");
>   break;
> case CONNECTION_RECONNECTED:
>   LOG.info("ZooKeeper connection RECONNECTED. Changes to the 
> submitted job " +
>   "graphs are monitored again.");
> case INITIALIZED:
>   LOG.info("SubmittedJobGraphsPathCacheListener initialized");
>   break;
> {code}
> For CONNECTION_SUSPENDED and CONNECTION_RECONNECTED, the break statement is 
> missing.
> This would result in unrelated event logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3029) JobGraph is not connected correctly in webfrontend for streaming job

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3029.
-
Resolution: Duplicate

Duplicate of FLINK-2942

> JobGraph is not connected correctly in webfrontend for streaming job
> 
>
> Key: FLINK-3029
> URL: https://issues.apache.org/jira/browse/FLINK-3029
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Minor
>
> I have a streaming job consisting of a Source with parallelism 4  and I have 
> 3 different flatmaps with parallelism 24 each. 
> In the web interface the source is only shown to be connected to the first 
> flatmap, and the remaining 2 flatmaps appear as though if they were sources 
> with no input stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3023) Show Flink version + commit id for -SNAPSHOT versions in web frontend

[Chiwan Park (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008835#comment-15008835
 ] 

Chiwan Park commented on FLINK-3023:


+1 and I think that this improvement would be good for Flink Scala shell.

> Show Flink version + commit id for -SNAPSHOT versions in web frontend
> -
>
> Key: FLINK-3023
> URL: https://issues.apache.org/jira/browse/FLINK-3023
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Robert Metzger
>
> The old frontend was showing the Flink version and the commit id for SNAPSHOT 
> builds.
> This is a helpful feature to quickly see which Flink version is running.
> It would be nice to add this again to the web interface.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3018) Job state discarded from web interface for restarting jobs

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008834#comment-15008834
 ] 

Stephan Ewen commented on FLINK-3018:
-

I would like to have an "accumulated num records" counter somewhere, that would 
be great. We could sum up over all exec attempts. It may have wither duplicates 
or missing records, though, as the metrics are not checkpointed.

> Job state discarded from web interface for restarting jobs
> --
>
> Key: FLINK-3018
> URL: https://issues.apache.org/jira/browse/FLINK-3018
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Gyula Fora
>Priority: Minor
>
> When a streaming job goes into the restarting status and recovers, the web ui 
> information is totally lost (number of records/bytes processed etc).
> This is very misleading and should not happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3009) Cannot build docs with Jekyll 3.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008858#comment-15008858
 ] 

ASF GitHub Bot commented on FLINK-3009:
---

Github user hsaputra commented on a diff in the pull request:

https://github.com/apache/flink/pull/1363#discussion_r45075237
  
--- Diff: docs/docker/run.sh ---
@@ -0,0 +1,74 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e -x -u
+
+SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+export IMAGE_NAME="flink/docs"
+
+pushd ${SCRIPT_DIR}
+
+docker build --rm=true -t ${IMAGE_NAME} .
+
+popd
+
+if [ "$(uname -s)" == "Linux" ]; then
+  USER_NAME=${SUDO_USER:=$USER}
+  USER_ID=$(id -u "${USER_NAME}")
+  GROUP_ID=$(id -g "${USER_NAME}")
+else # boot2docker uid and gid
+  USER_NAME=$USER
+  USER_ID=1000
+  GROUP_ID=50
+fi
+
+docker build -t "${IMAGE_NAME}-${USER_NAME}" -  Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Priority: Minor
>
> Building the docs with the newly released Jekyll 3.0.0 fails:
> {code}
> Configuration file: /Users/ufuk/code/flink/docs/_config.yml
> /Users/ufuk/code/flink/docs/_plugins/removeDuplicateLicenseHeaders.rb:63:in 
> `': cannot load such file -- jekyll/post (LoadError)
>   from 
> /Users/ufuk/code/flink/docs/_plugins/removeDuplicateLicenseHeaders.rb:25:in 
> `'
>   from 
> /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in
>  `require'
>   from 
> /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in
>  `require'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:75:in 
> `block (2 levels) in require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:74:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:74:in 
> `block in require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:73:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:73:in 
> `require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:18:in 
> `conscientious_require'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/site.rb:97:in `setup'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/site.rb:49:in 
> `initialize'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:30:in 
> `new'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:30:in 
> `process'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:18:in 
> `block (2 levels) in init_with_program'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `call'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `block in execute'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `execute'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/program.rb:42:in 
> `go'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary.rb:19:in `program'
>   from /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/bin/jekyll:17:in ` (required)>'
>   from /usr/local/bin/jekyll:23:in `load'
>   from /usr/local/bin/jekyll:23:in `'
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3009] Add dockerized jekyll environment

[hsaputra]

Github user hsaputra commented on a diff in the pull request:

https://github.com/apache/flink/pull/1363#discussion_r45075237
  
--- Diff: docs/docker/run.sh ---
@@ -0,0 +1,74 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e -x -u
+
+SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+
+export IMAGE_NAME="flink/docs"
+
+pushd ${SCRIPT_DIR}
+
+docker build --rm=true -t ${IMAGE_NAME} .
+
+popd
+
+if [ "$(uname -s)" == "Linux" ]; then
+  USER_NAME=${SUDO_USER:=$USER}
+  USER_ID=$(id -u "${USER_NAME}")
+  GROUP_ID=$(id -g "${USER_NAME}")
+else # boot2docker uid and gid
+  USER_NAME=$USER
+  USER_ID=1000
+  GROUP_ID=50
+fi
+
+docker build -t "${IMAGE_NAME}-${USER_NAME}" - 

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008768#comment-15008768
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157391821
  
What if a user registers a custom type in Kryo on the client side? That 
type would receive a different ID, because fewer types have been registered at 
the client compared to the Chill initialized Kryo instance at the server. Or am 
I missing something here?


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3013) Incorrect package declaration in GellyScalaAPICompletenessTest.scala

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008743#comment-15008743
 ] 

ASF GitHub Bot commented on FLINK-3013:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1356#issuecomment-157385739
  
Thanks a lot for the PR @smarthi! +1 to merge.


> Incorrect package declaration in GellyScalaAPICompletenessTest.scala
> 
>
> Key: FLINK-3013
> URL: https://issues.apache.org/jira/browse/FLINK-3013
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly, Scala API
>Affects Versions: 0.10.0
>Reporter: Suneel Marthi
>Assignee: Suneel Marthi
> Fix For: 1.0.0
>
>
> Incorrect package declaration in GellyScalaAPICompletenessTest.scala, 
> presently reads as:
> org.apache.flink.streaming.api.scala => org.apache.flink.graph.scala.test
> Also fix failing Gelly-Scala tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157391821
  
What if a user registers a custom type in Kryo on the client side? That 
type would receive a different ID, because fewer types have been registered at 
the client compared to the Chill initialized Kryo instance at the server. Or am 
I missing something here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2771) IterateTest.testSimpleIteration fails on Travis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008777#comment-15008777
 ] 

ASF GitHub Bot commented on FLINK-2771:
---

Github user mliesenberg commented on the pull request:

https://github.com/apache/flink/pull/1367#issuecomment-157394443
  
several travis builds failed:
oraclejdk8 & hadoop 2.7: https://issues.apache.org/jira/browse/FLINK-2771
oraclejdk8 & hadoop 2.5: https://issues.apache.org/jira/browse/FLINK-2392
openjdk7 & hadoop 2.4: error in test: 
JobManagerSubmittedJobGraphsRecoveryITCase.cleanUp:111 » ConnectionLoss 
Keeper...
 
I think all of those are not related to the javadoc change. :)


> IterateTest.testSimpleIteration fails on Travis
> ---
>
> Key: FLINK-2771
> URL: https://issues.apache.org/jira/browse/FLINK-2771
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> The {{IterateTest.testSimpleIteration}} failed on Travis with
> {code}
> Failed tests: 
>   IterateTest.testSimpleIteration:384 null
> {code}
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/81986242/log.txt 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3018) Job state discarded from web interface for restarting jobs

[Gyula Fora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gyula Fora updated FLINK-3018:
--
Priority: Minor  (was: Critical)

> Job state discarded from web interface for restarting jobs
> --
>
> Key: FLINK-3018
> URL: https://issues.apache.org/jira/browse/FLINK-3018
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Gyula Fora
>Priority: Minor
>
> When a streaming job goes into the restarting status and recovers, the web ui 
> information is totally lost (number of records/bytes processed etc).
> This is very misleading and should not happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Fix JavaDoc of ElasticsearchSink

[mliesenberg]

Github user mliesenberg commented on the pull request:

https://github.com/apache/flink/pull/1367#issuecomment-157394443
  
several travis builds failed:
oraclejdk8 & hadoop 2.7: https://issues.apache.org/jira/browse/FLINK-2771
oraclejdk8 & hadoop 2.5: https://issues.apache.org/jira/browse/FLINK-2392
openjdk7 & hadoop 2.4: error in test: 
JobManagerSubmittedJobGraphsRecoveryITCase.cleanUp:111 » ConnectionLoss 
Keeper...
 
I think all of those are not related to the javadoc change. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Vasia Kalavri (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008784#comment-15008784
 ] 

Vasia Kalavri commented on FLINK-3002:
--

No, in fact, I didn't go with the abstract version :-)
I do think Either should be abstract, but the Serializer needs to create an 
instance somehow.
I could remove the flags, just I think they're more user-friendly than 
getClass().
Also, when testing, I realized that {{copy}} and {{deserialize}} cannot use the 
{{reuse}} record if it's created with the empty Either constructor. Any ideas 
there? I have pushed the tests in the same branch.
I'll fix the arity, thanks!


> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2967) TM address detection might not always detect the right interface on slow networks / overloaded JMs

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2967.
-
   Resolution: Fixed
Fix Version/s: 0.10.1
   1.0.0

Fixed in 
  - 0.10 via cda00accd76d56eaea2ef679d5c9b3c0465059ca
  - 1.0 via a45212d2b18cc12e3c314ed43e8d19943693deed

> TM address detection might not always detect the right interface on slow 
> networks / overloaded JMs
> --
>
> Key: FLINK-2967
> URL: https://issues.apache.org/jira/browse/FLINK-2967
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.9, 0.10.0, 1.0.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.0.0, 0.10.1
>
>
> I'm talking to a user which is facing the following issue:
> Some of the TaskManagers select the wrong IP address out of the available 
> network interfaces.
> The first address we try to connect to is the one returned by 
> {{InetAddress.getLocalHost()}}. This address is the right IP address to use, 
> but the JobManager is not able to respond within the timeout (50ms) to that 
> connection request.
> So the TM tries the next address, which is not publicly reachable. However, 
> the TM can connect to the JM from there. Netty will later fail to connect to 
> the TM from the other TMs.
> There are two solutions for this issue:
> - Allow users to configure a higher timeout for the first address detection 
> strategy. In most cases, the address returned by 
> {{InetAddress.getLocalHost()}} is correct. By setting a high timeout, users 
> with slow networks / overloaded JMs can make sure the TM picks this address
> - add an Akka message which we send from the TM to the JM, and the JM tries 
> to connect to the TM. If that succeeds, we know that the TM is reachable from 
> the outside.
> The problem is that we have to start a separate actor system on the 
> TaskManager first. We have to do this because might use a wrong ip address 
> for the TM (so we might end up starting actor systems until we found an 
> externally reachable ip)
> I'm first going to implement the first approach. If that solution works well 
> for my user, I'll contribute this to 0.10 / 1.0.
> If not, I'll implement the second approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3025.
---

> Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug
> 
>
> Key: FLINK-3025
> URL: https://issues.apache.org/jira/browse/FLINK-3025
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0, 0.10.1
>
>
> In some cases the Flink kafka consumer might fail due to 
> https://issues.apache.org/jira/browse/KAFKA-824.
> Subsequently it can happen that the sources gets stuck in a Zookeeper client 
> call (zookeeper bug).
> A proposed fix would be bumping the zookeeper dependency to a version that 
> includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2942) Dangling operators in web UI's program visualization (non-deterministic)

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008814#comment-15008814
 ] 

Stephan Ewen commented on FLINK-2942:
-

Fixed in 0.10 via 9895e3e43823be11f058a00b9c4eb0c049dd91a8

> Dangling operators in web UI's program visualization (non-deterministic)
> 
>
> Key: FLINK-2942
> URL: https://issues.apache.org/jira/browse/FLINK-2942
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0
> Environment: OSX, Firefox and Chrome
>Reporter: Fabian Hueske
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2015-10-29 at 17.11.19.png, Screen Shot 
> 2015-10-29 at 20.51.46.png, Screen Shot 2015-10-29 at 20.52.13.png, Screen 
> Shot 2015-11-09 at 14.48.03.png
>
>
> When visualizing a program with three {{MapPartition}} operators that branch 
> off from an {{OuterJoin}} operator, two of the three {{MapPartition}} 
> operators are not connected to the {{OuterJoin}} operator and appear to have 
> no input.
> The problem is present in FireFox as well as in Chrome. I'll attach a 
> screenshot.
> The problem and be reproduced by executing the "Cascading for the impatient" 
> [TFIDF example 
> program|https://github.com/Cascading/Impatient/tree/master/part5] using the 
> [Cascading Flink Connector|https://github.com/dataArtisans/cascading-flink].
> Update: It appears that the problem is non-deterministic. I ran the same job 
> again (same setup) and the previously missing connections were visualized. 
> However, the UI showed only one input for a binary operator (OuterJoin). 
> Running the job a third time resulted in a graph layout which was again 
> different from both runs before. However, two of the {{MapPartition}} 
> operators had not inputs just as in the first run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2914) Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2914.
-
   Resolution: Fixed
Fix Version/s: 1.0.0

Fixed via 442fcb196f9bf9fada2278b78c8447e65f0c85e0

> Missing break in 
> ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
> -
>
> Key: FLINK-2914
> URL: https://issues.apache.org/jira/browse/FLINK-2914
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.0.0
>
>
> {code}
> case CONNECTION_SUSPENDED:
>   LOG.warn("ZooKeeper connection SUSPENDED. Changes to the submitted 
> job " +
>   "graphs are not monitored (temporarily).");
> case CONNECTION_LOST:
>   LOG.warn("ZooKeeper connection LOST. Changes to the submitted job " 
> +
>   "graphs are not monitored (permanently).");
>   break;
> case CONNECTION_RECONNECTED:
>   LOG.info("ZooKeeper connection RECONNECTED. Changes to the 
> submitted job " +
>   "graphs are monitored again.");
> case INITIALIZED:
>   LOG.info("SubmittedJobGraphsPathCacheListener initialized");
>   break;
> {code}
> For CONNECTION_SUSPENDED and CONNECTION_RECONNECTED, the break statement is 
> missing.
> This would result in unrelated event logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008831#comment-15008831
 ] 

Stephan Ewen commented on FLINK-3002:
-

The thing about {{Either}} is that you never create an {{Either}}, you always 
create a {{Left}} or {{Right}}.

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Vasia Kalavri (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008853#comment-15008853
 ] 

Vasia Kalavri commented on FLINK-3002:
--

I agree [~StephanEwen], that was my reasoning as well. What [~aljoscha] 
describes is exactly how I initially implemented this.
I just wasn't sure if it's allowed for {{createInstance}} to return {{null}}. 
Now, since this is OK, {{copy(Either from, Either reuse)}} and 
{{(Either reuse, DataInputView source)}} can't make use of the {{reuse}} 
record. Is this also OK?
Thank you both for the help!

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3009) Cannot build docs with Jekyll 3.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008862#comment-15008862
 ] 

ASF GitHub Bot commented on FLINK-3009:
---

Github user hsaputra commented on the pull request:

https://github.com/apache/flink/pull/1363#issuecomment-157408961
  
Other than small nit on text, +1

Seems like Travis fails unrelated to this patch


> Cannot build docs with Jekyll 3.0.0
> ---
>
> Key: FLINK-3009
> URL: https://issues.apache.org/jira/browse/FLINK-3009
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Priority: Minor
>
> Building the docs with the newly released Jekyll 3.0.0 fails:
> {code}
> Configuration file: /Users/ufuk/code/flink/docs/_config.yml
> /Users/ufuk/code/flink/docs/_plugins/removeDuplicateLicenseHeaders.rb:63:in 
> `': cannot load such file -- jekyll/post (LoadError)
>   from 
> /Users/ufuk/code/flink/docs/_plugins/removeDuplicateLicenseHeaders.rb:25:in 
> `'
>   from 
> /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in
>  `require'
>   from 
> /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in
>  `require'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:75:in 
> `block (2 levels) in require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:74:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:74:in 
> `block in require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:73:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:73:in 
> `require_plugin_files'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/plugin_manager.rb:18:in 
> `conscientious_require'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/site.rb:97:in `setup'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/site.rb:49:in 
> `initialize'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:30:in 
> `new'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:30:in 
> `process'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/lib/jekyll/commands/build.rb:18:in 
> `block (2 levels) in init_with_program'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `call'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `block in execute'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `each'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in 
> `execute'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary/program.rb:42:in 
> `go'
>   from 
> /Library/Ruby/Gems/2.0.0/gems/mercenary-0.3.5/lib/mercenary.rb:19:in `program'
>   from /Library/Ruby/Gems/2.0.0/gems/jekyll-3.0.0/bin/jekyll:17:in ` (required)>'
>   from /usr/local/bin/jekyll:23:in `load'
>   from /usr/local/bin/jekyll:23:in `'
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3009] Add dockerized jekyll environment

[hsaputra]

Github user hsaputra commented on the pull request:

https://github.com/apache/flink/pull/1363#issuecomment-157408961
  
Other than small nit on text, +1

Seems like Travis fails unrelated to this patch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008861#comment-15008861
 ] 

Stephan Ewen commented on FLINK-3002:
-

All object reuse is optional to be done where possible. If you do not reuse the 
objects, it is fine.

I think this is a fair case to not reuse them..

I would actually not return {{null}} in create instance, but create a {{Left}} 
(arbitrary). That record is used in some parts as a reuse record, having it non 
null may prevent null pointers in parts where we were not careful (just 
conservative coding). Since you do not reuse the types in the methods anyways, 
it is really arbitrary what non-null object you create in {{createInstance()}}.

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2879) Links in documentation are broken

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2879.
-
   Resolution: Fixed
Fix Version/s: 0.10.1
   1.0.0

Fixed in
 - 0.10 via d8ed8f260eb771a975135dedc1089df890a27e11
 - 1.0 via ffb8aecf2bf1ab64d2488ae37130f310f40b530e

> Links in documentation are broken
> -
>
> Key: FLINK-2879
> URL: https://issues.apache.org/jira/browse/FLINK-2879
> Project: Flink
>  Issue Type: Bug
>  Components: website
>Reporter: Nikolaas Steenbergen
>Assignee: Andra Lungu
>Priority: Minor
> Fix For: 1.0.0, 0.10.1
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html
> the image of system components link to wrong locations:
> e.g.:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/internals/general_arch.html
> instead of:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2879) Links in documentation are broken

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2879.
---

> Links in documentation are broken
> -
>
> Key: FLINK-2879
> URL: https://issues.apache.org/jira/browse/FLINK-2879
> Project: Flink
>  Issue Type: Bug
>  Components: website
>Reporter: Nikolaas Steenbergen
>Assignee: Andra Lungu
>Priority: Minor
> Fix For: 1.0.0, 0.10.1
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html
> the image of system components link to wrong locations:
> e.g.:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/internals/general_arch.html
> instead of:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2967) TM address detection might not always detect the right interface on slow networks / overloaded JMs

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2967.
---

> TM address detection might not always detect the right interface on slow 
> networks / overloaded JMs
> --
>
> Key: FLINK-2967
> URL: https://issues.apache.org/jira/browse/FLINK-2967
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.9, 0.10.0, 1.0.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.0.0, 0.10.1
>
>
> I'm talking to a user which is facing the following issue:
> Some of the TaskManagers select the wrong IP address out of the available 
> network interfaces.
> The first address we try to connect to is the one returned by 
> {{InetAddress.getLocalHost()}}. This address is the right IP address to use, 
> but the JobManager is not able to respond within the timeout (50ms) to that 
> connection request.
> So the TM tries the next address, which is not publicly reachable. However, 
> the TM can connect to the JM from there. Netty will later fail to connect to 
> the TM from the other TMs.
> There are two solutions for this issue:
> - Allow users to configure a higher timeout for the first address detection 
> strategy. In most cases, the address returned by 
> {{InetAddress.getLocalHost()}} is correct. By setting a high timeout, users 
> with slow networks / overloaded JMs can make sure the TM picks this address
> - add an Akka message which we send from the TM to the JM, and the JM tries 
> to connect to the TM. If that succeeds, we know that the TM is reachable from 
> the outside.
> The problem is that we have to start a separate actor system on the 
> TaskManager first. We have to do this because might use a wrong ip address 
> for the TM (so we might end up starting actor systems until we found an 
> externally reachable ip)
> I'm first going to implement the first approach. If that solution works well 
> for my user, I'll contribute this to 0.10 / 1.0.
> If not, I'll implement the second approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3025) Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3025.
-
   Resolution: Fixed
 Assignee: Stephan Ewen
Fix Version/s: 0.10.1
   1.0.0

Fixed in 
  - 0.10 via d209cdc64b960a335524430cefb7bd3a5fbabfed
  - 1.0 via a654760299f6698b2a2673ca5b6ffba38b39ea4e

> Flink Kafka consumer may get stuck due to Kafka/Zookeeper client bug
> 
>
> Key: FLINK-3025
> URL: https://issues.apache.org/jira/browse/FLINK-3025
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0, 0.10.1
>
>
> In some cases the Flink kafka consumer might fail due to 
> https://issues.apache.org/jira/browse/KAFKA-824.
> Subsequently it can happen that the sources gets stuck in a Zookeeper client 
> call (zookeeper bug).
> A proposed fix would be bumping the zookeeper dependency to a version that 
> includes the fix for this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008836#comment-15008836
 ] 

Aljoscha Krettek commented on FLINK-3002:
-

Hi,
you can remove the {{isLeft}}/{{isRight}} fields, add abstract methods 
{{isLeft}}/{{isRight}} in the base class and add concrete methods in {{Left}} 
and {{Right}}. This way you get around the wasteful fields.

Also, I think the base class should be abstract since it does not make sense to 
create an instance of that type. The {{TypeSerializer}} can just return 
{{null}} in {{createInstance}} since an un-initialized {{Either}} doesn't seem 
to be the correct behavior.

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2967) TM address detection might not always detect the right interface on slow networks / overloaded JMs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008691#comment-15008691
 ] 

ASF GitHub Bot commented on FLINK-2967:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1361#issuecomment-157378877
  
Will merge and address the comments...


> TM address detection might not always detect the right interface on slow 
> networks / overloaded JMs
> --
>
> Key: FLINK-2967
> URL: https://issues.apache.org/jira/browse/FLINK-2967
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.9, 0.10.0, 1.0.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I'm talking to a user which is facing the following issue:
> Some of the TaskManagers select the wrong IP address out of the available 
> network interfaces.
> The first address we try to connect to is the one returned by 
> {{InetAddress.getLocalHost()}}. This address is the right IP address to use, 
> but the JobManager is not able to respond within the timeout (50ms) to that 
> connection request.
> So the TM tries the next address, which is not publicly reachable. However, 
> the TM can connect to the JM from there. Netty will later fail to connect to 
> the TM from the other TMs.
> There are two solutions for this issue:
> - Allow users to configure a higher timeout for the first address detection 
> strategy. In most cases, the address returned by 
> {{InetAddress.getLocalHost()}} is correct. By setting a high timeout, users 
> with slow networks / overloaded JMs can make sure the TM picks this address
> - add an Akka message which we send from the TM to the JM, and the JM tries 
> to connect to the TM. If that succeeds, we know that the TM is reachable from 
> the outside.
> The problem is that we have to start a separate actor system on the 
> TaskManager first. We have to do this because might use a wrong ip address 
> for the TM (so we might end up starting actor systems until we found an 
> externally reachable ip)
> I'm first going to implement the first approach. If that solution works well 
> for my user, I'll contribute this to 0.10 / 1.0.
> If not, I'll implement the second approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2967] Enhance TaskManager network detec...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1361#issuecomment-157378877
  
Will merge and address the comments...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3029) JobGraph is not connected correctly in webfrontend for streaming job

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008699#comment-15008699
 ] 

Stephan Ewen commented on FLINK-3029:
-

Duplicate of https://issues.apache.org/jira/browse/FLINK-2942 ?

> JobGraph is not connected correctly in webfrontend for streaming job
> 
>
> Key: FLINK-3029
> URL: https://issues.apache.org/jira/browse/FLINK-3029
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Minor
>
> I have a streaming job consisting of a Source with parallelism 4  and I have 
> 3 different flatmaps with parallelism 24 each. 
> In the web interface the source is only shown to be connected to the first 
> flatmap, and the remaining 2 flatmaps appear as though if they were sources 
> with no input stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3011, 3019, 3028] Cancel jobs in RESTAR...

[gyfora]

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1369#issuecomment-157383381
  
I verified the intended behaviour on a cluster application, it works for 
cancelling from both the command line and also from the web interface.

+1 from my side, this is a critical fix for production environments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157383461
  
I can confirm that `chill-avro` was never used. I accidentally committed 
the dependency while trying out different approaches for handling Avro POJOs 
with Flink.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008720#comment-15008720
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157382231
  
Why to we need chill for collection data sets?

Also, are you sure chill-avro was never used? No magic reflection loading 
happening anywhere?


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008723#comment-15008723
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157383461
  
I can confirm that `chill-avro` was never used. I accidentally committed 
the dependency while trying out different approaches for handling Avro POJOs 
with Flink.


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3021] Fix class loading issue for strea...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1368#issuecomment-157384055
  
+1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3018) Job state discarded from web interface for restarting jobs

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008728#comment-15008728
 ] 

Stephan Ewen commented on FLINK-3018:
-

There are multiple execution attempts in cases of failures, they are all 
remembered, but the UI shows only the latest one.

For batch jobs, that makes sense. For streaming jobs, I am unsure. To get this 
right, we would need to include the metrics in the checkpointing, which is a 
major change. The metrics are currently communicates asynchronously, as part of 
heartbeat-like messages, they interfere with nothing.

I disagree that this is critical, though. Accumulated numbers mean little in 
many continuous streaming, settings, where metrics like throughput and latency 
mean much more, and look at the recent stats anyways.

> Job state discarded from web interface for restarting jobs
> --
>
> Key: FLINK-3018
> URL: https://issues.apache.org/jira/browse/FLINK-3018
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Gyula Fora
>Priority: Critical
>
> When a streaming job goes into the restarting status and recovers, the web ui 
> information is totally lost (number of records/bytes processed etc).
> This is very misleading and should not happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2879] [docs] Fixed broken links on the ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Change the parameter 'numSample' in DataSetUti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1362


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2914] Add missing break Statement in ZK...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1359


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3029) JobGraph is not connected correctly in webfrontend for streaming job

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3029:
-

 Summary: JobGraph is not connected correctly in webfrontend for 
streaming job
 Key: FLINK-3029
 URL: https://issues.apache.org/jira/browse/FLINK-3029
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 0.10.0, 1.0.0
Reporter: Gyula Fora
Priority: Minor


I have a streaming job consisting of a Source with parallelism 4  and I have 3 
different flatmaps with parallelism 24 each. 

In the web interface the source is only shown to be connected to the first 
flatmap, and the remaining 2 flatmaps appear as though if they were sources 
with no input stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008709#comment-15008709
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157381073
  
1. No, it seemed that `chill-avro` was not used.
2. Chill is required when Java programs are submitted to a cluster that 
include collection data sets. The cluster will have chill in the classpath and 
therefore use differently initialized Kryo serializers to read the collection 
data than the client without chill. I know this is a bit hacky, but the best 
solution I came up with.


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2901] Remove Record API dependencies fr...

[zentol]

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1306#issuecomment-157382048
  
@fhueske I've addressed most of your concerns.

Things that still need work / clarification:
* PreviewPlanDumpTest was previously executed with 2 different sets of 
arguments, now only with 1. Should this be changed back to the previous 
behaviour? The arguments affect paths for sources/sink, parallelism and the 
number of Iterations
* test.classloading.jar.KMeansForTest appears to be a good replacement for 
the IterativeKMeansITCase, what's your take on that?
* The removed delta ilteration PageRank program looks very similar to the 
ConnectedComponents implementation under flink-examples. I don't think this 
needs a separate port.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3011, 3019, 3028] Cancel jobs in RESTAR...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1369

[FLINK-3011, 3019, 3028] Cancel jobs in RESTARTING state

This addresses issues with cancelling jobs, which are in the `RESTARTING` 
state. A job enters this state  after a failure as soon as all job vertices are 
in their final state. It then stays in this state until it is redeployed (e.g. 
default 100s currently). In this state, the job cannot be cancelled. If the 
failure is permanent (for example missing slots), the job can never be 
cancelled.

This PR includes changes to the ExecutionGraph and to the clients:

**ExecutionGraph** (FLINK-3011)
- Remove the state transition from `FAILED` to `RESTARTING` in `restart()`. 
This was breaking the semantics of `FAILED` being a terminal state. It was only 
relevant for a test as far as I can tell.
- When cancelling during restarts, two job states are relevant:
  - `RESTARTING`: try to set the state directly to `CANCELED` as all 
vertices have been already failed when the job enters the `RESTARTING` state. 
If the state transition to `CANCELED` succeeds, the restart will be ignored 
with a log message.
  - `FAILING`: try to set the state to `CANCELLING` and wait for the 
failing of the vertices to finish. This will finish the cancellation as usual 
in `jobVertexInFinalState()`. 

When reviewing the `cancel()`, `jobVertexInFinalState()`, and `restart()` 
methods are relevant.

**CLIFrontend** (FLINK-3019)
- List restarting jobs with scheduled jobs

```
$ bin/flink list
No running jobs.
 Scheduled/Restarting Jobs ---
17.11.2015 15:14:01 : 4b3fa06c88e5a2a4963241e7afca7b7d : Streaming 
WordCount (RESTARTING)
--
```

**WebFrontend** (FLINK-3028)
- Show the cancel button if the job is restarting. It was only displayed 
for running or created jobs before.

---

I want to merge this for 0.10.1 and 1.0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 3011-restart

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1369.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1369


commit 0c5a3306808bec5b9a833703adbcd9f45bbe6de5
Author: Ufuk Celebi 
Date:   2015-11-16T15:18:20Z

[FLINK-3011] [runtime] Disallow ExecutionGraph state transition from FAILED 
to RESTARTING

Removes the possibility to go from FAILED state back to RESTARTING. This 
was only used in a test
case. It was a breaking the terminal state semantics of the FAILED state.

commit 19c602b2ce7686237d8611645a4662aa2b2a0cef
Author: Ufuk Celebi 
Date:   2015-11-17T10:40:54Z

[FLINK-3011] [runtime, tests] Translate ExecutionGraphRestartTest to Java

commit e13dd1bac7029af6ae4157af226131a10f5d02d0
Author: Ufuk Celebi 
Date:   2015-11-17T10:56:42Z

[FLINK-3011] [runtime] Fix cancel during restart

commit 657e34f31fe9c6325900f42c36257b5c5d2019be
Author: Ufuk Celebi 
Date:   2015-11-17T13:11:44Z

[FLINK-3019] [client] List restarting jobs with scheduled jobs

commit 8b2850610aff1197d204bdb7d790df8fb6b5df4c
Author: Ufuk Celebi 
Date:   2015-11-17T13:51:15Z

[FLINK-3028] [runtime-web] Show cancel button for restarting jobs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3011) Cannot cancel failing/restarting streaming job from the command line

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008719#comment-15008719
 ] 

ASF GitHub Bot commented on FLINK-3011:
---

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1369

[FLINK-3011, 3019, 3028] Cancel jobs in RESTARTING state

This addresses issues with cancelling jobs, which are in the `RESTARTING` 
state. A job enters this state  after a failure as soon as all job vertices are 
in their final state. It then stays in this state until it is redeployed (e.g. 
default 100s currently). In this state, the job cannot be cancelled. If the 
failure is permanent (for example missing slots), the job can never be 
cancelled.

This PR includes changes to the ExecutionGraph and to the clients:

**ExecutionGraph** (FLINK-3011)
- Remove the state transition from `FAILED` to `RESTARTING` in `restart()`. 
This was breaking the semantics of `FAILED` being a terminal state. It was only 
relevant for a test as far as I can tell.
- When cancelling during restarts, two job states are relevant:
  - `RESTARTING`: try to set the state directly to `CANCELED` as all 
vertices have been already failed when the job enters the `RESTARTING` state. 
If the state transition to `CANCELED` succeeds, the restart will be ignored 
with a log message.
  - `FAILING`: try to set the state to `CANCELLING` and wait for the 
failing of the vertices to finish. This will finish the cancellation as usual 
in `jobVertexInFinalState()`. 

When reviewing the `cancel()`, `jobVertexInFinalState()`, and `restart()` 
methods are relevant.

**CLIFrontend** (FLINK-3019)
- List restarting jobs with scheduled jobs

```
$ bin/flink list
No running jobs.
 Scheduled/Restarting Jobs ---
17.11.2015 15:14:01 : 4b3fa06c88e5a2a4963241e7afca7b7d : Streaming 
WordCount (RESTARTING)
--
```

**WebFrontend** (FLINK-3028)
- Show the cancel button if the job is restarting. It was only displayed 
for running or created jobs before.

---

I want to merge this for 0.10.1 and 1.0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 3011-restart

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1369.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1369


commit 0c5a3306808bec5b9a833703adbcd9f45bbe6de5
Author: Ufuk Celebi 
Date:   2015-11-16T15:18:20Z

[FLINK-3011] [runtime] Disallow ExecutionGraph state transition from FAILED 
to RESTARTING

Removes the possibility to go from FAILED state back to RESTARTING. This 
was only used in a test
case. It was a breaking the terminal state semantics of the FAILED state.

commit 19c602b2ce7686237d8611645a4662aa2b2a0cef
Author: Ufuk Celebi 
Date:   2015-11-17T10:40:54Z

[FLINK-3011] [runtime, tests] Translate ExecutionGraphRestartTest to Java

commit e13dd1bac7029af6ae4157af226131a10f5d02d0
Author: Ufuk Celebi 
Date:   2015-11-17T10:56:42Z

[FLINK-3011] [runtime] Fix cancel during restart

commit 657e34f31fe9c6325900f42c36257b5c5d2019be
Author: Ufuk Celebi 
Date:   2015-11-17T13:11:44Z

[FLINK-3019] [client] List restarting jobs with scheduled jobs

commit 8b2850610aff1197d204bdb7d790df8fb6b5df4c
Author: Ufuk Celebi 
Date:   2015-11-17T13:51:15Z

[FLINK-3028] [runtime-web] Show cancel button for restarting jobs




> Cannot cancel failing/restarting streaming job from the command line
> 
>
> Key: FLINK-3011
> URL: https://issues.apache.org/jira/browse/FLINK-3011
> Project: Flink
>  Issue Type: Bug
>  Components: Command-line client
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Assignee: Ufuk Celebi
>Priority: Critical
>
> I cannot seem to be able to cancel a failing/restarting job from the command 
> line client. The job cannot be rescheduled so it keeps failing:
> The exception I get:
> 13:58:11,240 INFO  org.apache.flink.runtime.jobmanager.JobManager 
>- Status of job 0c895d22c632de5dfe16c42a9ba818d5 (player-id) changed to 
> RESTARTING.
> 13:58:25,234 INFO  org.apache.flink.runtime.jobmanager.JobManager 
>- Trying to cancel job with ID 0c895d22c632de5dfe16c42a9ba818d5.
> 13:58:25,561 WARN  akka.remote.ReliableDeliverySupervisor 
>- Association with remote system [akka.tcp://flink@127.0.0.1:42012] has 
> 

[jira] [Commented] (FLINK-2901) Several flink-test ITCases depend on Record API features

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008718#comment-15008718
 ] 

ASF GitHub Bot commented on FLINK-2901:
---

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1306#issuecomment-157382048
  
@fhueske I've addressed most of your concerns.

Things that still need work / clarification:
* PreviewPlanDumpTest was previously executed with 2 different sets of 
arguments, now only with 1. Should this be changed back to the previous 
behaviour? The arguments affect paths for sources/sink, parallelism and the 
number of Iterations
* test.classloading.jar.KMeansForTest appears to be a good replacement for 
the IterativeKMeansITCase, what's your take on that?
* The removed delta ilteration PageRank program looks very similar to the 
ConnectedComponents implementation under flink-examples. I don't think this 
needs a separate port.


> Several flink-test ITCases depend on Record API features
> 
>
> Key: FLINK-2901
> URL: https://issues.apache.org/jira/browse/FLINK-2901
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Fabian Hueske
>Assignee: Chesnay Schepler
>
> There are several ITCases and utility classes in {{flink-tests}} that depend 
> on the Record API including:
> - ITCases for Record API operators in 
> {{flink-tests/src/test/java/org/apache/flink/test/operators}}
> - ITCases for Record API programs in 
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
> - Record API programs in 
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
> - Several ITCases for iterations in 
> {{flink-tests/src/test/java/org/apache/flink/test/iterative}}
> - Tests for job canceling in 
> {{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
> - Test for failing jobs in 
> {{flink-tests/src/test/java/org/apache/flink/test/failingPrograms/TaskFailureITCase}}
> - Optimizer tests in 
> {{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
> - Accumulator test in 
> {{flink-tests/src/test/java/org/apache/flink/test/accumulators/AccumulatorIterativeITCase}}
> - Broadcast test in 
> {{flink-tests/src/test/java/org/apache/flink/test/broadcastvasr/BroadcastBranchingITCase}}
> - distributed cache test in 
> {{flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest}}
> and probably a few more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157381073
  
1. No, it seemed that `chill-avro` was not used.
2. Chill is required when Java programs are submitted to a cluster that 
include collection data sets. The cluster will have chill in the classpath and 
therefore use differently initialized Kryo serializers to read the collection 
data than the client without chill. I know this is a bit hacky, but the best 
solution I came up with.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157382231
  
Why to we need chill for collection data sets?

Also, are you sure chill-avro was never used? No magic reflection loading 
happening anywhere?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157385199
  
I thought generic types within collections data sets are serialized with 
Kryo when they are shipped to the JM, no? Or are they going through Java 
serialization?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008730#comment-15008730
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157385199
  
I thought generic types within collections data sets are serialized with 
Kryo when they are shipped to the JM, no? Or are they going through Java 
serialization?


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3002) Add an EitherType to the Java API

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008740#comment-15008740
 ] 

Stephan Ewen commented on FLINK-3002:
-

You went with the abstract class version, that is good, IMHO.

Few issues:
  - I would drop the flags {{isLeft}} and {{isRight}}. The subclasses can track 
that, or you can directly check via {{getClass()}}. Saves space.
  - In the type info, the arity is should one, as the type has but one element: 
left or right.

> Add an EitherType to the Java API
> -
>
> Key: FLINK-3002
> URL: https://issues.apache.org/jira/browse/FLINK-3002
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Either types are recurring patterns and should be serialized efficiently, so 
> it makes sense to add them to the core Java API.
> Since Java does not have such a type as of Java 8, we would need to add our 
> own version.
> The Scala API handles the Scala Either Type already efficiently. I would not 
> use the Scala Either Type in the Java API, since we are trying to get the 
> {{flink-java}} project "Scala free" for people that don't use Scala and o not 
> want to worry about Scala version matches and mismatches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008746#comment-15008746
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157385892
  
Generic types go through Kryo, but kryo works without Chill. Chill only 
adds Scala Serializers, or does it also add Java and Util Serializers?


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2972) Remove Twitter Chill dependency from flink-java module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008689#comment-15008689
 ] 

ASF GitHub Bot commented on FLINK-2972:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157378685
  
Two questions:

1. The `chill-avro` dependency is added nowhere. Is it not necessary any 
more?
2. It is necessary to add the chill dependency also to `flink-clients`? Is 
that not covered by `flink-scala`. Asked differently, is there a need for Chill 
to be in the classpath of user code programs that do not use the Scala API?


> Remove Twitter Chill dependency from flink-java module
> --
>
> Key: FLINK-2972
> URL: https://issues.apache.org/jira/browse/FLINK-2972
> Project: Flink
>  Issue Type: Sub-task
>  Components: Java API
>Affects Versions: 1.0.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.0.0
>
>
> The {{flink-java}} module has a transitive dependency on Scala due to 
> Twitter's Chill library. Chill is used within Flink's KryoSerializer to 
> obtain a Kryo instance that has be initialized for Scala classes. 
> In order to decouple {{flink-java}} from Scala, its dependency on Chill needs 
> to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2972] [JavaAPI] Remove Chill dependency...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1331#issuecomment-157378685
  
Two questions:

1. The `chill-avro` dependency is added nowhere. Is it not necessary any 
more?
2. It is necessary to add the chill dependency also to `flink-clients`? Is 
that not covered by `flink-scala`. Asked differently, is there a need for Chill 
to be in the classpath of user code programs that do not use the Scala API?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3029) JobGraph is not connected correctly in webfrontend for streaming job

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008713#comment-15008713
 ] 

Gyula Fora commented on FLINK-3029:
---

Possibly, but this seems to be pretty deterministically not working.

> JobGraph is not connected correctly in webfrontend for streaming job
> 
>
> Key: FLINK-3029
> URL: https://issues.apache.org/jira/browse/FLINK-3029
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
>Priority: Minor
>
> I have a streaming job consisting of a Source with parallelism 4  and I have 
> 3 different flatmaps with parallelism 24 each. 
> In the web interface the source is only shown to be connected to the first 
> flatmap, and the remaining 2 flatmaps appear as though if they were sources 
> with no input stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3013: Incorrect package declaration in G...

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1356#issuecomment-157385739
  
Thanks a lot for the PR @smarthi! +1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: minor javadoc fix in JobManager

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1358


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---