[GitHub] flink pull request: [FLINK-2411] [gelly] Add Summarization Algorit...

[s1ck]

GitHub user s1ck opened a pull request:

https://github.com/apache/flink/pull/1269

[FLINK-2411] [gelly] Add Summarization Algorithm

* implemented algorithm
* implemented integration tests
* updated gelly guide

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/s1ck/flink FLINK-2411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1269


commit 2e1f4a5e6df91fe0b8754165dffd76169c40e496
Author: Martin Junghanns 
Date:   2015-10-20T07:28:40Z

[FLINK-2411] [gelly] Add Summarization Algorithm

* implemented algorithm
* implemented integration tests
* updated gelly guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2805] [blobmanager] Write JARs to file ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1227


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2804) Support blocking job submission with Job Manager recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964749#comment-14964749
 ] 

ASF GitHub Bot commented on FLINK-2804:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1249


> Support blocking job submission with Job Manager recovery
> -
>
> Key: FLINK-2804
> URL: https://issues.apache.org/jira/browse/FLINK-2804
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Till Rohrmann
>Priority: Minor
>
> Submitting a job in a blocking fashion with JobManager recovery and a failing 
> JobManager fails on the client side (the one submitting the job). The job 
> still continues to be recovered.
> I propose to add simple support to re-retrieve the leading job manager and 
> update the client actor with it and then wait for the result as before.
> As of the current standing in PR #1153 
> (https://github.com/apache/flink/pull/1153) the job manager assumes that the 
> same actor is running and just keeps on sending execution state updates etc. 
> (if the listening behaviour is not detached).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2354) Recover running jobs on JobManager failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964748#comment-14964748
 ] 

ASF GitHub Bot commented on FLINK-2354:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1153


> Recover running jobs on JobManager failure
> --
>
> Key: FLINK-2354
> URL: https://issues.apache.org/jira/browse/FLINK-2354
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 0.10
>
>
> tl;dr Persist JobGraphs in state backend and coordinate reference to state 
> handle via ZooKeeper.
> Problem: When running multiple JobManagers in high availability mode, the 
> leading job manager looses all running jobs when it fails. After a new 
> leading job manager is elected, it is not possible to recover any previously 
> running jobs.
> Solution: The leading job manager, which receives the job graph writes 1) the 
> job graph to a state backend, and 2) a reference to the respective state 
> handle to ZooKeeper. In general, job graphs can become large (multiple MBs, 
> because they include closures etc.). ZooKeeper is not designed for data of 
> this size. The level of indirection via the reference to the state backend 
> keeps the data in ZooKeeper small.
> Proposed ZooKeeper layout:
> /flink (default)
>   +- currentJobs
>+- job id i
> +- state handle reference of job graph i
> The 'currentJobs' node needs to be persistent to allow recovery of jobs 
> between job managers. The currentJobs node needs to satisfy the following 
> invariant: There is a reference to a job graph with id i IFF the respective 
> job graph needs to be recovered by a newly elected job manager leader.
> With this in place, jobs will be recovered from their initial state (as if 
> resubmitted). The next step is to backup the runtime state handles of 
> checkpoints in a similar manner.
> ---
> This work will be based on [~trohrm...@apache.org]'s implementation of 
> FLINK-2291. The leader election service notifies the job manager about 
> granted/revoked leadership. This notification happens via Akka and thus 
> serially *per* job manager, but results in eventually consistent state 
> between job managers. For some snapshots of time it is possible to have a new 
> leader granted leadership, before the old one has been revoked its leadership.
> [~trohrm...@apache.org], can you confirm that leadership does not guarantee 
> mutually exclusive access to the shared 'currentJobs' state?
> For example, the following can happen:
> - JM 1 is leader, JM 2 is standby
> - JOB i is running (and hence /flink/currentJobs/i exists)
> - ZK notifies leader election service (LES) of JM 1 and JM 2
> - LES 2 immediately notifies JM 2 about granted leadership, but LES 1 
> notification revoking leadership takes longer
> - JOB i finishes (TMs don't notice leadership change yet) and JM 1 receives 
> final JobStatusChange
> - JM 2 resubmits the job /flink/currentJobs/i
> - JM 1 removes /flink/currentJobs/i, because it is now finished
> => inconsistent state (wrt the specified invariant above)
> If it is indeed a problem, we can circumvent this with a Curator recipe for 
> [shared locks|http://curator.apache.org/curator-recipes/shared-lock.html] to 
> coordinate the access to currentJobs. The lock needs to be acquired on 
> leadership.
> ---
> Minimum required tests:
> - Unit tests for job graph serialization and writing to state backend and 
> ZooKeeper with expected nodes
> - Unit tests for job submission to job manager in leader/non-leader state
> - Unit tests for leadership granting/revoking and job submission/restarting 
> interleavings
> - Process failure integration tests with single and multiple running jobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2793) Redirect to leading JobManager web fronted in non-standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964750#comment-14964750
 ] 

ASF GitHub Bot commented on FLINK-2793:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1202


> Redirect to leading JobManager web fronted in non-standalone mode
> -
>
> Key: FLINK-2793
> URL: https://issues.apache.org/jira/browse/FLINK-2793
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>
> In case of a non-standalone recovery mode, the job manager frontend of 
> non-leading job managers prints the job manager information of its associated 
> job manager. Because the job manager is not leading, nothing shows up.
> The web frontend cannot directly communicate with the leading job manager, 
> because many job manager structures like the execution graph are not 
> serializable.
> A work around is to redirect to the web frontend of the leading job manager. 
> This makes sure that all interesting information is presented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2793] [dashboard] Redirect to leader in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1202


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2805) Make user jars available for all job managers to recover

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964747#comment-14964747
 ] 

ASF GitHub Bot commented on FLINK-2805:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1227


> Make user jars available for all job managers to recover
> 
>
> Key: FLINK-2805
> URL: https://issues.apache.org/jira/browse/FLINK-2805
> Project: Flink
>  Issue Type: Bug
>  Components: BlobManager, JobManager
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>
> This is a bug in https://github.com/apache/flink/pull/1153.
> In case of multiple job managers, the user jars need to be accessible by all 
> job managers (including those who arrive later).
> Since #1153 requires the file state backend to be configured, the simplest 
> solution is to make the blob server aware of the configured recovery mode and 
> put/get/delete the user jars from the file state backend as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2354] Add job graph and checkpoint reco...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1153


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2869) Apply JMH on IOManagerPerformanceBenchmark class.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964808#comment-14964808
 ] 

ASF GitHub Bot commented on FLINK-2869:
---

GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/1270

[FLINK-2869] [tests] Apply JMH on IOManagerPerformanceBenchmark class.

JMH is a Java harness for building, running, and analysing 
nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks 
method in order to get much more accurate results.Modify the 
`IOManagerPerformanceBenchmark` class and move it to `flink-benchmark` module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink IOManagerPerformanceBenchmark

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1270


commit dfcda0bc80fc9aaf01bfb73fb63b1e9351b8986a
Author: gallenvara 
Date:   2015-10-20T01:49:34Z

[FLINK-2869] [tests] Apply JMH on IOManagerPerformanceBenchmark class.




> Apply JMH on IOManagerPerformanceBenchmark class.
> -
>
> Key: FLINK-2869
> URL: https://issues.apache.org/jira/browse/FLINK-2869
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: GaoLun
>Assignee: GaoLun
>Priority: Minor
>  Labels: easyfix
>
> JMH is a Java harness for building, running, and analysing 
> nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks 
> method in order to get much more accurate results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2869] [tests] Apply JMH on IOManagerPer...

[gallenvara]

GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/1270

[FLINK-2869] [tests] Apply JMH on IOManagerPerformanceBenchmark class.

JMH is a Java harness for building, running, and analysing 
nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks 
method in order to get much more accurate results.Modify the 
`IOManagerPerformanceBenchmark` class and move it to `flink-benchmark` module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink IOManagerPerformanceBenchmark

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1270


commit dfcda0bc80fc9aaf01bfb73fb63b1e9351b8986a
Author: gallenvara 
Date:   2015-10-20T01:49:34Z

[FLINK-2869] [tests] Apply JMH on IOManagerPerformanceBenchmark class.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned FLINK-2873:
-

Assignee: Maximilian Michels

> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-2873:
--
Component/s: JobManager

> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2030][FLINK-2274][core][utils]Histogram...

[sachingoel0101]

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/861#issuecomment-149491074
  
@tillrohrmann , any updates regarding this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2030) Implement discrete and continuous histograms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964851#comment-14964851
 ] 

ASF GitHub Bot commented on FLINK-2030:
---

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/861#issuecomment-149491074
  
@tillrohrmann , any updates regarding this?


> Implement discrete and continuous histograms
> 
>
> Key: FLINK-2030
> URL: https://issues.apache.org/jira/browse/FLINK-2030
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Sachin Goel
>Assignee: Sachin Goel
>Priority: Minor
>
> For the implementation of the decision tree in 
> https://issues.apache.org/jira/browse/FLINK-1727, we need to implement an 
> histogram with online updates, merging and equalization features. A reference 
> implementation is provided in [1]
> [1].http://www.jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2030) Implement discrete and continuous histograms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964866#comment-14964866
 ] 

ASF GitHub Bot commented on FLINK-2030:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/861#issuecomment-149494676
  
Not yet. Just finished my current task. Will take a look now.

On Tue, Oct 20, 2015 at 11:25 AM, Sachin Goel 
wrote:

> @tillrohrmann  , any updates regarding
> this?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



> Implement discrete and continuous histograms
> 
>
> Key: FLINK-2030
> URL: https://issues.apache.org/jira/browse/FLINK-2030
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Sachin Goel
>Assignee: Sachin Goel
>Priority: Minor
>
> For the implementation of the decision tree in 
> https://issues.apache.org/jira/browse/FLINK-1727, we need to implement an 
> histogram with online updates, merging and equalization features. A reference 
> implementation is provided in [1]
> [1].http://www.jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2030][FLINK-2274][core][utils]Histogram...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/861#issuecomment-149494676
  
Not yet. Just finished my current task. Will take a look now.

On Tue, Oct 20, 2015 at 11:25 AM, Sachin Goel 
wrote:

> @tillrohrmann  , any updates regarding
> this?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2033) Add overloaded methods with explicit TypeInformation parameters to Gelly

[PJ Van Aeken (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964914#comment-14964914
 ] 

PJ Van Aeken commented on FLINK-2033:
-

Totally forgot about this one, sorry. Yes it's ok now. Gelly has the explicit 
declarations.

> Add overloaded methods with explicit TypeInformation parameters to Gelly
> 
>
> Key: FLINK-2033
> URL: https://issues.apache.org/jira/browse/FLINK-2033
> Project: Flink
>  Issue Type: Task
>  Components: Gelly
>Affects Versions: 0.9
>Reporter: PJ Van Aeken
>Assignee: PJ Van Aeken
> Fix For: 0.10
>
>
> For the implementation of the Scala API for Gelly (FLINK-1962), we need to 
> pass explicit TypeInformation since the Java TypeExtractor does not work for 
> all Scala Types (see FLINK-2023).
> To do this, the java Gelly API needs to be expanded with methods that allow 
> for explicit passing of TypeInformation.
> An example with mapVertices:
> {code}
>  public  Graph mapVertices(final MapFunction, 
> NV> mapper) {
> TypeInformation keyType = ((TupleTypeInfo) 
> vertices.getType()).getTypeAt(0);
> String callLocation = Utils.getCallLocationName();
> TypeInformation valueType = 
> TypeExtractor.getMapReturnTypes(mapper, vertices.getType(), callLocation, 
> false);
> TypeInformation> returnType = 
> (TypeInformation>) new TupleTypeInfo(
> Vertex.class, keyType, valueType);
> return mapVertices(mapper,returnType);
> }
> public  Graph mapVertices(final MapFunction, 
> NV> mapper, TypeInformation> returnType) {
> DataSet> mappedVertices = vertices.map(
> new MapFunction, Vertex>() {
> public Vertex map(Vertex value) throws 
> Exception {
> return new Vertex(value.f0, mapper.map(value));
> }
> }).returns(returnType);
> return new Graph(mappedVertices, this.edges, this.context);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2033) Add overloaded methods with explicit TypeInformation parameters to Gelly

[Vasia Kalavri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vasia Kalavri resolved FLINK-2033.
--
   Resolution: Fixed
Fix Version/s: 0.10

> Add overloaded methods with explicit TypeInformation parameters to Gelly
> 
>
> Key: FLINK-2033
> URL: https://issues.apache.org/jira/browse/FLINK-2033
> Project: Flink
>  Issue Type: Task
>  Components: Gelly
>Affects Versions: 0.9
>Reporter: PJ Van Aeken
>Assignee: PJ Van Aeken
> Fix For: 0.10
>
>
> For the implementation of the Scala API for Gelly (FLINK-1962), we need to 
> pass explicit TypeInformation since the Java TypeExtractor does not work for 
> all Scala Types (see FLINK-2023).
> To do this, the java Gelly API needs to be expanded with methods that allow 
> for explicit passing of TypeInformation.
> An example with mapVertices:
> {code}
>  public  Graph mapVertices(final MapFunction, 
> NV> mapper) {
> TypeInformation keyType = ((TupleTypeInfo) 
> vertices.getType()).getTypeAt(0);
> String callLocation = Utils.getCallLocationName();
> TypeInformation valueType = 
> TypeExtractor.getMapReturnTypes(mapper, vertices.getType(), callLocation, 
> false);
> TypeInformation> returnType = 
> (TypeInformation>) new TupleTypeInfo(
> Vertex.class, keyType, valueType);
> return mapVertices(mapper,returnType);
> }
> public  Graph mapVertices(final MapFunction, 
> NV> mapper, TypeInformation> returnType) {
> DataSet> mappedVertices = vertices.map(
> new MapFunction, Vertex>() {
> public Vertex map(Vertex value) throws 
> Exception {
> return new Vertex(value.f0, mapper.map(value));
> }
> }).returns(returnType);
> return new Graph(mappedVertices, this.edges, this.context);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2254) Add Bipartite Graph Support for Gelly

[Vasia Kalavri (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964862#comment-14964862
 ] 

Vasia Kalavri commented on FLINK-2254:
--

Hi [~ssaumitra],
thank you for your interest in working on this issue!

As far as I know, nobody has started working on this. Since this will be a 
quite big addition to Gelly, I have tagged the issue with the 
{{requires-design-doc}} label. This means that, before you start coding, we 
should first discuss the approach with the community and make a document that 
describes the design of the proposed solution. We hope that this will save 
contributors' time and make it easier for PRs to be accepted. Please, take a 
look at our recently updated [contribution 
guide|https://flink.apache.org/contribute-code.html] for more details.

To start, try to write down your initial ideas about how you would like to 
approach this and share them in the dev mailing list or here. Once we have some 
input from the community and a quite clear idea on how to proceed, we can 
polish the design document and start the implementation.

Let us know if you any have questions! Thank you!

> Add Bipartite Graph Support for Gelly
> -
>
> Key: FLINK-2254
> URL: https://issues.apache.org/jira/browse/FLINK-2254
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 0.10
>Reporter: Andra Lungu
>  Labels: requires-design-doc
>
> A bipartite graph is a graph for which the set of vertices can be divided 
> into two disjoint sets such that each edge having a source vertex in the 
> first set, will have a target vertex in the second set. We would like to 
> support efficient operations for this type of graphs along with a set of 
> metrics(http://jponnela.com/web_documents/twomode.pdf). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-2862) FlinkTopologyBuilder should use proper generic types

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned FLINK-2862:
--

Assignee: Matthias J. Sax

> FlinkTopologyBuilder should use proper generic types
> 
>
> Key: FLINK-2862
> URL: https://issues.apache.org/jira/browse/FLINK-2862
> Project: Flink
>  Issue Type: Bug
>  Components: Storm Compatibility
>Affects Versions: 0.10
>Reporter: Stephan Ewen
>Assignee: Matthias J. Sax
>
> The Flink Topology Builder uses only raw types, resulting in bugs due to 
> missing compile time checks of types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-149580271
  
Is there any news here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-2876) Minutiae

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-2876:
-

 Summary: Minutiae
 Key: FLINK-2876
 URL: https://issues.apache.org/jira/browse/FLINK-2876
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 0.10
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Trivial


A collection of small documentation and grammar updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2513) Extend state handle provider interface to list all state handles

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-2513.
--
Resolution: Invalid

> Extend state handle provider interface to list all state handles
> 
>
> Key: FLINK-2513
> URL: https://issues.apache.org/jira/browse/FLINK-2513
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Reporter: Ufuk Celebi
>
> This is a follow up issue to FLINK-2354.
> In FLINK-2354 we use ZooKeeper to persist state handles. In certain failure 
> scenarios, there can be lingering state handles, which have not been written 
> to ZooKeeper, but which were already created.
> These can be cleaned up on startup, if the state handle provider 
> implementation kept track of their state handles. With the current 
> implementations it would be possible, e.g. list the directory (file system) 
> of no state is persistent (job manager).
> It would be fair enough have this as an optional operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2874) Certain Avro generated getters/setters not recognized

[Ulf Karlsson (JIRA)]

Ulf Karlsson created FLINK-2874:
---

 Summary: Certain Avro generated getters/setters not recognized
 Key: FLINK-2874
 URL: https://issues.apache.org/jira/browse/FLINK-2874
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 0.9.0, 0.9.1, 0.10
Reporter: Ulf Karlsson
Priority: Minor


For Avro schemas where value null is not allowed, the field is unboxed e.g. int 
but the getter/setter methods provide the boxed Integer as interface:

```
{
 "fields": [
  {
   "type": "double", 
   "name": "time"
  }, 
}
```

This results in Java

```
  private double time;

  public java.lang.Double getTime() {
return time;
  }

  public void setTime(java.lang.Double value) {
this.time = value;
  }
```

There is also a problem when there is an underscore in the Avro schema, e.g.:

```
  {
   "default": null, 
   "type": [
"null", 
"long"
   ], 
   "name": "conn_id"
  }, 
```

This results in Java:

```
private java.lang.Long conn_id;

  public java.lang.Long getConnId() {
return conn_id;
  }

  public void setConnId(java.lang.Long value) {
this.conn_id = value;
  }
```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-2874) Certain Avro generated getters/setters not recognized

[Ulf Karlsson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ulf Karlsson updated FLINK-2874:

Description: 
For Avro schemas where value null is not allowed, the field is unboxed e.g. int 
but the getter/setter methods provide the boxed Integer as interface:

```
{
 "fields": [
  {
   "type": "double", 
   "name": "time"
  }, 
}
```

This results in Java

{code}
  private double time;

  public java.lang.Double getTime() {
return time;
  }

  public void setTime(java.lang.Double value) {
this.time = value;
  }
{code}

There is also a problem when there is an underscore in the Avro schema, e.g.:

{code}
  {
   "default": null, 
   "type": [
"null", 
"long"
   ], 
   "name": "conn_id"
  }, 
{code}

This results in Java:

{code}
private java.lang.Long conn_id;

  public java.lang.Long getConnId() {
return conn_id;
  }

  public void setConnId(java.lang.Long value) {
this.conn_id = value;
  }
{code}

  was:
For Avro schemas where value null is not allowed, the field is unboxed e.g. int 
but the getter/setter methods provide the boxed Integer as interface:

```
{
 "fields": [
  {
   "type": "double", 
   "name": "time"
  }, 
}
```

This results in Java

```
  private double time;

  public java.lang.Double getTime() {
return time;
  }

  public void setTime(java.lang.Double value) {
this.time = value;
  }
```

There is also a problem when there is an underscore in the Avro schema, e.g.:

```
  {
   "default": null, 
   "type": [
"null", 
"long"
   ], 
   "name": "conn_id"
  }, 
```

This results in Java:

```
private java.lang.Long conn_id;

  public java.lang.Long getConnId() {
return conn_id;
  }

  public void setConnId(java.lang.Long value) {
this.conn_id = value;
  }
```


> Certain Avro generated getters/setters not recognized
> -
>
> Key: FLINK-2874
> URL: https://issues.apache.org/jira/browse/FLINK-2874
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.9.1, 0.9.0, 0.10
>Reporter: Ulf Karlsson
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> For Avro schemas where value null is not allowed, the field is unboxed e.g. 
> int but the getter/setter methods provide the boxed Integer as interface:
> ```
> {
>  "fields": [
>   {
>"type": "double", 
>"name": "time"
>   }, 
> }
> ```
> This results in Java
> {code}
>   private double time;
>   public java.lang.Double getTime() {
> return time;
>   }
>   public void setTime(java.lang.Double value) {
> this.time = value;
>   }
> {code}
> There is also a problem when there is an underscore in the Avro schema, e.g.:
> {code}
>   {
>"default": null, 
>"type": [
> "null", 
> "long"
>], 
>"name": "conn_id"
>   }, 
> {code}
> This results in Java:
> {code}
> private java.lang.Long conn_id;
>   public java.lang.Long getConnId() {
> return conn_id;
>   }
>   public void setConnId(java.lang.Long value) {
> this.conn_id = value;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2287) Implement JobManager high availability

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi resolved FLINK-2287.

Resolution: Fixed

Fixed in sub tasks

> Implement JobManager high availability
> --
>
> Key: FLINK-2287
> URL: https://issues.apache.org/jira/browse/FLINK-2287
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Reporter: Ufuk Celebi
> Fix For: 0.10
>
>
> The problem: The JobManager (JM) is a single point of failure. When it 
> crashes, TaskManagers (TM) fail all running jobs and try to reconnect to the 
> same JM. A failed JM looses all state and can not resume the running jobs; 
> even if it recovers and the TMs reconnect.
> Solution: implement JM fault tolerance/high availability by having multiple 
> JM instances running with one as leader and the other(s) in standby. The 
> exact coordination and state update protocol between JM, TM, and clients is 
> covered in sub-tasks/issues.
> Related Wiki: 
> https://cwiki.apache.org/confluence/display/FLINK/JobManager+High+Availability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2874) Certain Avro generated getters/setters not recognized

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965062#comment-14965062
 ] 

ASF GitHub Bot commented on FLINK-2874:
---

Github user ukarlsson commented on the pull request:

https://github.com/apache/flink/pull/1252#issuecomment-149559944
  
@fhueske @rmetzger 

Hello,

I have added the Jira issue:

https://issues.apache.org/jira/browse/FLINK-2874

I have also added unit tests.


> Certain Avro generated getters/setters not recognized
> -
>
> Key: FLINK-2874
> URL: https://issues.apache.org/jira/browse/FLINK-2874
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.9.1, 0.9.0, 0.10
>Reporter: Ulf Karlsson
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> For Avro schemas where value null is not allowed, the field is unboxed e.g. 
> int but the getter/setter methods provide the boxed Integer as interface:
> {code}
> {
>  "fields": [
>   {
>"type": "double", 
>"name": "time"
>   }, 
> }
> {code}
> This results in Java
> {code}
>   private double time;
>   public java.lang.Double getTime() {
> return time;
>   }
>   public void setTime(java.lang.Double value) {
> this.time = value;
>   }
> {code}
> There is also a problem when there is an underscore in the Avro schema, e.g.:
> {code}
>   {
>"default": null, 
>"type": [
> "null", 
> "long"
>], 
>"name": "conn_id"
>   }, 
> {code}
> This results in Java:
> {code}
> private java.lang.Long conn_id;
>   public java.lang.Long getConnId() {
> return conn_id;
>   }
>   public void setConnId(java.lang.Long value) {
> this.conn_id = value;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2874] Fix Avro getter/setter recognitio...

[ukarlsson]

Github user ukarlsson commented on the pull request:

https://github.com/apache/flink/pull/1252#issuecomment-149559944
  
@fhueske @rmetzger 

Hello,

I have added the Jira issue:

https://issues.apache.org/jira/browse/FLINK-2874

I have also added unit tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-2874) Certain Avro generated getters/setters not recognized

[Ulf Karlsson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ulf Karlsson updated FLINK-2874:

Description: 
For Avro schemas where value null is not allowed, the field is unboxed e.g. int 
but the getter/setter methods provide the boxed Integer as interface:

{code}
{
 "fields": [
  {
   "type": "double", 
   "name": "time"
  }, 
}
{code}

This results in Java

{code}
  private double time;

  public java.lang.Double getTime() {
return time;
  }

  public void setTime(java.lang.Double value) {
this.time = value;
  }
{code}

There is also a problem when there is an underscore in the Avro schema, e.g.:

{code}
  {
   "default": null, 
   "type": [
"null", 
"long"
   ], 
   "name": "conn_id"
  }, 
{code}

This results in Java:

{code}
private java.lang.Long conn_id;

  public java.lang.Long getConnId() {
return conn_id;
  }

  public void setConnId(java.lang.Long value) {
this.conn_id = value;
  }
{code}

  was:
For Avro schemas where value null is not allowed, the field is unboxed e.g. int 
but the getter/setter methods provide the boxed Integer as interface:

```
{
 "fields": [
  {
   "type": "double", 
   "name": "time"
  }, 
}
```

This results in Java

{code}
  private double time;

  public java.lang.Double getTime() {
return time;
  }

  public void setTime(java.lang.Double value) {
this.time = value;
  }
{code}

There is also a problem when there is an underscore in the Avro schema, e.g.:

{code}
  {
   "default": null, 
   "type": [
"null", 
"long"
   ], 
   "name": "conn_id"
  }, 
{code}

This results in Java:

{code}
private java.lang.Long conn_id;

  public java.lang.Long getConnId() {
return conn_id;
  }

  public void setConnId(java.lang.Long value) {
this.conn_id = value;
  }
{code}


> Certain Avro generated getters/setters not recognized
> -
>
> Key: FLINK-2874
> URL: https://issues.apache.org/jira/browse/FLINK-2874
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.9.1, 0.9.0, 0.10
>Reporter: Ulf Karlsson
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> For Avro schemas where value null is not allowed, the field is unboxed e.g. 
> int but the getter/setter methods provide the boxed Integer as interface:
> {code}
> {
>  "fields": [
>   {
>"type": "double", 
>"name": "time"
>   }, 
> }
> {code}
> This results in Java
> {code}
>   private double time;
>   public java.lang.Double getTime() {
> return time;
>   }
>   public void setTime(java.lang.Double value) {
> this.time = value;
>   }
> {code}
> There is also a problem when there is an underscore in the Avro schema, e.g.:
> {code}
>   {
>"default": null, 
>"type": [
> "null", 
> "long"
>], 
>"name": "conn_id"
>   }, 
> {code}
> This results in Java:
> {code}
> private java.lang.Long conn_id;
>   public java.lang.Long getConnId() {
> return conn_id;
>   }
>   public void setConnId(java.lang.Long value) {
> this.conn_id = value;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2411) Add basic graph summarization algorithm

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964699#comment-14964699
 ] 

ASF GitHub Bot commented on FLINK-2411:
---

Github user s1ck closed the pull request at:

https://github.com/apache/flink/pull/1264


> Add basic graph summarization algorithm
> ---
>
> Key: FLINK-2411
> URL: https://issues.apache.org/jira/browse/FLINK-2411
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 0.10
>Reporter: Martin Junghanns
>Assignee: Martin Junghanns
>Priority: Minor
>
> Graph summarization determines a structural grouping of similar vertices and 
> edges to condense a graph and thus helps to uncover insights about patterns 
> hidden in the graph. It can be used in OLAP-style operations on the graph and 
> is similar to group by in SQL but on the graph structure instead of rows.
>  
> The graph summarization operator represents every vertex group by a single 
> vertex in the summarized graph; edges between vertices in the summary graph 
> represent a group of edges between the vertex group members of the original 
> graph. Summarization is defined by specifying grouping keys for vertices and 
> edges, respectively.
> One publication that presents a Map/Reduce based approach is "Pagrol: 
> Parallel graph olap over large-scale attributed graphs", however they 
> pre-compute the graph-cube before it can be analyzed. With Flink, we can give 
> the user an interactive way of summarizing the graph and do not need to 
> compute the  cube beforehand.
> A more complex approach focuses on summarization on graph patterns  
> "SynopSys: Large Graph Analytics in the SAP HANA Database Through 
> Summarization".
> However, I want to start with a simple algorithm that summarizes the graph on 
> vertex and optionally edge values and additionally stores a count aggregate 
> at summarized vertices/edges.
> Consider the following two examples (e.g., social network with users from 
> cities and friendships with timestamp):
>  
> h4. Input graph:
>  
> Vertices (id, value):
> (0, Leipzig)
> (1, Leipzig)
> (2, Dresden)
> (3, Dresden)
> (4, Dresden)
> (5, Berlin)
> Edges (source, target, value):
> (0, 1, 2014)
> (1, 0, 2014)
> (1, 2, 2013)
> (2, 1, 2013)
> (2, 3, 2014)
> (3, 2, 2014)
> (4, 0, 2013)
> (4, 1, 2015)
> (5, 2, 2015)
> (5, 3, 2015)
> h4. Output graph (summarized on vertex value):
> Vertices (id, value, count)
> (0, Leipzig, 2) // "2 users from Leipzig"
> (2, Dresden, 3) // "3 users from Dresden"
> (5, Berlin, 1) // "1 user from Berlin"
> Edges (source, target, count) 
> (0, 0, 2) // "2 edges between users in Leipzig"
> (0, 2, 1) // "1 edge from users in Leipzig to users in Dresden"
> (2, 0, 3) // "3 edges from users in Dresden to users in Leipzig"
> (2, 2, 2) // "2 edges between users in Dresden"
> (5, 2, 2) // "2 edges from users in Berlin to users in Dresden"
> h4. Output graph (summarized on vertex and edge value):
> Vertices (id, value, count)
> (0, Leipzig, 2)
> (2, Dresden, 3)
> (5, Berlin, 1)
> Edges (source, target, value, count) 
> (0, 0, 2014, 2) // ...
> (0, 2, 2013, 1) // ...
> (2, 0, 2013, 2) // "2 edges from users in Dresden to users in Leipzig with 
> timestamp 2013"
> (2, 0, 2015, 1) // "1 edge from users in Dresden to users in Leipzig with 
> timestamp 2015"
> (2, 2, 2014, 2) // ...
> (5, 2, 2015, 2) // ...
> I've already implemented two versions of the summarization algorithm in our 
> own project [Gradoop|https://github.com/dbs-leipzig/gradoop], which is a 
> graph analytics stack on top of Hadoop + Gelly/Flink with a fixed data model. 
> You can see the current WIP here: 
> 1 [Abstract 
> summarization|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/main/java/org/gradoop/model/impl/operators/Summarization.java]
> 2 [Implementation using 
> cross|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/main/java/org/gradoop/model/impl/operators/SummarizationCross.java]
> 3 [Implementation using 
> joins|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/main/java/org/gradoop/model/impl/operators/SummarizationJoin.java]
> 4 
> [Tests|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/gradoop-flink/src/test/java/org/gradoop/model/impl/EPGraphSummarizeTest.java]
> 5 
> [TestGraph|https://github.com/dbs-leipzig/gradoop/blob/%2345_gradoop_flink/dev-support/social-network.pdf]
> I would basically use the same implementation as in 3 in combination with 
> KeySelectors to select the grouping keys on vertices and edges.
> As you can see in the example, each vertex in the resulting graph has a 
> vertex id that is contained in the original graph. This id is the smallest id 
> among the grouped vertices (e.g., vertices 2, 3 and 4 

[jira] [Commented] (FLINK-2852) Fix flaky ScalaShellITSuite

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964757#comment-14964757
 ] 

Till Rohrmann commented on FLINK-2852:
--

Hi [~sachingoel0101], you're right that the test is not robust. I've reworked 
and pushed a commit which should hopefully solve the problem.

> Fix flaky ScalaShellITSuite
> ---
>
> Key: FLINK-2852
> URL: https://issues.apache.org/jira/browse/FLINK-2852
> Project: Flink
>  Issue Type: Bug
>  Components: Scala Shell
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 0.10
>
>
> The {{ScalaShellITSuite}} checks the log output whether a job has successful 
> completed or not. For that to happen it checks for a {{Job execution switched 
> to status FINISHED}} string in the log output. However, if the 
> {{FlinkClientActor}} receives first a {{JobResultSuccess}} message before 
> receiving the {{JobStatusChanged}} message, then it will send the execution 
> result back to the {{Client}} and terminate itself. This has the consequence 
> that the output will never contain the above-mentioned string.
> I propose to use a different mean to check whether a job has finished 
> successfully or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2852) Fix flaky ScalaShellITSuite

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-2852.

Resolution: Fixed

Fixed in 630798d

> Fix flaky ScalaShellITSuite
> ---
>
> Key: FLINK-2852
> URL: https://issues.apache.org/jira/browse/FLINK-2852
> Project: Flink
>  Issue Type: Bug
>  Components: Scala Shell
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 0.10
>
>
> The {{ScalaShellITSuite}} checks the log output whether a job has successful 
> completed or not. For that to happen it checks for a {{Job execution switched 
> to status FINISHED}} string in the log output. However, if the 
> {{FlinkClientActor}} receives first a {{JobResultSuccess}} message before 
> receiving the {{JobStatusChanged}} message, then it will send the execution 
> result back to the {{Client}} and terminate itself. This has the consequence 
> that the output will never contain the above-mentioned string.
> I propose to use a different mean to check whether a job has finished 
> successfully or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2652) Failing PartitionRequestClientFactoryTest

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-2652.
--
Resolution: Fixed

Fixed in b2339464

> Failing PartitionRequestClientFactoryTest
> -
>
> Key: FLINK-2652
> URL: https://issues.apache.org/jira/browse/FLINK-2652
> Project: Flink
>  Issue Type: Bug
>Reporter: Ufuk Celebi
>Priority: Minor
>  Labels: test-stability
>
> PartitionRequestClientFactoryTest fails when running {{mvn 
> -Dhadoop.version=2.6.0 clean verify}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2804) Support blocking job submission with Job Manager recovery

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-2804.

Resolution: Fixed

Added in d18f580

> Support blocking job submission with Job Manager recovery
> -
>
> Key: FLINK-2804
> URL: https://issues.apache.org/jira/browse/FLINK-2804
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Till Rohrmann
>Priority: Minor
>
> Submitting a job in a blocking fashion with JobManager recovery and a failing 
> JobManager fails on the client side (the one submitting the job). The job 
> still continues to be recovered.
> I propose to add simple support to re-retrieve the leading job manager and 
> update the client actor with it and then wait for the result as before.
> As of the current standing in PR #1153 
> (https://github.com/apache/flink/pull/1153) the job manager assumes that the 
> same actor is running and just keeps on sending execution state updates etc. 
> (if the listening behaviour is not detached).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2792) Set log level of actor messages to TRACE

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-2792.
--
Resolution: Fixed

Fixed in 3aaee1e

> Set log level of actor messages to TRACE
> 
>
> Key: FLINK-2792
> URL: https://issues.apache.org/jira/browse/FLINK-2792
> Project: Flink
>  Issue Type: Wish
>  Components: JobManager
>Reporter: Ufuk Celebi
>Priority: Trivial
>
> Logging of received job manager actor messages happens at log level DEBUG 
> right now. The used logger is that of the JobManager/TaskManager 
> respectively. This means that as soon as you debug something related to the 
> JobManager/TaskManager you are always flooded with a lot of debug messages.
> Therefore, I would like to set the log level to TRACE for these messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2354) Recover running jobs on JobManager failure

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-2354.
--
Resolution: Fixed

Added in a6890b2

> Recover running jobs on JobManager failure
> --
>
> Key: FLINK-2354
> URL: https://issues.apache.org/jira/browse/FLINK-2354
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 0.10
>
>
> tl;dr Persist JobGraphs in state backend and coordinate reference to state 
> handle via ZooKeeper.
> Problem: When running multiple JobManagers in high availability mode, the 
> leading job manager looses all running jobs when it fails. After a new 
> leading job manager is elected, it is not possible to recover any previously 
> running jobs.
> Solution: The leading job manager, which receives the job graph writes 1) the 
> job graph to a state backend, and 2) a reference to the respective state 
> handle to ZooKeeper. In general, job graphs can become large (multiple MBs, 
> because they include closures etc.). ZooKeeper is not designed for data of 
> this size. The level of indirection via the reference to the state backend 
> keeps the data in ZooKeeper small.
> Proposed ZooKeeper layout:
> /flink (default)
>   +- currentJobs
>+- job id i
> +- state handle reference of job graph i
> The 'currentJobs' node needs to be persistent to allow recovery of jobs 
> between job managers. The currentJobs node needs to satisfy the following 
> invariant: There is a reference to a job graph with id i IFF the respective 
> job graph needs to be recovered by a newly elected job manager leader.
> With this in place, jobs will be recovered from their initial state (as if 
> resubmitted). The next step is to backup the runtime state handles of 
> checkpoints in a similar manner.
> ---
> This work will be based on [~trohrm...@apache.org]'s implementation of 
> FLINK-2291. The leader election service notifies the job manager about 
> granted/revoked leadership. This notification happens via Akka and thus 
> serially *per* job manager, but results in eventually consistent state 
> between job managers. For some snapshots of time it is possible to have a new 
> leader granted leadership, before the old one has been revoked its leadership.
> [~trohrm...@apache.org], can you confirm that leadership does not guarantee 
> mutually exclusive access to the shared 'currentJobs' state?
> For example, the following can happen:
> - JM 1 is leader, JM 2 is standby
> - JOB i is running (and hence /flink/currentJobs/i exists)
> - ZK notifies leader election service (LES) of JM 1 and JM 2
> - LES 2 immediately notifies JM 2 about granted leadership, but LES 1 
> notification revoking leadership takes longer
> - JOB i finishes (TMs don't notice leadership change yet) and JM 1 receives 
> final JobStatusChange
> - JM 2 resubmits the job /flink/currentJobs/i
> - JM 1 removes /flink/currentJobs/i, because it is now finished
> => inconsistent state (wrt the specified invariant above)
> If it is indeed a problem, we can circumvent this with a Curator recipe for 
> [shared locks|http://curator.apache.org/curator-recipes/shared-lock.html] to 
> coordinate the access to currentJobs. The lock needs to be acquired on 
> leadership.
> ---
> Minimum required tests:
> - Unit tests for job graph serialization and writing to state backend and 
> ZooKeeper with expected nodes
> - Unit tests for job submission to job manager in leader/non-leader state
> - Unit tests for leadership granting/revoking and job submission/restarting 
> interleavings
> - Process failure integration tests with single and multiple running jobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2827) Potential resource leak in TwitterSource#loadAuthenticationProperties()

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964778#comment-14964778
 ] 

Maximilian Michels commented on FLINK-2827:
---

Yes, please go ahead.

> Potential resource leak in TwitterSource#loadAuthenticationProperties()
> ---
>
> Key: FLINK-2827
> URL: https://issues.apache.org/jira/browse/FLINK-2827
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Ted Yu
>Assignee: Saumitra Shahapure
>Priority: Minor
>  Labels: starter
>
> Here is related code:
> {code}
> Properties properties = new Properties();
> try {
> InputStream input = new FileInputStream(authPath);
> properties.load(input);
> input.close();
> } catch (Exception e) {
> throw new RuntimeException("Cannot open .properties 
> file: " + authPath, e);
> }
> {code}
> If there is exception coming out of properties.load() call, input would be 
> left open.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2411] Add graph summarization algorithm

[s1ck]

Github user s1ck closed the pull request at:

https://github.com/apache/flink/pull/1264


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2111) Add "stop" signal to cleanly shutdown streaming jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965150#comment-14965150
 ] 

ASF GitHub Bot commented on FLINK-2111:
---

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-149580271
  
Is there any news here?


> Add "stop" signal to cleanly shutdown streaming jobs
> 
>
> Key: FLINK-2111
> URL: https://issues.apache.org/jira/browse/FLINK-2111
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime, JobManager, Local Runtime, 
> Streaming, TaskManager, Webfrontend
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> Currently, streaming jobs can only be stopped using "cancel" command, what is 
> a "hard" stop with no clean shutdown.
> The new introduced "stop" signal, will only affect streaming source tasks 
> such that the sources can stop emitting data and shutdown cleanly, resulting 
> in a clean shutdown of the whole streaming job.
> This feature is a pre-requirment for 
> https://issues.apache.org/jira/browse/FLINK-1929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2877) Move Streaming API out of Staging

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-2877:
---

 Summary: Move Streaming API out of Staging
 Key: FLINK-2877
 URL: https://issues.apache.org/jira/browse/FLINK-2877
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Reporter: Aljoscha Krettek


As discussed on the mailing list we want to move the Streaming API out of the 
staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965247#comment-14965247
 ] 

ASF GitHub Bot commented on FLINK-2873:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/1271

[FLINK-2873] detect & serve the job manager log files correctly

When multiple job masters are started or old log files are present, the
log file could not be served through the web frontend.

This commit determines the log file path through the "log.dir"
environment variable or the config entry JOB_MANAGER_WEB_LOG_PATH_KEY.

The latter is merely a shortcut for the unit tests or if multiple web
frontends run inside one VM.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-2873

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1271.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1271


commit 32b723df432eb75b06f979c2a3aa387d31894da5
Author: Maximilian Michels 
Date:   2015-10-20T15:07:40Z

[FLINK-2873] detect & serve the job manager log files correctly

When multiple job masters are started or old log files are present, the
log file could not be served through the web frontend.

This commit determines the log file path through the "log.dir"
environment variable or the config entry JOB_MANAGER_WEB_LOG_PATH_KEY.

The latter is merely a shortcut for the unit tests or if multiple web
frontends run inside one VM.




> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2879) Links in documentation are broken

[Nikolaas Steenbergen (JIRA)]

Nikolaas Steenbergen created FLINK-2879:
---

 Summary: Links in documentation are broken
 Key: FLINK-2879
 URL: https://issues.apache.org/jira/browse/FLINK-2879
 Project: Flink
  Issue Type: Bug
  Components: website
Reporter: Nikolaas Steenbergen
Priority: Minor


https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html

the image of system components link to wrong locations:
e.g.:
https://ci.apache.org/projects/flink/flink-docs-master/internals/internals/general_arch.html
instead of:
https://ci.apache.org/projects/flink/flink-docs-master/internals/general_arch.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-2880) Allow DeserializationSchema to throw IOException in deserialization method

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen updated FLINK-2880:

Priority: Critical  (was: Major)

> Allow DeserializationSchema to throw IOException in deserialization method
> --
>
> Key: FLINK-2880
> URL: https://issues.apache.org/jira/browse/FLINK-2880
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 0.10
>
>
> The DeserializationSchema should be allowed to forward exceptions on 
> deserialization, as they are thrown by many advanced de-coder methods (like 
> Avro)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2880) Allow DeserializationSchema to throw IOException in deserialization method

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-2880:
---

 Summary: Allow DeserializationSchema to throw IOException in 
deserialization method
 Key: FLINK-2880
 URL: https://issues.apache.org/jira/browse/FLINK-2880
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.10


The DeserializationSchema should be allowed to forward exceptions on 
deserialization, as they are thrown by many advanced de-coder methods (like 
Avro)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels resolved FLINK-2873.
---
Resolution: Fixed

Fixed in {{ef1b708}}.

> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2827] Closing FileInputStream in finall...

[ssaumitra]

GitHub user ssaumitra opened a pull request:

https://github.com/apache/flink/pull/1276

[FLINK-2827] Closing FileInputStream in finally block to avoid unused  open 
stream.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssaumitra/flink FLINK-2254

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1276.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1276


commit 4a16423d24f73533674b555e9fdf26b2b0423182
Author: Saumitra Shahapure 
Date:   2015-10-20T17:28:31Z

[FLINK-2827] Closing FileInputStream in finally block to avoid unused open 
stream.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2827) Potential resource leak in TwitterSource#loadAuthenticationProperties()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965428#comment-14965428
 ] 

ASF GitHub Bot commented on FLINK-2827:
---

GitHub user ssaumitra opened a pull request:

https://github.com/apache/flink/pull/1276

[FLINK-2827] Closing FileInputStream in finally block to avoid unused  open 
stream.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssaumitra/flink FLINK-2254

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1276.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1276


commit 4a16423d24f73533674b555e9fdf26b2b0423182
Author: Saumitra Shahapure 
Date:   2015-10-20T17:28:31Z

[FLINK-2827] Closing FileInputStream in finally block to avoid unused open 
stream.




> Potential resource leak in TwitterSource#loadAuthenticationProperties()
> ---
>
> Key: FLINK-2827
> URL: https://issues.apache.org/jira/browse/FLINK-2827
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Ted Yu
>Assignee: Saumitra Shahapure
>Priority: Minor
>  Labels: starter
>
> Here is related code:
> {code}
> Properties properties = new Properties();
> try {
> InputStream input = new FileInputStream(authPath);
> properties.load(input);
> input.close();
> } catch (Exception e) {
> throw new RuntimeException("Cannot open .properties 
> file: " + authPath, e);
> }
> {code}
> If there is exception coming out of properties.load() call, input would be 
> left open.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2876) Minutiae

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965500#comment-14965500
 ] 

ASF GitHub Bot commented on FLINK-2876:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1277

[FLINK-2876] Minutiae

A collection of small documentation and grammar updates.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 2876_minutiae

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1277.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1277


commit f08da1f7bb42827d9d1d35bf41d9ecc7b265294c
Author: Greg Hogan 
Date:   2015-10-20T15:57:33Z

[FLINK-2876] Minutiae

A collection of small documentation and grammar updates.




> Minutiae
> 
>
> Key: FLINK-2876
> URL: https://issues.apache.org/jira/browse/FLINK-2876
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
>
> A collection of small documentation and grammar updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2876] Minutiae

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1277

[FLINK-2876] Minutiae

A collection of small documentation and grammar updates.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 2876_minutiae

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1277.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1277


commit f08da1f7bb42827d9d1d35bf41d9ecc7b265294c
Author: Greg Hogan 
Date:   2015-10-20T15:57:33Z

[FLINK-2876] Minutiae

A collection of small documentation and grammar updates.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2862] [Storm Compatibility] FlinkTopolo...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/1274#issuecomment-149639349
  
@StephanEwen Please give short review. Should be good to get merged. Will 
address https://issues.apache.org/jira/browse/FLINK-2861 after this is merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2862) FlinkTopologyBuilder should use proper generic types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965424#comment-14965424
 ] 

ASF GitHub Bot commented on FLINK-2862:
---

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/1274#issuecomment-149639349
  
@StephanEwen Please give short review. Should be good to get merged. Will 
address https://issues.apache.org/jira/browse/FLINK-2861 after this is merged.


> FlinkTopologyBuilder should use proper generic types
> 
>
> Key: FLINK-2862
> URL: https://issues.apache.org/jira/browse/FLINK-2862
> Project: Flink
>  Issue Type: Bug
>  Components: Storm Compatibility
>Affects Versions: 0.10
>Reporter: Stephan Ewen
>Assignee: Matthias J. Sax
>
> The Flink Topology Builder uses only raw types, resulting in bugs due to 
> missing compile time checks of types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2881) WebRuntimeMonitor overly restrictive file patterns

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-2881:
-

 Summary: WebRuntimeMonitor overly restrictive file patterns
 Key: FLINK-2881
 URL: https://issues.apache.org/jira/browse/FLINK-2881
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 0.10
Reporter: Greg Hogan
Priority: Blocker


{{WebRuntimeMonitor}} verifies the log and stdout directory by checking that 
files matching the patterns {noformat}.*jobmanager[^\\.]*\\.log{noformat} and 
{noformat}.*jobmanager[^\\.]*\\.out{noformat} were created. The filenames use 
the system hostname which can contain periods, which are excluded by the file 
pattern and prevent the web monitor from starting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2877] Move Streaming API out of Staging...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1273#issuecomment-149657090
  
+1 for `flink-streaming-java` and `flink-streaming-scala`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2877) Move Streaming API out of Staging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965521#comment-14965521
 ] 

ASF GitHub Bot commented on FLINK-2877:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1273#issuecomment-149657090
  
+1 for `flink-streaming-java` and `flink-streaming-scala`


> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2834) Global round-robin for temporary directories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965426#comment-14965426
 ] 

ASF GitHub Bot commented on FLINK-2834:
---

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/1272#discussion_r42526400
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/FileIOChannel.java
 ---
@@ -154,20 +155,25 @@ private static String randomString(Random random) {
public static final class Enumerator {
 
private final File[] paths;
-   
+
private final String namePrefix;
 
-   private int counter;
+   private int localCounter;
+
+   private static AtomicInteger globalCounter = new 
AtomicInteger();
--- End diff --

Fixed.


> Global round-robin for temporary directories
> 
>
> Key: FLINK-2834
> URL: https://issues.apache.org/jira/browse/FLINK-2834
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Currently each {{FileIOChannel}} performs a local round-robin among temporary 
> directories. Initially every writer will be concurrently spilling to the same 
> first directory, then the same second directory, etc. Even when evenly 
> dispersed this effect will naturally occur due to the bunching effect.
> One solution is to use a global counter when selecting the temporary path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...

[greghogan]

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/1272#discussion_r42526400
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/FileIOChannel.java
 ---
@@ -154,20 +155,25 @@ private static String randomString(Random random) {
public static final class Enumerator {
 
private final File[] paths;
-   
+
private final String namePrefix;
 
-   private int counter;
+   private int localCounter;
+
+   private static AtomicInteger globalCounter = new 
AtomicInteger();
--- End diff --

Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2827) Potential resource leak in TwitterSource#loadAuthenticationProperties()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965432#comment-14965432
 ] 

ASF GitHub Bot commented on FLINK-2827:
---

Github user ssaumitra commented on a diff in the pull request:

https://github.com/apache/flink/pull/1276#discussion_r42527314
  
--- Diff: 
flink-staging/flink-streaming/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
 ---
@@ -133,13 +134,21 @@ protected OAuth1 authenticate() {
private Properties loadAuthenticationProperties() {

Properties properties = new Properties();
+   InputStream input = null;
try {
-   InputStream input = new FileInputStream(authPath);
+   input = new FileInputStream(authPath);
properties.load(input);
-   input.close();
} catch (Exception e) {
throw new RuntimeException("Cannot open .properties 
file: " + authPath, e);
}
+   finally {
+   try {
--- End diff --

This try..catch does not look beautiful. But `input.close()` mandates it. 
Also `close()` will not be called if `FileInputStream` is not open.


> Potential resource leak in TwitterSource#loadAuthenticationProperties()
> ---
>
> Key: FLINK-2827
> URL: https://issues.apache.org/jira/browse/FLINK-2827
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Ted Yu
>Assignee: Saumitra Shahapure
>Priority: Minor
>  Labels: starter
>
> Here is related code:
> {code}
> Properties properties = new Properties();
> try {
> InputStream input = new FileInputStream(authPath);
> properties.load(input);
> input.close();
> } catch (Exception e) {
> throw new RuntimeException("Cannot open .properties 
> file: " + authPath, e);
> }
> {code}
> If there is exception coming out of properties.load() call, input would be 
> left open.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2877] Move Streaming API out of Staging...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1273#issuecomment-149637552
  
Looks pretty straightforward. Some diffs are displayed probably because of 
git bugs / limitations. As this was discussed on the mailing list before and 
all agreed, this should be good.

Only point of discussion could be names. I personally like the names 
`flink-streaming-java` and `flink-streaming-scala`, so +1 from my side


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [docs] Fix documentation for building Flink wi...

[mxm]

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/1260#issuecomment-149623979
  
+1 looks good to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2880) Allow DeserializationSchema to throw IOException in deserialization method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965418#comment-14965418
 ] 

ASF GitHub Bot commented on FLINK-2880:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/1275

[FLINK-2880] [streaming] [API-Breaking] Allow DeserializationSchema to 
forward exceptions

The DeserializationSchema is now allowed to forward exceptions on 
deserialization.
This is useful because many non-trivial decoders (Flink's TypeSerializers, 
Avro, etc) may throw exceptions. Otherwise, this leads to clumly catching and 
nesting of exceptions.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
deser_schema_exceptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1275.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1275


commit 0be44e7244ea7405ca1417735ca4d00d7d640082
Author: Stephan Ewen 
Date:   2015-10-20T17:17:59Z

[FLINK-2880] [streaming] Allow DeserializationSchema to forward exceptions.




> Allow DeserializationSchema to throw IOException in deserialization method
> --
>
> Key: FLINK-2880
> URL: https://issues.apache.org/jira/browse/FLINK-2880
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 0.10
>
>
> The DeserializationSchema should be allowed to forward exceptions on 
> deserialization, as they are thrown by many advanced de-coder methods (like 
> Avro)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2880] [streaming] [API-Breaking] Allow ...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/1275

[FLINK-2880] [streaming] [API-Breaking] Allow DeserializationSchema to 
forward exceptions

The DeserializationSchema is now allowed to forward exceptions on 
deserialization.
This is useful because many non-trivial decoders (Flink's TypeSerializers, 
Avro, etc) may throw exceptions. Otherwise, this leads to clumly catching and 
nesting of exceptions.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
deser_schema_exceptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1275.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1275


commit 0be44e7244ea7405ca1417735ca4d00d7d640082
Author: Stephan Ewen 
Date:   2015-10-20T17:17:59Z

[FLINK-2880] [streaming] Allow DeserializationSchema to forward exceptions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1272

[FLINK-2834] Global round-robin for temporary directories

Multiple TaskManager filesystems can be used by configuring multiple 
temporary directories.
This patch changes the process of spilling files from a per-operator 
round-robin to a global
round-robin such that each directory is written to in turn across all 
operators, reducing
unbalanced I/O due to bunching.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
2834_global_round_robin_for_temporary_directories

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1272


commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b
Author: Greg Hogan 
Date:   2015-10-20T14:47:58Z

[FLINK-2834] Global round-robin for temporary directories

Multiple TaskManager filesystems can be used by configuring multiple 
temporary directories.
This patch changes the process of spilling files from a per-operator 
round-robin to a global
round-robin such that each directory is written to in turn across all 
operators, reducing
unbalanced I/O due to bunching.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965368#comment-14965368
 ] 

ASF GitHub Bot commented on FLINK-2873:
---

Github user mxm closed the pull request at:

https://github.com/apache/flink/pull/1271


> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965364#comment-14965364
 ] 

Maximilian Michels edited comment on FLINK-2873 at 10/20/15 4:43 PM:
-

Fixed in {{e711969}}.


was (Author: mxm):
Fixed in {{ef1b708}}.

> WebRuntimeMonitor does not start when multiple log files are present
> 
>
> Key: FLINK-2873
> URL: https://issues.apache.org/jira/browse/FLINK-2873
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 0.10
>
>
> The {{WebRuntimeMonitor}} checks that in the given log directory only a 
> single file following the pattern {{.*jobmanager[^\.]*\.log}} is present. If 
> this is not the case, then the {{WebRuntimeMonitor}} will fail to start. This 
> is a problem if you have multiple {{JobManager}} started which all log to the 
> same directory as in the HA case, for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2877) Move Streaming API out of Staging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965396#comment-14965396
 ] 

ASF GitHub Bot commented on FLINK-2877:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/1273

[FLINK-2877] Move Streaming API out of Staging package



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink stream-out-of-staging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1273


commit 444226970e31856787fbebdd7793805293faf13b
Author: Aljoscha Krettek 
Date:   2015-10-11T09:37:29Z

[FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant

This adds method state() on Trigger context that should be used to
create an OperatorState to deal with fault-tolerant state.

WindowAssigner now has a method getWindowSerializer() that is used to
get a TypeSerializer for the Windows that it assigns. The Serializer for
the Key is retrieved from the input KeyedStream and the serializer for
the input elements is already available.

During checkpointing all currently in-flight windows (per key, per
window) are serialized using the TypeSerializers. The state that is
accessible in Triggers using state() is kept in a
HashMap, this is serialized using java
serialization.

commit f760b616af0e1608cb4c190aeb264da72f624f4c
Author: Aljoscha Krettek 
Date:   2015-10-17T11:35:24Z

Replace Trigger.onTime by Trigger.onProcessingTime/onEventTime

This also renames WatermarkTrigger to EventTimeTrigger and
ContinuousWatermarkTrigger to ContinuousEventTimeTrigger.

commit b697080673d37deb1aafc17180207c6f8871e409
Author: Rufus Refactor 
Date:   2015-10-20T15:04:21Z

[FLINK-2877] Move Streaming API out of Staging package




> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2877] Move Streaming API out of Staging...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1273#issuecomment-149636785
  
GitHub sync is lagging behind, so only one commit (the latest) in the PR is 
relevant, the others are already committed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1272#discussion_r42524229
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/FileIOChannel.java
 ---
@@ -154,20 +155,25 @@ private static String randomString(Random random) {
public static final class Enumerator {
 
private final File[] paths;
-   
+
private final String namePrefix;
 
-   private int counter;
+   private int localCounter;
+
+   private static AtomicInteger globalCounter = new 
AtomicInteger();
--- End diff --

Minor (optional) comment: In the (undocumented) code style that Flink 
follows implicitly, the static member fields are the first in the class, before 
the instance member fields.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2873] detect & serve the job manager lo...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/1271

[FLINK-2873] detect & serve the job manager log files correctly

When multiple job masters are started or old log files are present, the
log file could not be served through the web frontend.

This commit determines the log file path through the "log.dir"
environment variable or the config entry JOB_MANAGER_WEB_LOG_PATH_KEY.

The latter is merely a shortcut for the unit tests or if multiple web
frontends run inside one VM.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-2873

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1271.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1271


commit 32b723df432eb75b06f979c2a3aa387d31894da5
Author: Maximilian Michels 
Date:   2015-10-20T15:07:40Z

[FLINK-2873] detect & serve the job manager log files correctly

When multiple job masters are started or old log files are present, the
log file could not be served through the web frontend.

This commit determines the log file path through the "log.dir"
environment variable or the config entry JOB_MANAGER_WEB_LOG_PATH_KEY.

The latter is merely a shortcut for the unit tests or if multiple web
frontends run inside one VM.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-2877) Move Streaming API out of Staging

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965284#comment-14965284
 ] 

Stephan Ewen edited comment on FLINK-2877 at 10/20/15 3:49 PM:
---

+1 for {{flink-streaming-java}}. That makes for a nice set of artifacts:

  - {{flink-core}}
  - {{flink-java}}
  - {{flink-scala}}
  - {{flink-streaming-java}}
  - {{flink-streaming-scala}}

(the core project being cross batch/streaming, the batch projects missing a 
prefix in some sense)


was (Author: stephanewen):
+1 for {{flink-streaming-java}}. That makes for a nice set of artifacts:

  - {{flink-core}}
  - {{flink-java}}
  - {{flink-scala}}
  - {{flink-streaming-java}}
  - {{flink-streaming-scala}}

> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2834) Global round-robin for temporary directories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965407#comment-14965407
 ] 

ASF GitHub Bot commented on FLINK-2834:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1272#issuecomment-149636019
  
Looks good to me, +1 to merge this!


> Global round-robin for temporary directories
> 
>
> Key: FLINK-2834
> URL: https://issues.apache.org/jira/browse/FLINK-2834
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Currently each {{FileIOChannel}} performs a local round-robin among temporary 
> directories. Initially every writer will be concurrently spilling to the same 
> first directory, then the same second directory, etc. Even when evenly 
> dispersed this effect will naturally occur due to the bunching effect.
> One solution is to use a global counter when selecting the temporary path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1272#issuecomment-149636019
  
Looks good to me, +1 to merge this!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2862] [Storm Compatibility] FlinkTopolo...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/flink/pull/1274

[FLINK-2862] [Storm Compatibility] FlinkTopologyBuilder should use proper 
generic types



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/flink 
flink-2862-genericsFlinkTopologyBuilder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1274


commit 25f81ea2e3bf501e253ac68ec59ffc1ad26a6413
Author: mjsax 
Date:   2015-10-20T16:06:00Z

[FLINK-2862] [Storm Compatibility] FlinkTopologyBuilder should use proper 
generic types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2834) Global round-robin for temporary directories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965367#comment-14965367
 ] 

ASF GitHub Bot commented on FLINK-2834:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1272

[FLINK-2834] Global round-robin for temporary directories

Multiple TaskManager filesystems can be used by configuring multiple 
temporary directories.
This patch changes the process of spilling files from a per-operator 
round-robin to a global
round-robin such that each directory is written to in turn across all 
operators, reducing
unbalanced I/O due to bunching.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
2834_global_round_robin_for_temporary_directories

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1272


commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b
Author: Greg Hogan 
Date:   2015-10-20T14:47:58Z

[FLINK-2834] Global round-robin for temporary directories

Multiple TaskManager filesystems can be used by configuring multiple 
temporary directories.
This patch changes the process of spilling files from a per-operator 
round-robin to a global
round-robin such that each directory is written to in turn across all 
operators, reducing
unbalanced I/O due to bunching.




> Global round-robin for temporary directories
> 
>
> Key: FLINK-2834
> URL: https://issues.apache.org/jira/browse/FLINK-2834
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Currently each {{FileIOChannel}} performs a local round-robin among temporary 
> directories. Initially every writer will be concurrently spilling to the same 
> first directory, then the same second directory, etc. Even when evenly 
> dispersed this effect will naturally occur due to the bunching effect.
> One solution is to use a global counter when selecting the temporary path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2862) FlinkTopologyBuilder should use proper generic types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965416#comment-14965416
 ] 

ASF GitHub Bot commented on FLINK-2862:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/flink/pull/1274

[FLINK-2862] [Storm Compatibility] FlinkTopologyBuilder should use proper 
generic types



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/flink 
flink-2862-genericsFlinkTopologyBuilder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1274


commit 25f81ea2e3bf501e253ac68ec59ffc1ad26a6413
Author: mjsax 
Date:   2015-10-20T16:06:00Z

[FLINK-2862] [Storm Compatibility] FlinkTopologyBuilder should use proper 
generic types




> FlinkTopologyBuilder should use proper generic types
> 
>
> Key: FLINK-2862
> URL: https://issues.apache.org/jira/browse/FLINK-2862
> Project: Flink
>  Issue Type: Bug
>  Components: Storm Compatibility
>Affects Versions: 0.10
>Reporter: Stephan Ewen
>Assignee: Matthias J. Sax
>
> The Flink Topology Builder uses only raw types, resulting in bugs due to 
> missing compile time checks of types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2877] Move Streaming API out of Staging...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/1273

[FLINK-2877] Move Streaming API out of Staging package



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink stream-out-of-staging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1273


commit 444226970e31856787fbebdd7793805293faf13b
Author: Aljoscha Krettek 
Date:   2015-10-11T09:37:29Z

[FLINK-2864] Make State of General-Purpose Window Operators Fault-Tolerant

This adds method state() on Trigger context that should be used to
create an OperatorState to deal with fault-tolerant state.

WindowAssigner now has a method getWindowSerializer() that is used to
get a TypeSerializer for the Windows that it assigns. The Serializer for
the Key is retrieved from the input KeyedStream and the serializer for
the input elements is already available.

During checkpointing all currently in-flight windows (per key, per
window) are serialized using the TypeSerializers. The state that is
accessible in Triggers using state() is kept in a
HashMap, this is serialized using java
serialization.

commit f760b616af0e1608cb4c190aeb264da72f624f4c
Author: Aljoscha Krettek 
Date:   2015-10-17T11:35:24Z

Replace Trigger.onTime by Trigger.onProcessingTime/onEventTime

This also renames WatermarkTrigger to EventTimeTrigger and
ContinuousWatermarkTrigger to ContinuousEventTimeTrigger.

commit b697080673d37deb1aafc17180207c6f8871e409
Author: Rufus Refactor 
Date:   2015-10-20T15:04:21Z

[FLINK-2877] Move Streaming API out of Staging package




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2877) Move Streaming API out of Staging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965413#comment-14965413
 ] 

ASF GitHub Bot commented on FLINK-2877:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1273#issuecomment-149637552
  
Looks pretty straightforward. Some diffs are displayed probably because of 
git bugs / limitations. As this was discussed on the mailing list before and 
all agreed, this should be good.

Only point of discussion could be names. I personally like the names 
`flink-streaming-java` and `flink-streaming-scala`, so +1 from my side


> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2820) Configuration not passed to JobGraphGenerator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965637#comment-14965637
 ] 

ASF GitHub Bot commented on FLINK-2820:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1278

[FLINK-2820] Configuration not passed to JobGraphGenerator

This was previously reported as FLINK-2625 (commit 
8a84937215ea575fa94a00d11c2517902d252756).
The Client class was concurrently refactored with FLINK-2097 (commit 
71bf2f570861daae53b24bfcf1d06aedb85311b9).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
2820_configuration_not_passed_to_jobgraphgenerator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1278


commit acd72bb02d5245c705c0dce910ebd2709c14e4bc
Author: Greg Hogan 
Date:   2015-10-20T16:13:10Z

[FLINK-2820] Configuration not passed to JobGraphGenerator

This was previously reported as FLINK-2625 (commit 
8a84937215ea575fa94a00d11c2517902d252756).
The Client class was concurrently refactored with FLINK-2097 (commit 
71bf2f570861daae53b24bfcf1d06aedb85311b9).




> Configuration not passed to JobGraphGenerator
> -
>
> Key: FLINK-2820
> URL: https://issues.apache.org/jira/browse/FLINK-2820
> Project: Flink
>  Issue Type: Bug
>  Components: Command-line client
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 0.10
>
>
> This was previously reported as FLINK-2625 (commit 
> 8a84937215ea575fa94a00d11c2517902d252756). The Client class was concurrently 
> refactored with FLINK-2097 (commit 71bf2f570861daae53b24bfcf1d06aedb85311b9).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2872] [Documentation] Update the docume...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1268


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2872) Update the documentation for Scala part to add readFileOfPrimitives

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965755#comment-14965755
 ] 

ASF GitHub Bot commented on FLINK-2872:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1268


> Update the documentation for Scala part to add readFileOfPrimitives
> ---
>
> Key: FLINK-2872
> URL: https://issues.apache.org/jira/browse/FLINK-2872
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Henry Saputra
>Assignee: Henry Saputra
>Priority: Minor
>
> Currently the Scala part of the ExecutionEnvironment missing 
> readFileOfPrimitives to create Dataset from file for primitive types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2882) Improve performance of string conversions

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965649#comment-14965649
 ] 

Ufuk Celebi commented on FLINK-2882:


Regarding {{toShortString}}: This is exclusively used in the network stack and 
there it's only used for debugging as far as I know, as part of the 
{{toString}} methods of result partitions, because they require two IDs to 
identify their source/target (which gets very long even as hex strings). Can 
you tell what fraction of calls come from {{toShortString}}? We can also think 
about removing that variant as it was mostly useful in the early days during 
debugging the network stack.

In general, I'm wondering why the ID toString methods are called so often. Can 
you give the top 10 stack traces leading to it or so? And what LOG level you 
are using?

In any case, both your suggestions sound reasonable to me.

> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call 
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert 
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports 
> 600,000 calls taking 13.4 seconds. Improving 
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds. 
> Additionally memoizing the string values in {{AbstractID}} reduce the time to 
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to 
> ~1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2685) TaskManager deadlock on NetworkBufferPool

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965650#comment-14965650
 ] 

Ufuk Celebi commented on FLINK-2685:


Greg, any news on this one? Can you share the program?

> TaskManager deadlock on NetworkBufferPool
> -
>
> Key: FLINK-2685
> URL: https://issues.apache.org/jira/browse/FLINK-2685
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Ufuk Celebi
>
> This deadlock occurs intermittently. I have a {{join}} followed by a 
> {{chain}} followed by a {{reduceGroup}}. Stack traces and local 
> variables from one each of the {{join}} threads below.
> The {{join}}'s are waiting on a buffer to become available 
> ({{networkBufferPool.availableMemorySegments.count=0}}). Both 
> {{LocalBufferPool}}'s have been given extra capacity ({{currentPoolSize=60 > 
> numberOfRequiredMemorySegments=32}}). The first {{join}} is at full capacity 
> ({{currentPoolSize=numberOfRequestedMemorySegments=60}}) yet the second 
> {{join}} has not acquired any ({{numberOfRequestedMemorySegments=0}}).
> {{LocalBufferPool.returnExcessMemorySegments}} only recycles 
> {{MemorySegment}}'s from its {{availableMemorySegments}}, so any requested 
> {{Buffer}}'s will only be released when explicitly recycled.
> First join stack trace and variable values from 
> {{LocalBufferPool.requestBuffer}}:
> {noformat}
> owns: SpanningRecordSerializer  (id=723)   
> waiting for: ArrayDeque  (id=724)  
> Object.wait(long) line: not available [native method] 
> LocalBufferPool.requestBuffer(boolean) line: 163  
> LocalBufferPool.requestBufferBlocking() line: 133 
> RecordWriter.emit(T) line: 92  
> OutputCollector.collect(T) line: 65
> JoinOperator$ProjectFlatJoinFunction.join(T1, T2, Collector) 
> line: 1088   
> ReusingBuildSecondHashMatchIterator.callWithNextKey(FlatJoinFunction,
>  Collector) line: 137   
> JoinDriver.run() line: 208
> RegularPactTask.run() line: 489 
> RegularPactTask.invoke() line: 354  
> Task.run() line: 581  
> Thread.run() line: 745
> {noformat}
> {noformat}
> this  LocalBufferPool  (id=403)   
>   availableMemorySegments ArrayDeque  (id=398) 
>   elementsObject[16]  (id=422)
>   head14  
>   tail14  
>   currentPoolSize 60  
>   isDestroyed false   
>   networkBufferPool   NetworkBufferPool  (id=354) 
>   allBufferPools  HashSet  (id=424)
>   availableMemorySegments ArrayBlockingQueue  (id=427) 
>   count   0   
>   items   Object[10240]  (id=674) 
>   itrsnull
>   lockReentrantLock  (id=675) 
>   notEmpty
> AbstractQueuedSynchronizer$ConditionObject  (id=678)
>   notFull AbstractQueuedSynchronizer$ConditionObject  
> (id=679)
>   putIndex6954
>   takeIndex   6954
>   factoryLock Object  (id=430)
>   isDestroyed false   
>   managedBufferPools  HashSet  (id=431)
>   memorySegmentSize   32768   
>   numTotalRequiredBuffers 3226
>   totalNumberOfMemorySegments 10240   
>   numberOfRequestedMemorySegments 60  
>   numberOfRequiredMemorySegments  32  
>   owner   null
>   registeredListeners ArrayDeque  (id=421) 
>   elementsObject[16]  (id=685)
>   head0   
>   tail0   
> askToRecycle  false   
> isBlockingtrue
> {noformat}
> Second join stack trace and variable values from 
> {{SingleInputGate.getNextBufferOrEvent}}:
> {noformat}
> Unsafe.park(boolean, long) line: not available [native method]
> LockSupport.parkNanos(Object, long) line: 215 
> AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2078
> LinkedBlockingQueue.poll(long, TimeUnit) line: 467 
> SingleInputGate.getNextBufferOrEvent() line: 414  
> MutableRecordReader(AbstractRecordReader).getNextRecord(T) line: 79 
> MutableRecordReader.next(T) line: 34   
> ReaderIterator.next(T) line: 59
> MutableHashTable$ProbeIterator.next() line: 1581  
> MutableHashTable.processProbeIter() line: 457  
> MutableHashTable.nextRecord() line: 555
> ReusingBuildSecondHashMatchIterator.callWithNextKey(FlatJoinFunction,
>  Collector) line: 110   
> JoinDriver.run() line: 208
> RegularPactTask.run() line: 

[jira] [Commented] (FLINK-2872) Update the documentation for Scala part to add readFileOfPrimitives

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965227#comment-14965227
 ] 

ASF GitHub Bot commented on FLINK-2872:
---

Github user hsaputra commented on the pull request:

https://github.com/apache/flink/pull/1268#issuecomment-149593562
  
Will merge this sometime today if no more comments.


> Update the documentation for Scala part to add readFileOfPrimitives
> ---
>
> Key: FLINK-2872
> URL: https://issues.apache.org/jira/browse/FLINK-2872
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Henry Saputra
>Assignee: Henry Saputra
>Priority: Minor
>
> Currently the Scala part of the ExecutionEnvironment missing 
> readFileOfPrimitives to create Dataset from file for primitive types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2872] [Documentation] Update the docume...

[hsaputra]

Github user hsaputra commented on the pull request:

https://github.com/apache/flink/pull/1268#issuecomment-149593562
  
Will merge this sometime today if no more comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-2878) JobManager warns: Unexpected leader address pattern

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-2878:
-

 Summary: JobManager warns: Unexpected leader address pattern
 Key: FLINK-2878
 URL: https://issues.apache.org/jira/browse/FLINK-2878
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 0.10
Reporter: Maximilian Michels
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 0.10


The JobManager log shows this multiple times when viewing the log through the 
web frontend:

{noformat}
16:58:37,201 WARN  
org.apache.flink.runtime.webmonitor.handlers.HandlerRedirectUtils  - Unexpected 
leader address pattern akka://flink/user/jobmanager. Cannot extract host.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-2877) Move Streaming API out of Staging

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned FLINK-2877:
---

Assignee: Aljoscha Krettek

> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2877) Move Streaming API out of Staging

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965240#comment-14965240
 ] 

Aljoscha Krettek commented on FLINK-2877:
-

I think I already did it, was not too hard. I'm committing this as another user 
to not muddle the Github statistics, some people might not like it if I 
suddenly have ~80.000+ on my stats. 

One thing that remains is to decide whether to rename flink-streaming-core to 
flink-streaming-java. I think it should be renamed, because it is what you add 
as a dependency if you write a java program. This also gives us the option to 
refactor it into a proper flink-streaming-core package in the future without 
breaking it for users.

> Move Streaming API out of Staging
> -
>
> Key: FLINK-2877
> URL: https://issues.apache.org/jira/browse/FLINK-2877
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discussed on the mailing list we want to move the Streaming API out of the 
> staging package structure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2884) Apply JMH on HashVsSortMiniBenchmark class.

[GaoLun (JIRA)]

GaoLun created FLINK-2884:
-

 Summary: Apply JMH on HashVsSortMiniBenchmark class.
 Key: FLINK-2884
 URL: https://issues.apache.org/jira/browse/FLINK-2884
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: GaoLun
Assignee: GaoLun
Priority: Minor


JMH is a Java harness for building, running, and analysing 
nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks 
method in order to get much more accurate results.Modify the 
HashVsSortMiniBenchmark class and move it to flink-benchmark module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2858) Cannot build Flink Scala 2.11 with IntelliJ

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965087#comment-14965087
 ] 

Till Rohrmann commented on FLINK-2858:
--

Thanks for your help [~aalexandrov] :-) I did not know how to properly change 
the Scala version. I'm ok with changing the version via the shell script and 
documenting it on the web site.

> Cannot build Flink Scala 2.11 with IntelliJ
> ---
>
> Key: FLINK-2858
> URL: https://issues.apache.org/jira/browse/FLINK-2858
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>
> If I activate the scala-2.11 profile from within IntelliJ (and thus 
> deactivate the scala-2.10 profile) in order to build Flink with Scala 2.11, 
> then Flink cannot be built. The problem is that some Scala macros cannot be 
> expanded because they were compiled with the wrong version (I assume 2.10).
> This makes debugging tests with Scala 2.11 in IntelliJ impossible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [docs] Fix documentation for building Flink wi...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1260#issuecomment-149567354
  
Looks good to me. +1 for merging. Maybe we could also add some 
documentation for how to change the Scala version with your favourite IDE as 
described by you in https://issues.apache.org/jira/browse/FLINK-2858.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-2651) Failing NettyServerLowAndHighWatermarkTest

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi resolved FLINK-2651.

   Resolution: Fixed
Fix Version/s: 0.10

Fixed in 371817101848fbb993c02e14a705b5399f75

> Failing NettyServerLowAndHighWatermarkTest
> --
>
> Key: FLINK-2651
> URL: https://issues.apache.org/jira/browse/FLINK-2651
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 0.10
>
>
> https://travis-ci.org/aljoscha/flink/jobs/79610050



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2738) StackOverflow in case of unbounded restarts

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi resolved FLINK-2738.

   Resolution: Fixed
Fix Version/s: 0.10

Fixed via 73c73e92750ab8fb068d0a3cb37afcb642084fc0

> StackOverflow in case of unbounded restarts
> ---
>
> Key: FLINK-2738
> URL: https://issues.apache.org/jira/browse/FLINK-2738
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.10
>Reporter: Ufuk Celebi
> Fix For: 0.10
>
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/81593797/log.txt
> With unbounded number of restarts and a permanent failure (like no resources 
> available), the ExecutionGraph can end up in an infinite restart loop leading 
> to stack overflow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)