[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15129042#comment-15129042
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/1579

[FLINK-3226] Add DataSet scan and  conversion to DataSet[Row]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink dataSetTrans

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1579


commit c0073a13fcbbe4a730c2d967561838a28d574c2d
Author: Fabian Hueske 
Date:   2016-02-02T16:15:28Z

[FLINK-3226] Add DataSet scan and  conversion to DataSet[Row]




> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Add DataSet scan and conversion t...

[fhueske]

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/1579

[FLINK-3226] Add DataSet scan and  conversion to DataSet[Row]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink dataSetTrans

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1579


commit c0073a13fcbbe4a730c2d967561838a28d574c2d
Author: Fabian Hueske 
Date:   2016-02-02T16:15:28Z

[FLINK-3226] Add DataSet scan and  conversion to DataSet[Row]




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-178836268
  
> I had some more comments concerning the failure case handling of stop 
calls. The first problem is still the handling of exceptions when calling stop 
on the Invokable in Task.stopExecution. The exception will only be logged but 
no further action is taken. This can lead to a situation where we have a 
corrupted state. I think, we should fail the task in such a situation.

I see what you mean, but we would not get a corrupted state, do we? 
However, I agree that if `Task.stopExecution` throws an exception,  we should 
report this to the user somehow (not just by logging). Failing the task is fine 
with me. I updated the code for this.

> Additionally, the case that a task cannot be found on the TaskManager and 
that an exception occurs in Task.stopExecution are treated identically by 
sending a TaskOperationResult with success == false to the JobManager. On the 
JobManager side this will only be logged. I think the exception case should be 
handled differently. Failing the execution, for example.

This should not be necessary. There are two exception cases here: (1) The 
task in not stoppable (ie, for a batch job). Failing the execution would of 
course not be appropriate. (2) `Task.stopExecution` throws and exception. This 
case is handled already by the discussion from above.

> And it is still possible that you send a StopJob message to the 
JobManager, see that the job is in state RUNNING, then the ExecutionGraph 
switches to RESTARTING, and then the stop call is executed on the 
ExecutionGraph which won't have an effect. As a user you will receive a 
StoppingSuccess message but the job will simply be restarted. I think we should 
also allow stopping jobs when they are in the state RESTARTING.

Design decision.  I extended allowed states for STOP to CREATED, RUNNING, 
and RESTARTING, which is the easier solution. Not sure if we should include 
CREATED or not though... 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Scala DataStream accessors fix

[mbalassi]

Github user mbalassi commented on the pull request:

https://github.com/apache/flink/pull/1574#issuecomment-178823650
  
Thanks for the comments. @gumchum: Internally it is stored as a Java 
`Iterator` and that is the reason why  left it so, but it definitely make sense 
to convert it to Scala before passing it to the users, so their code does not 
get "polluted" with Java stuff. I am modifying it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3310) Add back pressure statistics to web frontend

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128681#comment-15128681
 ] 

Ufuk Celebi commented on FLINK-3310:


Sorry, I didn't see it. I would like to close it as superseded by this issue. 
Is that OK with you? If not, I can also do it vice versa.

> Add back pressure statistics to web frontend
> 
>
> Key: FLINK-3310
> URL: https://issues.apache.org/jira/browse/FLINK-3310
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Minor
>
> When a task is receiving data at a higher rate than it can process, the task 
> is back pressuring preceding tasks. Currently, there is no way to tell 
> whether this is the case or not. An indicator for back pressure is tasks 
> being stuck in buffer requests on the network stack. This means that they 
> have filled all their buffers with data, but the following tasks/network are 
> not consuming them fast enough.
> A simple way to measure back pressure is to sample running tasks and report 
> back pressure if they are stuck in the blocking buffers calls.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Scala DataStream accessors fix

[gumchum]

Github user gumchum commented on the pull request:

https://github.com/apache/flink/pull/1574#issuecomment-178783922
  
Looks good. It looks like collect is returning a Java iterator instead of a 
Scala one. Is there a reason for this given that the code is for Scala users?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3208] [gelly] rename vertex-centric ite...

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1514#issuecomment-178453684
  
Cool, I'll rebase and merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3208) Rename Gelly vertex-centric model to scatter-gather

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127920#comment-15127920
 ] 

ASF GitHub Bot commented on FLINK-3208:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1514#issuecomment-178453684
  
Cool, I'll rebase and merge.


> Rename Gelly vertex-centric model to scatter-gather
> ---
>
> Key: FLINK-3208
> URL: https://issues.apache.org/jira/browse/FLINK-3208
> Project: Flink
>  Issue Type: Sub-task
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>
> The idea is to have the following naming:
> - Pregel model: vertex-centric iteration
> - Spargel model: scatter-gather iteration
> - GSA model: as is
> Open to suggestions!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3316) Links to Gelly and FlinkML libraries on main site broken

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128374#comment-15128374
 ] 

Till Rohrmann commented on FLINK-3316:
--

In fact, all links in the first paragraph on the landing page are broken. I 
guess that the feature page was updated without updating the links accordingly.

> Links to Gelly and FlinkML libraries on main site broken
> 
>
> Key: FLINK-3316
> URL: https://issues.apache.org/jira/browse/FLINK-3316
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Theodore Vasiloudis
>Priority: Trivial
>  Labels: starter, trivial
>
> The landing page of flink.apache.org includes links to the Gelly and FlinkML 
> libraries under the text:
> {quote}
> Flink also bundles libraries for domain-specific use cases:
> 1.Machine Learning library, and
> 2.Gelly, a graph processing API and library.
> {quote}
> These point to anchor links in the Features page that seem to longer exist.
> I guess linking to the docs instead could be a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3316) Links to Gelly and FlinkML libraries on main site broken

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-3316:


Assignee: Till Rohrmann

> Links to Gelly and FlinkML libraries on main site broken
> 
>
> Key: FLINK-3316
> URL: https://issues.apache.org/jira/browse/FLINK-3316
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Theodore Vasiloudis
>Assignee: Till Rohrmann
>Priority: Trivial
>  Labels: starter, trivial
>
> The landing page of flink.apache.org includes links to the Gelly and FlinkML 
> libraries under the text:
> {quote}
> Flink also bundles libraries for domain-specific use cases:
> 1.Machine Learning library, and
> 2.Gelly, a graph processing API and library.
> {quote}
> These point to anchor links in the Features page that seem to longer exist.
> I guess linking to the docs instead could be a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3247) Kafka Connector unusable with quickstarts - shading issue

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3247.
---
Resolution: Fixed

Resolved in http://git-wip-us.apache.org/repos/asf/flink/commit/360f02b1

> Kafka Connector unusable with quickstarts - shading issue
> -
>
> Key: FLINK-3247
> URL: https://issues.apache.org/jira/browse/FLINK-3247
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The Kafka Connector now requires Curator, which is referenced as 
> {{flink-shaded-curator}}. The quickstarts make sure it is not packaged into 
> the jar file via exclusions.
> The curator classes are however only in relocated form in the flink-dist.jar 
> - relocated manually in the {{flink-runtime}} project. The connector can thus 
> not use find the Curator classes and fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3287.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

http://git-wip-us.apache.org/repos/asf/flink/commit/cb1a5ecb

> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.0.0
>
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3035) Redis as State Backend

[Subhobrata Dey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subhobrata Dey reassigned FLINK-3035:
-

Assignee: Subhobrata Dey

> Redis as State Backend
> --
>
> Key: FLINK-3035
> URL: https://issues.apache.org/jira/browse/FLINK-3035
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming
>Reporter: Matthias J. Sax
>Assignee: Subhobrata Dey
>Priority: Minor
>
> Add Redis as a state backend for distributed snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[Gabor Gevay (JIRA)]

Gabor Gevay created FLINK-3322:
--

 Summary: MemoryManager creates too much GC pressure with iterative 
jobs
 Key: FLINK-3322
 URL: https://issues.apache.org/jira/browse/FLINK-3322
 Project: Flink
  Issue Type: Bug
  Components: Distributed Runtime
Affects Versions: 1.0.0
Reporter: Gabor Gevay


When taskmanager.memory.preallocate is false (the default), released memory 
segments are not added to a pool, but the GC is expected to take care of them. 
This puts too much pressure on the GC with iterative jobs, where the operators 
reallocate all memory at every superstep.

See the following discussion on the mailing list:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html

Reproducing the issue:
https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
The class to start is malom.Solver. If you increase the memory given to the JVM 
from 1 to 50 GB, performance gradually degrades by more than 10 times. (It will 
generate some lookuptables to /tmp on first run for a few minutes.) (I think 
the slowdown might also depend somewhat on taskmanager.memory.fraction, because 
more unused non-managed memory results in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3321) TupleSerializerBase.getLength should know the length when all fields know it

[Gabor Gevay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabor Gevay updated FLINK-3321:
---
Labels: starter  (was: )

> TupleSerializerBase.getLength should know the length when all fields know it
> 
>
> Key: FLINK-3321
> URL: https://issues.apache.org/jira/browse/FLINK-3321
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API
>Reporter: Gabor Gevay
>Priority: Minor
>  Labels: starter
>
> TupleSerializerBase.getLength currently always returns -1, but it could 
> actually know the length, when all the field serializers know their lengths 
> (since no null can appear anywhere in Tuples, nor can a subclass of Tuple 
> with additional fields appear).
> (The serializer knowing the exact size has various performance benefits, for 
> example see FixedLengthRecordSorter, or 
> CompactingHashTable.getInitialTableSize.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3316) Links to Gelly and FlinkML libraries on main site broken

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-3316.
--
Resolution: Fixed

Fixed via ffe31bf53ca28a842d4a352a568f288251d56490

> Links to Gelly and FlinkML libraries on main site broken
> 
>
> Key: FLINK-3316
> URL: https://issues.apache.org/jira/browse/FLINK-3316
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Theodore Vasiloudis
>Assignee: Till Rohrmann
>Priority: Trivial
>  Labels: starter, trivial
>
> The landing page of flink.apache.org includes links to the Gelly and FlinkML 
> libraries under the text:
> {quote}
> Flink also bundles libraries for domain-specific use cases:
> 1.Machine Learning library, and
> 2.Gelly, a graph processing API and library.
> {quote}
> These point to anchor links in the Features page that seem to longer exist.
> I guess linking to the docs instead could be a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[Gabor Gevay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabor Gevay updated FLINK-3322:
---
Fix Version/s: 1.0.0

> MemoryManager creates too much GC pressure with iterative jobs
> --
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
> Fix For: 1.0.0
>
>
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3321) TupleSerializerBase.getLength should know the length when all fields know it

[Gabor Gevay (JIRA)]

Gabor Gevay created FLINK-3321:
--

 Summary: TupleSerializerBase.getLength should know the length when 
all fields know it
 Key: FLINK-3321
 URL: https://issues.apache.org/jira/browse/FLINK-3321
 Project: Flink
  Issue Type: Improvement
  Components: Java API
Reporter: Gabor Gevay
Priority: Minor


TupleSerializerBase.getLength currently always returns -1, but it could 
actually know the length, when all the field serializers know their lengths 
(since no null can appear anywhere in Tuples, nor can a subclass of Tuple with 
additional fields appear).

(The serializer knowing the exact size has various performance benefits, for 
example see FixedLengthRecordSorter, or 
CompactingHashTable.getInitialTableSize.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128328#comment-15128328
 ] 

ASF GitHub Bot commented on FLINK-3287:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1573


> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3287] [FLINK-3247] Fix curator shading ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1573


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3035) Redis as State Backend

[Subhobrata Dey (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128367#comment-15128367
 ] 

Subhobrata Dey commented on FLINK-3035:
---

Hello [~mjsax],

Yes, I'm interested in the issue & would assign myself to the ticket. Thanks.

> Redis as State Backend
> --
>
> Key: FLINK-3035
> URL: https://issues.apache.org/jira/browse/FLINK-3035
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming
>Reporter: Matthias J. Sax
>Priority: Minor
>
> Add Redis as a state backend for distributed snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3049] move Either type to org.apache.fl...

[vasia]

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1572

[FLINK-3049] move Either type to org.apache.flink.types

I went for option 2, i.e. creating the package in `flink-java`. Let me know 
if you think it makes more sense to move the type to `flink-core`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink flink-3049

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1572.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1572


commit 681ca600c209d5f7bfc6f5f27d3f5a53b8159d40
Author: vasia 
Date:   2016-02-02T09:41:56Z

[FLINK-3049] move Either type to org.apache.flink.types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127992#comment-15127992
 ] 

ASF GitHub Bot commented on FLINK-3049:
---

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1572

[FLINK-3049] move Either type to org.apache.flink.types

I went for option 2, i.e. creating the package in `flink-java`. Let me know 
if you think it makes more sense to move the type to `flink-core`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink flink-3049

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1572.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1572


commit 681ca600c209d5f7bfc6f5f27d3f5a53b8159d40
Author: vasia 
Date:   2016-02-02T09:41:56Z

[FLINK-3049] move Either type to org.apache.flink.types




> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2600) Failing ElasticsearchSinkITCase.testNodeClient test case

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127998#comment-15127998
 ] 

Robert Metzger commented on FLINK-2600:
---

One more: https://s3.amazonaws.com/archive.travis-ci.org/jobs/106246892/log.txt

> Failing ElasticsearchSinkITCase.testNodeClient test case
> 
>
> Key: FLINK-2600
> URL: https://issues.apache.org/jira/browse/FLINK-2600
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>Assignee: Aljoscha Krettek
>  Labels: test-stability
>
> I observed that the {{ElasticsearchSinkITCase.testNodeClient}} test case 
> fails on Travis. The stack trace is
> {code}
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:414)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.testingUtils.TestingJobManager$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManager.scala:285)
>   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:104)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: An error occured in ElasticsearchSink.
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink.close(ElasticsearchSink.java:307)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:40)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:75)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:243)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:185)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:581)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: IndexMissingException[[my-index] 
> missing]
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink$1.afterBulk(ElasticsearchSink.java:240)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:316)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:299)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:281)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:260)
>   at 
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:246)
>   at 
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink.invoke(ElasticsearchSink.java:286)
>   at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:37)
>   at 
> 

[jira] [Resolved] (FLINK-3208) Rename Gelly vertex-centric model to scatter-gather

[Vasia Kalavri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vasia Kalavri resolved FLINK-3208.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

> Rename Gelly vertex-centric model to scatter-gather
> ---
>
> Key: FLINK-3208
> URL: https://issues.apache.org/jira/browse/FLINK-3208
> Project: Flink
>  Issue Type: Sub-task
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> The idea is to have the following naming:
> - Pregel model: vertex-centric iteration
> - Spargel model: scatter-gather iteration
> - GSA model: as is
> Open to suggestions!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3208] [gelly] rename vertex-centric ite...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1514


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2678) DataSet API does not support multi-dimensional arrays as keys

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128035#comment-15128035
 ] 

ASF GitHub Bot commented on FLINK-2678:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1566#discussion_r51551151
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/GenericArrayComparator.java
 ---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils.base;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.core.memory.MemorySegment;
+
+import java.io.IOException;
+import java.lang.reflect.Array;
+import java.util.Arrays;
+
+public class GenericArrayComparator extends TypeComparator 
implements java.io.Serializable {
+
+   private static final long serialVersionUID = 1L;
+
+   private transient T[] reference;
+
+   protected final boolean ascendingComparison;
+
+   private final TypeSerializer serializer;
+
+   // For use by getComparators
+   @SuppressWarnings("rawtypes")
+   private final TypeComparator[] comparators = new TypeComparator[] 
{this};
+
+   public GenericArrayComparator(boolean ascending, TypeSerializer 
serializer) {
+   this.ascendingComparison = ascending;
+   this.serializer = serializer;
+   }
+
+   @Override
+   public void setReference(T[] reference) {
+   this.reference = reference;
+   }
+
+   @Override
+   public boolean equalToReference(T[] candidate) {
+   return compare(this.reference, candidate) == 0;
+   }
+
+   @Override
+   public int compareToReference(TypeComparator referencedComparator) 
{
+   int comp = compare(((GenericArrayComparator) 
referencedComparator).reference, reference);
+   return ascendingComparison ? comp : -comp;
+   }
+
+   @Override
+   public int compareSerialized(DataInputView firstSource, DataInputView 
secondSource) throws IOException {
+   T[] firstArray = serializer.deserialize(firstSource);
+   T[] secondArray = serializer.deserialize(secondSource);
+
+   int comp = compare(firstArray, secondArray);
+   return ascendingComparison ? comp : -comp;
+   }
+
+   @Override
+   public int extractKeys(Object record, Object[] target, int index) {
+   target[index] = record;
+   return 1;
+   }
+
+   @Override
+   public TypeComparator[] getFlatComparators() {
+   return comparators;
+   }
+
+   @Override
+   public boolean supportsNormalizedKey() {
+   return false;
+   }
+
+   @Override
+   public boolean supportsSerializationWithKeyNormalization() {
+   return false;
+   }
+
+   @Override
+   public int getNormalizeKeyLen() {
+   return 0;
+   }
+
+   @Override
+   public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public void putNormalizedKey(T[] record, MemorySegment target, int 
offset, int numBytes) {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public void writeWithKeyNormalization(T[] record, DataOutputView 
target) throws IOException {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public T[] readWithKeyDenormalization(T[] reuse, DataInputView source) 
throws IOException {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   

[GitHub] flink pull request: [FLINK-2678]DataSet API does not support multi...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1566#discussion_r51551151
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/GenericArrayComparator.java
 ---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils.base;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.core.memory.MemorySegment;
+
+import java.io.IOException;
+import java.lang.reflect.Array;
+import java.util.Arrays;
+
+public class GenericArrayComparator extends TypeComparator 
implements java.io.Serializable {
+
+   private static final long serialVersionUID = 1L;
+
+   private transient T[] reference;
+
+   protected final boolean ascendingComparison;
+
+   private final TypeSerializer serializer;
+
+   // For use by getComparators
+   @SuppressWarnings("rawtypes")
+   private final TypeComparator[] comparators = new TypeComparator[] 
{this};
+
+   public GenericArrayComparator(boolean ascending, TypeSerializer 
serializer) {
+   this.ascendingComparison = ascending;
+   this.serializer = serializer;
+   }
+
+   @Override
+   public void setReference(T[] reference) {
+   this.reference = reference;
+   }
+
+   @Override
+   public boolean equalToReference(T[] candidate) {
+   return compare(this.reference, candidate) == 0;
+   }
+
+   @Override
+   public int compareToReference(TypeComparator referencedComparator) 
{
+   int comp = compare(((GenericArrayComparator) 
referencedComparator).reference, reference);
+   return ascendingComparison ? comp : -comp;
+   }
+
+   @Override
+   public int compareSerialized(DataInputView firstSource, DataInputView 
secondSource) throws IOException {
+   T[] firstArray = serializer.deserialize(firstSource);
+   T[] secondArray = serializer.deserialize(secondSource);
+
+   int comp = compare(firstArray, secondArray);
+   return ascendingComparison ? comp : -comp;
+   }
+
+   @Override
+   public int extractKeys(Object record, Object[] target, int index) {
+   target[index] = record;
+   return 1;
+   }
+
+   @Override
+   public TypeComparator[] getFlatComparators() {
+   return comparators;
+   }
+
+   @Override
+   public boolean supportsNormalizedKey() {
+   return false;
+   }
+
+   @Override
+   public boolean supportsSerializationWithKeyNormalization() {
+   return false;
+   }
+
+   @Override
+   public int getNormalizeKeyLen() {
+   return 0;
+   }
+
+   @Override
+   public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public void putNormalizedKey(T[] record, MemorySegment target, int 
offset, int numBytes) {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public void writeWithKeyNormalization(T[] record, DataOutputView 
target) throws IOException {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public T[] readWithKeyDenormalization(T[] reuse, DataInputView source) 
throws IOException {
+   throw new UnsupportedOperationException();
+   }
+
+   @Override
+   public boolean invertNormalizedKey() {
+   return !ascendingComparison;
+   }
+
+   @Override
+   public int hash(T[] record) {
+   return Arrays.hashCode(record);
+   }
+
+   private int 

[jira] [Commented] (FLINK-3303) Move all non-batch specific classes in flink-java to flink-core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128043#comment-15128043
 ] 

ASF GitHub Bot commented on FLINK-3303:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1570#issuecomment-178504289
  
Merging this...


> Move all non-batch specific classes in flink-java to flink-core
> ---
>
> Key: FLINK-3303
> URL: https://issues.apache.org/jira/browse/FLINK-3303
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Currently, {{flink-java}} has a lot of classes that are also needed by the 
> streaming api and that are useful to the classes in {{flink-core}}.
> In particular, certain improvements to the state API are blocked by the fact 
> that certain classes are not in {{flink-core}}.
> I suggest to move classes from {{flink-java}} to {{flink-core}} after the 
> following pattern:
>   - {{flink-core}} will contain all classes that are common across the batch 
> and streaming API.
>   - {{flink-java}} will contain all batch API specific classes (we may 
> eventually even think about renaming it to {{flink-batch-java}}.
> Because {{flink-java}} references {{flink-core}}, this will not be a breaking 
> change. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3049] move Either type to org.apache.fl...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1572#issuecomment-178503870
  
Ah, I actually included this already in #1570



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128040#comment-15128040
 ] 

ASF GitHub Bot commented on FLINK-3049:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1572#issuecomment-178503870
  
Ah, I actually included this already in #1570



> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3034) Redis SInk Connector

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128045#comment-15128045
 ] 

Matthias J. Sax commented on FLINK-3034:


[~sbcd90], if the PR gets opened, please help to review.

> Redis SInk Connector
> 
>
> Key: FLINK-3034
> URL: https://issues.apache.org/jira/browse/FLINK-3034
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Matthias J. Sax
>Assignee: subhankar biswas
>Priority: Minor
>
> Flink does not provide a sink connector for Redis.
> See FLINK-3033



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3128] [flink-ml] Add Isotonic Regressio...

[f-sander]

Github user f-sander commented on the pull request:

https://github.com/apache/flink/pull/1565#issuecomment-178487703
  
Are the build failures related to us? I don't really understand how...

The first failure happens in oraclejd8 with hadoop 2.7.1:
```
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 351.368 sec 
<<< FAILURE! - in 
org.apache.flink.test.recovery.TaskManagerProcessFailureBatchRecoveryITCase

testTaskManagerProcessFailure[0](org.apache.flink.test.recovery.TaskManagerProcessFailureBatchRecoveryITCase)
  Time elapsed: 318.792 sec  <<< FAILURE!
java.lang.AssertionError: The program did not finish in time
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at 
org.apache.flink.test.recovery.AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure(AbstractTaskManagerProcessFailureRecoveryTest.java:212)
```
The second on in openjdk 7 with hadoop 1 appears to experience a deadlock 
(?):
```

==
Maven produced no output for 300 seconds.

==

==
The following Java processes are running (JPS)

==
2286 Launcher
77113 Jps
76276 surefirebooter4006285424712115006.jar

==
Printing stack trace of Java process 2286

==
```
After that only lots of lots of process stack traces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3128) Add Isotonic Regression To ML Library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128001#comment-15128001
 ] 

ASF GitHub Bot commented on FLINK-3128:
---

Github user f-sander commented on the pull request:

https://github.com/apache/flink/pull/1565#issuecomment-178487703
  
Are the build failures related to us? I don't really understand how...

The first failure happens in oraclejd8 with hadoop 2.7.1:
```
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 351.368 sec 
<<< FAILURE! - in 
org.apache.flink.test.recovery.TaskManagerProcessFailureBatchRecoveryITCase

testTaskManagerProcessFailure[0](org.apache.flink.test.recovery.TaskManagerProcessFailureBatchRecoveryITCase)
  Time elapsed: 318.792 sec  <<< FAILURE!
java.lang.AssertionError: The program did not finish in time
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at 
org.apache.flink.test.recovery.AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure(AbstractTaskManagerProcessFailureRecoveryTest.java:212)
```
The second on in openjdk 7 with hadoop 1 appears to experience a deadlock 
(?):
```

==
Maven produced no output for 300 seconds.

==

==
The following Java processes are running (JPS)

==
2286 Launcher
77113 Jps
76276 surefirebooter4006285424712115006.jar

==
Printing stack trace of Java process 2286

==
```
After that only lots of lots of process stack traces.


> Add Isotonic Regression To ML Library
> -
>
> Key: FLINK-3128
> URL: https://issues.apache.org/jira/browse/FLINK-3128
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Fridtjof Sander
>Assignee: Fridtjof Sander
>Priority: Minor
>
> Isotonic Regression fits a monotonically increasing function (also called 
> isotonic function) to a plane of datapoints.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3313:
-

 Summary: Kafka08ITCase.testOneSourceMultiplePartitions fails with 
EOFException
 Key: FLINK-3313
 URL: https://issues.apache.org/jira/browse/FLINK-3313
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Reporter: Robert Metzger


The deserialization has recently been modified: 
https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f

this is the log: 
https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3208) Rename Gelly vertex-centric model to scatter-gather

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128007#comment-15128007
 ] 

ASF GitHub Bot commented on FLINK-3208:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1514


> Rename Gelly vertex-centric model to scatter-gather
> ---
>
> Key: FLINK-3208
> URL: https://issues.apache.org/jira/browse/FLINK-3208
> Project: Flink
>  Issue Type: Sub-task
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>
> The idea is to have the following naming:
> - Pregel model: vertex-centric iteration
> - Spargel model: scatter-gather iteration
> - GSA model: as is
> Open to suggestions!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2736) Failing KafkaConsumerTestBase.runCancelingOnEmptyInputTest

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-2736.
---
Resolution: Won't Fix

The issue didn't occur in a while.

> Failing KafkaConsumerTestBase.runCancelingOnEmptyInputTest
> --
>
> Key: FLINK-2736
> URL: https://issues.apache.org/jira/browse/FLINK-2736
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Ufuk Celebi
>Assignee: Robert Metzger
>Priority: Minor
>  Labels: test-stability
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/81591058/log.txt
> https://flink.a.o.uce.east.s3.amazonaws.com/travis-artifacts/uce/flink/673/673.1.tar.gz
> ---
> {{KafkaConsumer}} fails with {{LeaderNotAvailableException}} in the 
> constructor. The job fails to submit and the test concurrently cancels it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3303] Move all non-batch specific class...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1570#issuecomment-178504289
  
Merging this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3035) Redis as State Backend

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128050#comment-15128050
 ] 

Matthias J. Sax commented on FLINK-3035:


[~sbcd90], maybe you are interested in this ticket ;)

> Redis as State Backend
> --
>
> Key: FLINK-3035
> URL: https://issues.apache.org/jira/browse/FLINK-3035
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming
>Reporter: Matthias J. Sax
>Priority: Minor
>
> Add Redis as a state backend for distributed snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3033) Redis Source Connector

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128048#comment-15128048
 ] 

Matthias J. Sax commented on FLINK-3033:


[~sbcd90], if the PR gets opened, please help to review.

> Redis Source Connector
> --
>
> Key: FLINK-3033
> URL: https://issues.apache.org/jira/browse/FLINK-3033
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Matthias J. Sax
>Assignee: subhankar biswas
>Priority: Minor
>
> Flink does not provide a source connector for Redis.
> See FLINK-3034



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2380) Allow to configure default FS for file inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128057#comment-15128057
 ] 

ASF GitHub Bot commented on FLINK-2380:
---

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1524#issuecomment-178507931
  
Could you explain what more tests do you have in mind? So far I am testing 
1) if the scheme provided in the configuration is used when one is not 
explicitly provided, 2) if an explicit scheme overrides the configuration one, 
and 3) if a scheme from the configuration overrides the default one.


> Allow to configure default FS for file inputs
> -
>
> Key: FLINK-2380
> URL: https://issues.apache.org/jira/browse/FLINK-2380
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9, 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Kostas
>Priority: Minor
>  Labels: starter
> Fix For: 1.0.0
>
>
> File inputs use "file://" as default prefix. A user asked to make this 
> configurable, e.g. "hdfs://" as default.
> (I'm not sure whether this is already possible or not. I will check and 
> either close or implement this for the user.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3207] [gelly] adds the vertex-centric i...

[vasia]

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1575

[FLINK-3207] [gelly] adds the vertex-centric iteration abstraction

As per the mailing list discussion, this PR adds the vertex-centric / 
pregel iteration abstraction to gelly. In this model, superstep computation 
happens inside a single UDF, which has access to received messages, can update 
the vertex state and can generate messages for other vertices. Documentation is 
also updated with a comparison among available iteration options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink gelly-pregel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1575


commit 638e754e6d288b6205e172cda79dc7c9ce682fbf
Author: vasia 
Date:   2016-02-02T11:13:51Z

[FLINK-3207] [gelly] add pregel iteration abstraction to gelly

commit 875a72b0546aa12bd5ad0f1f809c3633c249f825
Author: vasia 
Date:   2016-02-02T11:41:51Z

[FLINK-3207] [gelly] add a pregel SSSP example with combiner

commit 7f8f92b918a8ef7d3a530a77a3d8be5fa0758639
Author: vasia 
Date:   2016-02-02T14:00:44Z

[FLINK-3207] [gelly] add compiler and translation tests

Add the vertex-centric abstraction to the gelly guide
and a comparison among all iteration models.

commit 257afeee315765122d856411d3de49164dc99788
Author: vasia 
Date:   2016-02-02T15:43:04Z

[FLINK-3207] [gelly] add pregel methods to gelly-scala




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1577

[FLINK-3313] [kafka] Fix concurrent TypeInformationSerializationSchema 
usage in LegacyFetcher

The LegacyFetcher used the given KeyedDeserializationSchema across multiple 
threads even though
it is not thread-safe. This commit fixes the problem by cloning the 
KeyedDeserializationSchema
before giving it to the SimpleConsumerThread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixLegacyFetcher

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1577


commit 8b46eb1518a46edc518474f4a1bbc9425cc708d9
Author: Till Rohrmann 
Date:   2016-02-02T16:38:21Z

[FLINK-3313] [kafka] Fix TypeInformationSerializationSchema usage in 
LegacyFetcher

The LegacyFetcher used the given KeyedDeserializationSchema across multiple 
threads even though
it is not thread-safe. This commit fixes the problem by cloning the 
KeyedDeserializationSchema
before giving it to the SimpleConsumerThread.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Scala DataStream accessors fix

[cmonty]

Github user cmonty commented on the pull request:

https://github.com/apache/flink/pull/1574#issuecomment-178683549
  
The Scala interface looks good to me and gets rid of the need to call 
`getJavaStream` directly.

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178683735
  
Good catch & fix!
+1 to merge asap (too many tests are failing on this one right now)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128536#comment-15128536
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178683735
  
Good catch & fix!
+1 to merge asap (too many tests are failing on this one right now)


> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1577#discussion_r51597662
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/LegacyFetcher.java
 ---
@@ -199,8 +200,11 @@ public void close() {

FetchPartition[] partitions = 
partitionsList.toArray(new FetchPartition[partitionsList.size()]);
 
+   @SuppressWarnings("unchecked")
+   final KeyedDeserializationSchema clonedDeserializer 
= (KeyedDeserializationSchema)SerializationUtils.clone(deserializer);
--- End diff --

Good catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128549#comment-15128549
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178685691
  
Thanks for the review. Will merge it once Travis gives green light.


> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178685691
  
Thanks for the review. Will merge it once Travis gives green light.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128546#comment-15128546
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1577#discussion_r51597662
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/LegacyFetcher.java
 ---
@@ -199,8 +200,11 @@ public void close() {

FetchPartition[] partitions = 
partitionsList.toArray(new FetchPartition[partitionsList.size()]);
 
+   @SuppressWarnings("unchecked")
+   final KeyedDeserializationSchema clonedDeserializer 
= (KeyedDeserializationSchema)SerializationUtils.clone(deserializer);
--- End diff --

Good catch


> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3310) Add back pressure statistics to web frontend

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128559#comment-15128559
 ] 

Maximilian Michels commented on FLINK-3310:
---

I also created an issue some time ago: FLINK-3127

> Add back pressure statistics to web frontend
> 
>
> Key: FLINK-3310
> URL: https://issues.apache.org/jira/browse/FLINK-3310
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Minor
>
> When a task is receiving data at a higher rate than it can process, the task 
> is back pressuring preceding tasks. Currently, there is no way to tell 
> whether this is the case or not. An indicator for back pressure is tasks 
> being stuck in buffer requests on the network stack. This means that they 
> have filled all their buffers with data, but the following tasks/network are 
> not consuming them fast enough.
> A simple way to measure back pressure is to sample running tasks and report 
> back pressure if they are stuck in the blocking buffers calls.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128491#comment-15128491
 ] 

Till Rohrmann commented on FLINK-3313:
--

The problem ist that the {{TypeInformationSerializationSchema}} is used across 
multiple threads in the {{FlinkKafkaConsumer08}}. I propose to clone the 
serialization schema for each thread.

> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-3313:


Assignee: Till Rohrmann

> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128515#comment-15128515
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

Github user uce commented on a diff in the pull request:

https://github.com/apache/flink/pull/1577#discussion_r51595056
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/LegacyFetcher.java
 ---
@@ -199,8 +200,11 @@ public void close() {

FetchPartition[] partitions = 
partitionsList.toArray(new FetchPartition[partitionsList.size()]);
 
+   @SuppressWarnings("unchecked")
+   final KeyedDeserializationSchema clonedDeserializer 
= (KeyedDeserializationSchema)SerializationUtils.clone(deserializer);
--- End diff --

whitespace after cast missing


> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128513#comment-15128513
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1577

[FLINK-3313] [kafka] Fix concurrent TypeInformationSerializationSchema 
usage in LegacyFetcher

The LegacyFetcher used the given KeyedDeserializationSchema across multiple 
threads even though
it is not thread-safe. This commit fixes the problem by cloning the 
KeyedDeserializationSchema
before giving it to the SimpleConsumerThread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixLegacyFetcher

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1577


commit 8b46eb1518a46edc518474f4a1bbc9425cc708d9
Author: Till Rohrmann 
Date:   2016-02-02T16:38:21Z

[FLINK-3313] [kafka] Fix TypeInformationSerializationSchema usage in 
LegacyFetcher

The LegacyFetcher used the given KeyedDeserializationSchema across multiple 
threads even though
it is not thread-safe. This commit fixes the problem by cloning the 
KeyedDeserializationSchema
before giving it to the SimpleConsumerThread.




> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[uce]

Github user uce commented on a diff in the pull request:

https://github.com/apache/flink/pull/1577#discussion_r51595056
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/LegacyFetcher.java
 ---
@@ -199,8 +200,11 @@ public void close() {

FetchPartition[] partitions = 
partitionsList.toArray(new FetchPartition[partitionsList.size()]);
 
+   @SuppressWarnings("unchecked")
+   final KeyedDeserializationSchema clonedDeserializer 
= (KeyedDeserializationSchema)SerializationUtils.clone(deserializer);
--- End diff --

whitespace after cast missing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3313] [kafka] Fix concurrent TypeInform...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178680229
  
Good catch! :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128072#comment-15128072
 ] 

Robert Metzger commented on FLINK-3287:
---

The user confirmed that shading curator into the kafka-connector jars resolved 
the issue.

> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3216] [FLINK-3217] [cep] Initial versio...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1557#issuecomment-178515178
  
The Scala problem seems to be related to missing suffixes in the pom file. 
I fixed it. Once travis gives green light, I'll merge the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3216) Define pattern specification

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128080#comment-15128080
 ] 

ASF GitHub Bot commented on FLINK-3216:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1557#issuecomment-178515178
  
The Scala problem seems to be related to missing suffixes in the pom file. 
I fixed it. Once travis gives green light, I'll merge the PR.


> Define pattern specification
> 
>
> Key: FLINK-3216
> URL: https://issues.apache.org/jira/browse/FLINK-3216
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Till Rohrmann
>
> In order to detect event patterns we first have to define the pattern. This 
> issue tracks the progress of implementing a user facing API to define event 
> patterns. 
> Patterns should support the following operations
> * next(): The given event has to follow directly after the preceding event
> followedBy(): The given event has to follow the preceding event. There might 
> occur other events in-between
> * every(): In a follow-by relationship a starting event can be matched with 
> multiple successive events. Consider the pattern a → b where → denotes the 
> follow-by relationship. The event sequence a, b, b can be matched as a, b or 
> a, (b), b where the first b is left out. The essential question is whether a 
> is allowed to match multiple times or only the first time. The method every 
> specifies exactly that. Every events in a pattern can match with multiple 
> successive events. This makes only sense in a follow-by relationship, though.
> * followedByEvery(): Similar to followedBy just that the specified element 
> can be matched with multiple successive events
> * or(): Alternative event which can be matched instead of the original event: 
> every(“e1”).where().or(“e2”).where()
> * within(): Defines a time interval in which the pattern has to be completed, 
> otherwise an incomplete pattern can be emitted (timeout case)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3287] [FLINK-3247] Fix curator shading ...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1573

[FLINK-3287][FLINK-3247] Fix curator shading for Kafka connector

Some maven versions are not properly evaluating star * excludes. Therefore, 
I've manually listed our shaded artifacts in the quickstart pom.

Also, a user reported a version mismatch with curator on HDP. As a fix, I'm 
shading curator into the kafka connector.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3287

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1573


commit 02e33aaeb357d9a3ab6a282a0aabb87f2ac96032
Author: Robert Metzger 
Date:   2016-01-28T14:16:16Z

[FLINK-3287] Shade Curator dependency into flink-connector-kafka-0.8

commit c155758379906590d0b19412303cbdfcc0825d5c
Author: Robert Metzger 
Date:   2016-02-02T11:04:26Z

[FLINK-3247] Remove * exclude from quickstarts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3315) Fix Slot Sharing in Streaming API

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-3315:
---

 Summary: Fix Slot Sharing in Streaming API
 Key: FLINK-3315
 URL: https://issues.apache.org/jira/browse/FLINK-3315
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 1.0.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


Right now, the slot sharing/resource group logic is a bit "nebulous". The slot 
sharing group that operators are put in depends on the order in which 
operations are created. For example, in this case:

{code}
Source a = env.source()
Source b = env.source()

a.map().startNewResourceGroup().sink() 
b.map().sink()
{code}

We end up with two resource groups:
- group 1: source a
- group 2: map(), sink(), source b, map(), sink()

The reason is that the slot sharing id is incremented when transforming the 
{{startNewResouceGroup()}} call and all operators that are transformed 
afterwards in graph traversal get that new slot sharing id.

(There is also {{isolateResources()}} which can be used to isolate an operator.)

What I propose is to remove {{startNewResourceGroup()}} and 
{{isolateResouces()}} and replace it with {{slotSharingGroup(String)}}. By 
default, operations would be in slot sharing group "default". This allows very 
fine grained control over what operators end up in which slot sharing group. 
For example, I could have this topology:

{code}
Source a = env.source().slotSharingGroup("sources")
Source b = env.source().slotSharingGroup("sources")

a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks") 
b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks")
{code}

Which would isolate the lightweight sources and sinks in a group and put heavy 
operations inside their own slot groups.

This is a bit more low level than the previous API and requires more calls than 
a simple {{startNewResourceGroup()}} but I think not many people would use this 
feature and this design makes it very clear what operations end up in the same 
group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51556295
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -237,53 +236,26 @@ public ExecutionConfig setParallelism(int 
parallelism) {
}
 
/**
-* Gets the number of times the system will try to re-execute failed 
tasks. A value
-* of {@code -1} indicates that the system default value (as defined in 
the configuration)
-* should be used.
+* Sets the restart strategy configuration which defines which restart 
strategy shall be used
+* for the execution graph of the corresponding job.
--- End diff --

Agreed. Good point. I've simplified the the description and added a code 
example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128092#comment-15128092
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51556295
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -237,53 +236,26 @@ public ExecutionConfig setParallelism(int 
parallelism) {
}
 
/**
-* Gets the number of times the system will try to re-execute failed 
tasks. A value
-* of {@code -1} indicates that the system default value (as defined in 
the configuration)
-* should be used.
+* Sets the restart strategy configuration which defines which restart 
strategy shall be used
+* for the execution graph of the corresponding job.
--- End diff --

Agreed. Good point. I've simplified the the description and added a code 
example.


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3311) Add a connector for streaming data into Cassandra

[Andrea Sella (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128123#comment-15128123
 ] 

Andrea Sella commented on FLINK-3311:
-

Hi,
I am already working on to cassandra-connector into a private branch and I am 
using the java drivers. After some internal tests I will glad to submit my 
solution to the community, I need just a couple of days to finish it. 

Andrea

> Add a connector for streaming data into Cassandra
> -
>
> Key: FLINK-3311
> URL: https://issues.apache.org/jira/browse/FLINK-3311
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> We had users in the past asking for a Flink+Cassandra integration.
> It seems that there is a well-developed java client for connecting into 
> Cassandra: https://github.com/datastax/java-driver (ASL 2.0)
> There are also tutorials out there on how to start a local cassandra instance 
> (for the tests): 
> http://prettyprint.me/prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/index.html
> For the data types, I think we should support TupleX types, and map standard 
> java types to the respective cassandra types.
> In addition, it seems that there is a object mapper from datastax to store 
> POJOs in Cassandra (there are annotations for defining the primary key and 
> types)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128066#comment-15128066
 ] 

ASF GitHub Bot commented on FLINK-3049:
---

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1572


> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128065#comment-15128065
 ] 

ASF GitHub Bot commented on FLINK-3049:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1572#issuecomment-178509129
  
Then I'll close it :)


> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3049] move Either type to org.apache.fl...

[vasia]

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1572


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3314) Early cancel calls can cause Tasks to not cancel properly

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3314:
---

 Summary: Early cancel calls can cause Tasks to not cancel properly
 Key: FLINK-3314
 URL: https://issues.apache.org/jira/browse/FLINK-3314
 Project: Flink
  Issue Type: Bug
  Components: Local Runtime
Affects Versions: 1.0.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Priority: Blocker
 Fix For: 1.0.0


When a task receives the "cancel()" call before the operators are properly 
instantiated, it can be that the operator never receives a cancel call.

In certain cases, this causes the operator to hang.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51556358
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/restart/FixedDelayRestartStrategy.java
 ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.executiongraph.restart;
+
+import com.google.common.base.Preconditions;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.executiongraph.ExecutionGraph;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.duration.Duration;
+
+import java.util.concurrent.Callable;
+
+import static akka.dispatch.Futures.future;
+
+/**
+ * Restart strategy which tries to restart the given {@link 
ExecutionGraph} a fixed number of times
+ * with a fixed time delay in between.
+ */
+public class FixedDelayRestartStrategy implements RestartStrategy {
+   private static final Logger LOG = 
LoggerFactory.getLogger(FixedDelayRestartStrategy.class);
+
+
+   private final int maxNumberRestartAttempts;
+   private final long delayBetweenRestartAttempts;
+   private int currentRestartAttempt;
+
+   public FixedDelayRestartStrategy(
+   int maxNumberRestartAttempts,
+   long delayBetweenRestartAttempts) {
+
+   Preconditions.checkArgument(maxNumberRestartAttempts >= 0, 
"Maximum number of restart attempts must be positive.");
+   Preconditions.checkArgument(delayBetweenRestartAttempts >= 0, 
"Delay between restart attempts must be positive");
+
+   this.maxNumberRestartAttempts = maxNumberRestartAttempts;
+   this.delayBetweenRestartAttempts = delayBetweenRestartAttempts;
+   currentRestartAttempt = 0;
+   }
+
+   @Override
+   public boolean canRestart() {
+   return currentRestartAttempt < maxNumberRestartAttempts;
+   }
+
+   @Override
+   public void restart(final ExecutionGraph executionGraph) {
+   currentRestartAttempt++;
+
+   future(new Callable() {
+   @Override
+   public Object call() throws Exception {
+   try {
+   LOG.info("Delaying retry of job 
execution for {} ms ...", delayBetweenRestartAttempts);
+   // do the delay
+   
Thread.sleep(delayBetweenRestartAttempts);
+   }
+   catch(InterruptedException e){
--- End diff --

good catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128093#comment-15128093
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51556358
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/restart/FixedDelayRestartStrategy.java
 ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.executiongraph.restart;
+
+import com.google.common.base.Preconditions;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.executiongraph.ExecutionGraph;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.duration.Duration;
+
+import java.util.concurrent.Callable;
+
+import static akka.dispatch.Futures.future;
+
+/**
+ * Restart strategy which tries to restart the given {@link 
ExecutionGraph} a fixed number of times
+ * with a fixed time delay in between.
+ */
+public class FixedDelayRestartStrategy implements RestartStrategy {
+   private static final Logger LOG = 
LoggerFactory.getLogger(FixedDelayRestartStrategy.class);
+
+
+   private final int maxNumberRestartAttempts;
+   private final long delayBetweenRestartAttempts;
+   private int currentRestartAttempt;
+
+   public FixedDelayRestartStrategy(
+   int maxNumberRestartAttempts,
+   long delayBetweenRestartAttempts) {
+
+   Preconditions.checkArgument(maxNumberRestartAttempts >= 0, 
"Maximum number of restart attempts must be positive.");
+   Preconditions.checkArgument(delayBetweenRestartAttempts >= 0, 
"Delay between restart attempts must be positive");
+
+   this.maxNumberRestartAttempts = maxNumberRestartAttempts;
+   this.delayBetweenRestartAttempts = delayBetweenRestartAttempts;
+   currentRestartAttempt = 0;
+   }
+
+   @Override
+   public boolean canRestart() {
+   return currentRestartAttempt < maxNumberRestartAttempts;
+   }
+
+   @Override
+   public void restart(final ExecutionGraph executionGraph) {
+   currentRestartAttempt++;
+
+   future(new Callable() {
+   @Override
+   public Object call() throws Exception {
+   try {
+   LOG.info("Delaying retry of job 
execution for {} ms ...", delayBetweenRestartAttempts);
+   // do the delay
+   
Thread.sleep(delayBetweenRestartAttempts);
+   }
+   catch(InterruptedException e){
--- End diff --

good catch


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed 

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51557132
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/restartstrategy/RestartStrategies.java
 ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.restartstrategy;
+
+import java.io.Serializable;
+
+/**
+ * This class defines methods to generate RestartStrategyConfigurations. 
These configurations are
+ * used to create RestartStrategies at runtime.
+ *
+ * The RestartStrategyConfigurations are used to decouple the core module 
from the runtime module.
+ */
+public class RestartStrategies {
+
+   /**
+* Generates NoRestartStrategyConfiguration
+*
+* @return NoRestartStrategyConfiguration
+*/
+   public static RestartStrategyConfiguration noRestart() {
+   return new NoRestartStrategyConfiguration();
+   }
+
+   /**
+* Generates a FixedDelayRestartStrategyConfiguration.
+*
+* @param restartAttempts Number of restart attempts for the 
FixedDelayRestartStrategy
+* @param delayBetweenAttempts Delay in-between restart attempts for 
the FixedDelayRestartStrategy
+* @return FixedDelayRestartStrategy
+*/
+   public static RestartStrategyConfiguration fixedDelayRestart(
+   int restartAttempts,
+   long delayBetweenAttempts) {
+
+   return new 
FixedDelayRestartStrategyConfiguration(restartAttempts, delayBetweenAttempts);
+   }
+
+   public abstract static class RestartStrategyConfiguration implements 
Serializable {
+   private static final long serialVersionUID = 
6285853591578313960L;
+
+   private RestartStrategyConfiguration() {}
+
+   /**
+* Returns a description which is shown in the web interface
+* @return
--- End diff --

Added them


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: FLINK-2380: allow to specify the default files...

[kl0u]

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1524#issuecomment-178507931
  
Could you explain what more tests do you have in mind? So far I am testing 
1) if the scheme provided in the configuration is used when one is not 
explicitly provided, 2) if an explicit scheme overrides the configuration one, 
and 3) if a scheme from the configuration overrides the default one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3049] move Either type to org.apache.fl...

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1572#issuecomment-178509129
  
Then I'll close it :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128086#comment-15128086
 ] 

ASF GitHub Bot commented on FLINK-3287:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1573

[FLINK-3287][FLINK-3247] Fix curator shading for Kafka connector

Some maven versions are not properly evaluating star * excludes. Therefore, 
I've manually listed our shaded artifacts in the quickstart pom.

Also, a user reported a version mismatch with curator on HDP. As a fix, I'm 
shading curator into the kafka connector.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3287

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1573


commit 02e33aaeb357d9a3ab6a282a0aabb87f2ac96032
Author: Robert Metzger 
Date:   2016-01-28T14:16:16Z

[FLINK-3287] Shade Curator dependency into flink-connector-kafka-0.8

commit c155758379906590d0b19412303cbdfcc0825d5c
Author: Robert Metzger 
Date:   2016-02-02T11:04:26Z

[FLINK-3247] Remove * exclude from quickstarts




> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3287] [FLINK-3247] Fix curator shading ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1573#issuecomment-178518689
  
Thanks for fixing this, this was a critical issue.

+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128091#comment-15128091
 ] 

ASF GitHub Bot commented on FLINK-3287:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1573#issuecomment-178518689
  
Thanks for fixing this, this was a critical issue.

+1 to merge


> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51557088
  
--- Diff: docs/apis/fault_tolerance.md ---
@@ -193,73 +193,169 @@ state updates) of Flink coupled with bundled sinks:
 
 [Back to top](#top)
 
+Restart Strategies
+--
 
-Batch Processing Fault Tolerance (DataSet API)
---
+Flink supports different restart strategies which control how the jobs are 
restarted in case of a failure.
+The cluster can be started with a default restart strategy which is always 
used when no job specific restart strategy has been defined.
+In case that the job is submitted with a restart strategy, this strategy 
overrides the cluster's default setting.
--- End diff --

Yes, definitely


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128108#comment-15128108
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51557088
  
--- Diff: docs/apis/fault_tolerance.md ---
@@ -193,73 +193,169 @@ state updates) of Flink coupled with bundled sinks:
 
 [Back to top](#top)
 
+Restart Strategies
+--
 
-Batch Processing Fault Tolerance (DataSet API)
---
+Flink supports different restart strategies which control how the jobs are 
restarted in case of a failure.
+The cluster can be started with a default restart strategy which is always 
used when no job specific restart strategy has been defined.
+In case that the job is submitted with a restart strategy, this strategy 
overrides the cluster's default setting.
--- End diff --

Yes, definitely


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128109#comment-15128109
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1470#discussion_r51557132
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/restartstrategy/RestartStrategies.java
 ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.restartstrategy;
+
+import java.io.Serializable;
+
+/**
+ * This class defines methods to generate RestartStrategyConfigurations. 
These configurations are
+ * used to create RestartStrategies at runtime.
+ *
+ * The RestartStrategyConfigurations are used to decouple the core module 
from the runtime module.
+ */
+public class RestartStrategies {
+
+   /**
+* Generates NoRestartStrategyConfiguration
+*
+* @return NoRestartStrategyConfiguration
+*/
+   public static RestartStrategyConfiguration noRestart() {
+   return new NoRestartStrategyConfiguration();
+   }
+
+   /**
+* Generates a FixedDelayRestartStrategyConfiguration.
+*
+* @param restartAttempts Number of restart attempts for the 
FixedDelayRestartStrategy
+* @param delayBetweenAttempts Delay in-between restart attempts for 
the FixedDelayRestartStrategy
+* @return FixedDelayRestartStrategy
+*/
+   public static RestartStrategyConfiguration fixedDelayRestart(
+   int restartAttempts,
+   long delayBetweenAttempts) {
+
+   return new 
FixedDelayRestartStrategyConfiguration(restartAttempts, delayBetweenAttempts);
+   }
+
+   public abstract static class RestartStrategyConfiguration implements 
Serializable {
+   private static final long serialVersionUID = 
6285853591578313960L;
+
+   private RestartStrategyConfiguration() {}
+
+   /**
+* Returns a description which is shown in the web interface
+* @return
--- End diff --

Added them


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3287] [FLINK-3247] Fix curator shading ...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1573#issuecomment-178527480
  
There is one issue I need to figure out, then, I'll merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3287) Flink Kafka Consumer fails due to Curator version conflict

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128133#comment-15128133
 ] 

ASF GitHub Bot commented on FLINK-3287:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1573#issuecomment-178527480
  
There is one issue I need to figure out, then, I'll merge it.


> Flink Kafka Consumer fails due to Curator version conflict
> --
>
> Key: FLINK-3287
> URL: https://issues.apache.org/jira/browse/FLINK-3287
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.0
> Environment: HDP version 2.2.9.0-3393
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> {code}
> 14:32:38,542 INFO  org.apache.flink.yarn.YarnJobManager   
>- Status of job 8eb92c1e3a1c050ecaccd50c6298ac7a (Flink Streaming Job) 
> changed to FAILING.
> java.lang.NoSuchMethodError: 
> org.apache.curator.utils.ZKPaths.fixForNamespace(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/String;
> at 
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at 
> org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87)
> at 
> org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:457)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.getOffsetFromZooKeeper(ZookeeperOffsetHandler.java:122)
> at 
> org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.seekFetcherToInitialOffsets(ZookeeperOffsetHandler.java:90)
> at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.open(FlinkKafkaConsumer.java:401)
> at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:567)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This flink snapshot version was built from master commit 
> c7ada8d785087e0209071a8219ff841006b96639



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3291) Object reuse bug in MergeIterator.HeadStream.nextHead

[Gabor Gevay (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128243#comment-15128243
 ] 

Gabor Gevay commented on FLINK-3291:


> `ReduceDriver.run` owns two objects but is not tracking the object returned 
> in `while ((value = input.next(reuse2)) != null) {`

The problem is more complicated than that: This can't be done with just 1 or 2 
reuse objects, each of the head streams need to own objects themselves.

> Object reuse bug in MergeIterator.HeadStream.nextHead
> -
>
> Key: FLINK-3291
> URL: https://issues.apache.org/jira/browse/FLINK-3291
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: Gabor Gevay
>Priority: Critical
>
> MergeIterator.HeadStream.nextHead saves a reference into `this.head` of the 
> `reuse` object that it got as an argument. This object might be modified 
> later by the caller.
> This actually happens when ReduceDriver.run calls input.next (which will 
> actually be MergeIterator.next(E reuse)) in the inner while loop of the 
> objectReuseEnabled branch, and that calls top.nextHead with the reference 
> that it got from ReduceDriver, which erroneously saves the reference, and 
> then ReduceDriver later uses that same object for doing the reduce.
> Another way in which this fails is when MergeIterator.next(E reuse) gives 
> `reuse` to different `top`s in different calls, and then the heads end up 
> being the same object.
> You can observe the latter situation in action by running ReducePerformance 
> here:
> https://github.com/ggevay/flink/tree/merge-iterator-object-reuse-bug
> Set memory to -Xmx200m (so that the MergeIterator actually has merging to 
> do), put a breakpoint at the beginning of MergeIterator.next(reuse), and then 
> watch `reuse`, and the heads of the first two elements of `this.heap` in the 
> debugger. They will get to be the same object after hitting continue about 6 
> times.
> You can also look at the count that is printed at the end, which shouldn't be 
> larger than the key range. Also, if you look into the output file 
> /tmp/xxxobjectreusebug, for example the key 77 appears twice.
> The good news is that I think I can see an easy fix that doesn't affect 
> performance: MergeIterator.HeadStream could have a reuse object of its own as 
> a member, and give that to iterator.next in nextHead(E reuse). And then we 
> wouldn't need the overload of nextHead that has the reuse parameter, and 
> MergeIterator.next(E reuse) could just call its other overload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Scala DataStream accessors fix

[mbalassi]

GitHub user mbalassi opened a pull request:

https://github.com/apache/flink/pull/1574

Scala DataStream accessors fix

The PR contains two orthogonal approaches of fixing the access to 
`DataStreamUtils.collect` for a scala environment. One or both of the 
approaches can be merged.

The first simply reexposes access to the underlying java `DataStream`, 
while the second effectively adds a scala API for `DataStreamUtils`.

I am in favor of adding both. Usage of the latter looks as follows:

```scala
import org.apache.flink.streaming.api.scala._
import org.apache.flink.contrib.streaming.scala.utils._

object ScalaStreamCollect {

  def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment

val it = env.generateSequence(0L, 10L)
  .collect()

while (it.hasNext){
  print(it.next())
}
  }
}
``` 

At least the first commit should be merged asap as the fix was requested on 
the user mailing list. [1]

[1] 
https://mail-archives.apache.org/mod_mbox/flink-user/201602.mbox/%3CCAFqo6nQ24dtExjPOX%3DrSuWSww8skCH23Q8i7CJ3Ef5LYhDj2pA%40mail.gmail.com%3E

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mbalassi/flink scala-env-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1574


commit ff728a7de3b83d9eb97a065744f0eaa5464c23ab
Author: Márton Balassi 
Date:   2016-02-02T13:10:47Z

[streaming] [scala] Revert removing getJavaStream() from DataStream

commit 1215c0d864e0fd2824f70b303fcac3d6a6294a6d
Author: Márton Balassi 
Date:   2016-02-02T13:18:05Z

[streaming] [scala] Scala wrapper for DataStreamUtils




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3317) Add timeout handler to CEP operator

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-3317:


 Summary: Add timeout handler to CEP operator
 Key: FLINK-3317
 URL: https://issues.apache.org/jira/browse/FLINK-3317
 Project: Flink
  Issue Type: Improvement
  Components: CEP
Affects Versions: 1.0.0
Reporter: Till Rohrmann
Priority: Minor


Currently, event sequences which exceed the defined pattern timeout will be 
discarded. However, in some cases the user might be interested in getting to 
know when such a timeout occurred to return a default value for these event 
sequences.

Thus, the pattern API should be extended to be able to define a timeout 
handler. Furthermore, the {{NFA}} has to be extended to also return the 
discarded event sequences. The {{CEPOperator}} would then call for every 
discarded event sequence the timeout handler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3318) Add support for quantifiers to CEP's pattern API

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-3318:


 Summary: Add support for quantifiers to CEP's pattern API
 Key: FLINK-3318
 URL: https://issues.apache.org/jira/browse/FLINK-3318
 Project: Flink
  Issue Type: Improvement
  Components: CEP
Affects Versions: 1.0.0
Reporter: Till Rohrmann
Priority: Minor


It would be a good addition to extend the pattern API to support quantifiers 
known from regular expressions (e.g. Kleene star, ?, +, or count bounds). This 
would considerably enrich the set of supported patterns.

Implementing the count bounds could be done by unrolling the pattern state. In 
order to support the Kleene star operator, the {{NFACompiler}} has to be 
extended to insert epsilon-transition between a Kleene start state and the 
succeeding pattern state. In order to support {{?}}, one could insert two paths 
from the preceding state, one which accepts the event and another which 
directly goes into the next pattern state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3310) Add back pressure statistics to web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128602#comment-15128602
 ] 

ASF GitHub Bot commented on FLINK-3310:
---

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1578

[FLINK-3310] [runtime, runtime-web] Add back pressure statistics

This PR introduces back pressure statistics to the web frontend.

The idea is that tasks, which are back pressured by the network or slow 
upstream tasks are blocked on  buffer requests to the network stack. An example 
stack trace for these calls looks like this:
```
java.lang.Object.wait(Native Method)
o.a.f.[...].LocalBufferPool.requestBuffer(LocalBufferPool.java:163)
o.a.f.[...].LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:133) 
<--- BLOCKING
...
```

This PR introduces a mechanism to gather a sample of stack traces of 
running tasks. Depending on the number of stack traces stuck in the above 
calls, the task is assumed to experience back pressure. Depending on the ratio, 
back pressure is categorized into classes `OK` (<= 0.10) , `LOW` (<= 0.5), and 
`HIGH` (else).

The user can trigger the sampling via the web frontend. Results are cached 
for a certain amount of time (`60s`) before triggering further stack trace 
samples in order to not flood the task managers.

## Questions

What's your opinion on the following defaults:
- Categories `OK` (<= 0.10) , `LOW` (<= 0.5), and `HIGH` (else).
- `100` samples with `50 ms` delay between samples.
- `60s` default refreshing interval before web front end triggers new 
samples.

Furthermore, do you think we should allow to disable these features?

In future versions of Flink, I think that we need to start thinking about 
splitting the ExecutionGraph up a little bit. We need something like the old 
`ManagementGraph` in order to stop overloading the `ExecutionGraph` with 
non-core stuff.

## Screenshots

### Back pressure tabs

![screen shot 2016-02-02 at 17 57 
05](https://cloud.githubusercontent.com/assets/1756620/12757297/c378bcce-c9d7-11e5-8799-c4a44fe12b54.png)

### First click, sampling in progress

![screen shot 2016-02-02 at 17 57 
11](https://cloud.githubusercontent.com/assets/1756620/12757304/c8ddef9a-c9d7-11e5-86bf-9849c5be9c09.png)

### Sampling done

![screen shot 2016-02-02 at 17 57 
19](https://cloud.githubusercontent.com/assets/1756620/12757310/ccbce62a-c9d7-11e5-8ea4-c9c893c93478.png)

### Show sub tasks

![screen shot 2016-02-02 at 17 57 
24](https://cloud.githubusercontent.com/assets/1756620/12757354/fbc1d278-c9d7-11e5-8bd7-c6de60046a9e.png)

### Sample is refreshed

![screen shot 2016-02-02 at 17 58 
18](https://cloud.githubusercontent.com/assets/1756620/12757364/069cfad8-c9d8-11e5-8287-daf340df8b93.png)

![screen shot 2016-02-02 at 17 58 
28](https://cloud.githubusercontent.com/assets/1756620/12757376/0d1a46fe-c9d8-11e5-9968-46a0e72811a9.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink backpressure-sampling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1578


commit 7641159ea2a43f95e653e204c8112aede2971f14
Author: Ufuk Celebi 
Date:   2016-02-01T16:28:34Z

[hotfix] [runtime] Create TestingJobManager in TestingUtils

commit 91d6f89d8c029d35f654e92e981043a0ea906a99
Author: Ufuk Celebi 
Date:   2016-02-01T19:39:44Z

[hotfix] [docs] Add config docs about checkpoint stats

commit 6e0a99be04d3b54601f616e2be20fed20d244bc8
Author: Ufuk Celebi 
Date:   2016-02-01T19:51:23Z

[FLINK-3310] [runtime] Add support to sample stack traces of running tasks

commit 37b3f95882500cbf2100170f3f065110fd5f8817
Author: Ufuk Celebi 
Date:   2016-02-01T19:57:48Z

[FLINK-3310] [runtime-web] Add back pressure statistics to web monitor 
(backend)

commit 1d2e8694a3abf386cb68d5247b2840047824bf39
Author: Ufuk Celebi 
Date:   2016-02-01T20:01:26Z

[FLINK-3310] [runtime-web] Add back pressure statistics to web dashboard 
(frontend)




> Add back pressure statistics to web frontend
> 
>
> Key: FLINK-3310
> URL: https://issues.apache.org/jira/browse/FLINK-3310
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Minor
>
> When a task is receiving data at a higher rate than it 

[GitHub] flink pull request: [FLINK-3310] [runtime, runtime-web] Add back p...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/1578

[FLINK-3310] [runtime, runtime-web] Add back pressure statistics

This PR introduces back pressure statistics to the web frontend.

The idea is that tasks, which are back pressured by the network or slow 
upstream tasks are blocked on  buffer requests to the network stack. An example 
stack trace for these calls looks like this:
```
java.lang.Object.wait(Native Method)
o.a.f.[...].LocalBufferPool.requestBuffer(LocalBufferPool.java:163)
o.a.f.[...].LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:133) 
<--- BLOCKING
...
```

This PR introduces a mechanism to gather a sample of stack traces of 
running tasks. Depending on the number of stack traces stuck in the above 
calls, the task is assumed to experience back pressure. Depending on the ratio, 
back pressure is categorized into classes `OK` (<= 0.10) , `LOW` (<= 0.5), and 
`HIGH` (else).

The user can trigger the sampling via the web frontend. Results are cached 
for a certain amount of time (`60s`) before triggering further stack trace 
samples in order to not flood the task managers.

## Questions

What's your opinion on the following defaults:
- Categories `OK` (<= 0.10) , `LOW` (<= 0.5), and `HIGH` (else).
- `100` samples with `50 ms` delay between samples.
- `60s` default refreshing interval before web front end triggers new 
samples.

Furthermore, do you think we should allow to disable these features?

In future versions of Flink, I think that we need to start thinking about 
splitting the ExecutionGraph up a little bit. We need something like the old 
`ManagementGraph` in order to stop overloading the `ExecutionGraph` with 
non-core stuff.

## Screenshots

### Back pressure tabs

![screen shot 2016-02-02 at 17 57 
05](https://cloud.githubusercontent.com/assets/1756620/12757297/c378bcce-c9d7-11e5-8799-c4a44fe12b54.png)

### First click, sampling in progress

![screen shot 2016-02-02 at 17 57 
11](https://cloud.githubusercontent.com/assets/1756620/12757304/c8ddef9a-c9d7-11e5-86bf-9849c5be9c09.png)

### Sampling done

![screen shot 2016-02-02 at 17 57 
19](https://cloud.githubusercontent.com/assets/1756620/12757310/ccbce62a-c9d7-11e5-8ea4-c9c893c93478.png)

### Show sub tasks

![screen shot 2016-02-02 at 17 57 
24](https://cloud.githubusercontent.com/assets/1756620/12757354/fbc1d278-c9d7-11e5-8bd7-c6de60046a9e.png)

### Sample is refreshed

![screen shot 2016-02-02 at 17 58 
18](https://cloud.githubusercontent.com/assets/1756620/12757364/069cfad8-c9d8-11e5-8287-daf340df8b93.png)

![screen shot 2016-02-02 at 17 58 
28](https://cloud.githubusercontent.com/assets/1756620/12757376/0d1a46fe-c9d8-11e5-9968-46a0e72811a9.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink backpressure-sampling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1578


commit 7641159ea2a43f95e653e204c8112aede2971f14
Author: Ufuk Celebi 
Date:   2016-02-01T16:28:34Z

[hotfix] [runtime] Create TestingJobManager in TestingUtils

commit 91d6f89d8c029d35f654e92e981043a0ea906a99
Author: Ufuk Celebi 
Date:   2016-02-01T19:39:44Z

[hotfix] [docs] Add config docs about checkpoint stats

commit 6e0a99be04d3b54601f616e2be20fed20d244bc8
Author: Ufuk Celebi 
Date:   2016-02-01T19:51:23Z

[FLINK-3310] [runtime] Add support to sample stack traces of running tasks

commit 37b3f95882500cbf2100170f3f065110fd5f8817
Author: Ufuk Celebi 
Date:   2016-02-01T19:57:48Z

[FLINK-3310] [runtime-web] Add back pressure statistics to web monitor 
(backend)

commit 1d2e8694a3abf386cb68d5247b2840047824bf39
Author: Ufuk Celebi 
Date:   2016-02-01T20:01:26Z

[FLINK-3310] [runtime-web] Add back pressure statistics to web dashboard 
(frontend)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2803) Add test case for Flink's memory allocation

[Lokesh Ravindranathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128580#comment-15128580
 ] 

Lokesh Ravindranathan commented on FLINK-2803:
--

Hi Stephen, 

Thanks so much for the pointers. I was looking at the code for memory 
allocation in all the three locations listed in the JIRA. Should I create 3 
sub-tasks and submit the tests individually? Also, should I test the 
testmanager.sh from Java?

Thanks,
Lokesh

> Add test case for Flink's memory allocation
> ---
>
> Key: FLINK-2803
> URL: https://issues.apache.org/jira/browse/FLINK-2803
> Project: Flink
>  Issue Type: Test
>  Components: Start-Stop Scripts, TaskManager, YARN Client
>Affects Versions: 0.9, 0.10.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> We need a test case which checks the correct memory settings for heap and 
> off-heap memory allocation.
> Memory is calculated in
> 1. The startup scripts ({{taskmanager.sh}})
> 2. The ({{TaskManager}})
> 3. The YARN {{ApplicationMasterActor}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3049.
-
Resolution: Fixed

Fixed via 54743866e86cbe7689ae1dcf001deb559629747b

Sorry for the confusion...

> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3049) Move "Either" type to package "org.apache.flink.types"

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3049.
---

> Move "Either" type to package "org.apache.flink.types"
> --
>
> Key: FLINK-3049
> URL: https://issues.apache.org/jira/browse/FLINK-3049
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Vasia Kalavri
> Fix For: 1.0.0
>
>
> Types reside in package "org.apache.flink.types", the type utils in 
> "typeutils".
> One can either move the type to "flink-core" or create that package in 
> flink-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3303) Move all non-batch specific classes in flink-java to flink-core

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3303.
---

> Move all non-batch specific classes in flink-java to flink-core
> ---
>
> Key: FLINK-3303
> URL: https://issues.apache.org/jira/browse/FLINK-3303
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Currently, {{flink-java}} has a lot of classes that are also needed by the 
> streaming api and that are useful to the classes in {{flink-core}}.
> In particular, certain improvements to the state API are blocked by the fact 
> that certain classes are not in {{flink-core}}.
> I suggest to move classes from {{flink-java}} to {{flink-core}} after the 
> following pattern:
>   - {{flink-core}} will contain all classes that are common across the batch 
> and streaming API.
>   - {{flink-java}} will contain all batch API specific classes (we may 
> eventually even think about renaming it to {{flink-batch-java}}.
> Because {{flink-java}} references {{flink-core}}, this will not be a breaking 
> change. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2348) IterateExampleITCase failing

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-2348.
-
Resolution: Fixed
  Assignee: Stephan Ewen

Stabilized in bfff86c841f0b873f25367419db0b3dd504a1197

> IterateExampleITCase failing
> 
>
> Key: FLINK-2348
> URL: https://issues.apache.org/jira/browse/FLINK-2348
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Stephan Ewen
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec 
> <<< FAILURE! - in 
> org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase
> testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase)
>  Time elapsed: 16.34 sec <<< FAILURE!
> java.lang.AssertionError: Different number of lines in expected and obtained 
> result. expected:<18> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272)
> at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258)
> at 
> org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38)
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - 
> in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase
> Results :
> Failed tests:
> IterateExampleITCase>StreamingProgramTestBase.testJobWithoutObjectReuse:109->postSubmit:38->TestBaseUtils.compareResultsByLinesInMemory:258->TestBaseUtils.compareResultsByLinesInMemory:272
>  Different number of lines in expected and obtained result. expected:<18> but 
> was:<0>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3303) Move all non-batch specific classes in flink-java to flink-core

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3303.
-
Resolution: Done

Done in 21a715867d655bb61df9a9f7eef37e42b99e206a

> Move all non-batch specific classes in flink-java to flink-core
> ---
>
> Key: FLINK-3303
> URL: https://issues.apache.org/jira/browse/FLINK-3303
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Java API
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Currently, {{flink-java}} has a lot of classes that are also needed by the 
> streaming api and that are useful to the classes in {{flink-core}}.
> In particular, certain improvements to the state API are blocked by the fact 
> that certain classes are not in {{flink-core}}.
> I suggest to move classes from {{flink-java}} to {{flink-core}} after the 
> following pattern:
>   - {{flink-core}} will contain all classes that are common across the batch 
> and streaming API.
>   - {{flink-java}} will contain all batch API specific classes (we may 
> eventually even think about renaming it to {{flink-batch-java}}.
> Because {{flink-java}} references {{flink-core}}, this will not be a breaking 
> change. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2348) IterateExampleITCase failing

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-2348.
---

> IterateExampleITCase failing
> 
>
> Key: FLINK-2348
> URL: https://issues.apache.org/jira/browse/FLINK-2348
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Matthias J. Sax
>Assignee: Stephan Ewen
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec 
> <<< FAILURE! - in 
> org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase
> testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase)
>  Time elapsed: 16.34 sec <<< FAILURE!
> java.lang.AssertionError: Different number of lines in expected and obtained 
> result. expected:<18> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272)
> at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258)
> at 
> org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38)
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - 
> in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase
> Results :
> Failed tests:
> IterateExampleITCase>StreamingProgramTestBase.testJobWithoutObjectReuse:109->postSubmit:38->TestBaseUtils.compareResultsByLinesInMemory:258->TestBaseUtils.compareResultsByLinesInMemory:272
>  Different number of lines in expected and obtained result. expected:<18> but 
> was:<0>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3314) Early cancel calls can cause Tasks to not cancel properly

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3314.
-
Resolution: Fixed

Fixed via 8fc7e7af2c7decb8e531b76e3edcc2601f73fe9d

> Early cancel calls can cause Tasks to not cancel properly
> -
>
> Key: FLINK-3314
> URL: https://issues.apache.org/jira/browse/FLINK-3314
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When a task receives the "cancel()" call before the operators are properly 
> instantiated, it can be that the operator never receives a cancel call.
> In certain cases, this causes the operator to hang.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3314) Early cancel calls can cause Tasks to not cancel properly

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3314.
---

> Early cancel calls can cause Tasks to not cancel properly
> -
>
> Key: FLINK-3314
> URL: https://issues.apache.org/jira/browse/FLINK-3314
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When a task receives the "cancel()" call before the operators are properly 
> instantiated, it can be that the operator never receives a cancel call.
> In certain cases, this causes the operator to hang.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [hotfix][yarn] Shutdown JVM when Flink-on-yarn...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1576

[hotfix][yarn] Shutdown JVM when Flink-on-yarn is shutting down

With this change, we call System.exit() when the YARN session is shutting 
down.

On some Linux distributions, YARN is not able to stop containers because 
the "kill" command has different arguments. For example when running Flink on 
GCE ("Debian GNU/Linux 7.9 (wheezy)"), YARN containers will not properly shut 
down.

I propose to apply this change to 0.10 as well.

I tested this on GCE (multiple times, also by externally killing the app)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink yarn_system_exit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1576


commit bc47608ed2568ba35c7103257a106b0abcd7c28e
Author: Robert Metzger 
Date:   2016-02-02T15:36:49Z

[hotfix][yarn] Shutdown JVM when Flink-on-yarn is shutting down




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3207) Add a Pregel iteration abstraction to Gelly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128467#comment-15128467
 ] 

ASF GitHub Bot commented on FLINK-3207:
---

GitHub user vasia opened a pull request:

https://github.com/apache/flink/pull/1575

[FLINK-3207] [gelly] adds the vertex-centric iteration abstraction

As per the mailing list discussion, this PR adds the vertex-centric / 
pregel iteration abstraction to gelly. In this model, superstep computation 
happens inside a single UDF, which has access to received messages, can update 
the vertex state and can generate messages for other vertices. Documentation is 
also updated with a comparison among available iteration options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vasia/flink gelly-pregel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1575


commit 638e754e6d288b6205e172cda79dc7c9ce682fbf
Author: vasia 
Date:   2016-02-02T11:13:51Z

[FLINK-3207] [gelly] add pregel iteration abstraction to gelly

commit 875a72b0546aa12bd5ad0f1f809c3633c249f825
Author: vasia 
Date:   2016-02-02T11:41:51Z

[FLINK-3207] [gelly] add a pregel SSSP example with combiner

commit 7f8f92b918a8ef7d3a530a77a3d8be5fa0758639
Author: vasia 
Date:   2016-02-02T14:00:44Z

[FLINK-3207] [gelly] add compiler and translation tests

Add the vertex-centric abstraction to the gelly guide
and a comparison among all iteration models.

commit 257afeee315765122d856411d3de49164dc99788
Author: vasia 
Date:   2016-02-02T15:43:04Z

[FLINK-3207] [gelly] add pregel methods to gelly-scala




> Add a Pregel iteration abstraction to Gelly
> ---
>
> Key: FLINK-3207
> URL: https://issues.apache.org/jira/browse/FLINK-3207
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Vasia Kalavri
>
> This issue proposes to add a Pregel/Giraph-like iteration abstraction to 
> Gelly that will only expose one UDF to the user, {{compute()}}. {{compute()}} 
> will have access to both the vertex state and the incoming messages, and will 
> be able to produce messages and update the vertex value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3313) Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128517#comment-15128517
 ] 

ASF GitHub Bot commented on FLINK-3313:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1577#issuecomment-178680229
  
Good catch! :-)


> Kafka08ITCase.testOneSourceMultiplePartitions fails with EOFException
> -
>
> Key: FLINK-3313
> URL: https://issues.apache.org/jira/browse/FLINK-3313
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
>  Labels: test-stability
>
> The deserialization has recently been modified: 
> https://github.com/apache/flink/commit/92efcd34a5da2bccb07666f2c647974ea3e7c94f
> this is the log: 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/106401688/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)