[jira] [Commented] (FLINK-5417) Fix the wrong config file name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822727#comment-15822727
 ] 

ASF GitHub Bot commented on FLINK-5417:
---

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3071
  
Hi @zentol , I've updated the svg using Inkscape. 
Is the whitespace you refer to on the topest(the red highlighted part)? I 
think it's normal as the original one has them to. It would not affect view in 
documents.

![default](https://cloud.githubusercontent.com/assets/5276001/21953009/8eef0426-da67-11e6-850a-935268ad019e.JPG)



> Fix the wrong config file name 
> ---
>
> Key: FLINK-5417
> URL: https://issues.apache.org/jira/browse/FLINK-5417
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Tao Wang
>Priority: Trivial
>
> As the config file name is conf/flink-conf.yaml, the usage 
> "conf/flink-config.yaml" in document is wrong and easy to confuse user. We 
> should correct them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3071: [FLINK-5417][DOCUMENTATION]correct the wrong config file ...

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3071
  
Hi @zentol , I've updated the svg using Inkscape. 
Is the whitespace you refer to on the topest(the red highlighted part)? I 
think it's normal as the original one has them to. It would not affect view in 
documents.

![default](https://cloud.githubusercontent.com/assets/5276001/21953009/8eef0426-da67-11e6-850a-935268ad019e.JPG)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-5489) maven release:prepare fails due to invalid JDOM comments in pom.xml

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reassigned FLINK-5489:
-

Assignee: Haohui Mai

> maven release:prepare fails due to invalid JDOM comments in pom.xml
> ---
>
> Key: FLINK-5489
> URL: https://issues.apache.org/jira/browse/FLINK-5489
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
>  Labels: newbie
>
> When I was trying to publish Flink to our internal artifactory, I found out 
> that {{maven release:prepare}} has failed because the plugin complains about 
> the some of the comments pom.xml do not conform with the JDOM format:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare (default-cli) on 
> project flink-parent: Execution default-cli of goal 
> org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare failed: The data 
> "-
> [ERROR] This module is used a dependency in the root pom. It activates 
> shading for all sub modules
> [ERROR] through an include rule in the shading configuration. This assures 
> that Maven always generates
> [ERROR] an effective pom for all modules, i.e. get rid of Maven properties. 
> In particular, this is needed
> [ERROR] to define the Scala version property in the root pom but not let the 
> root pom depend on Scala
> [ERROR] and thus be suffixed along with all other modules.
> [ERROR] " is not legal for a JDOM comment: Comment data cannot start with a 
> hyphen.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-5250.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Implemented on release-1.2 in:
8041deabe8e1e968ba724ba87f534570cdc18522
33c5df6dd0aaf604fdbc97834cfd643a43ca7d08

> Make AbstractUdfStreamOperator aware of WrappingFunction
> 
>
> Key: FLINK-5250
> URL: https://issues.apache.org/jira/browse/FLINK-5250
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
> Fix For: 1.2.0
>
>
> Right now, when using a {{WrappingFunction}}, as happens for 
> {{WindowFunction}} and also for some functions in the Scala API then using 
> custom interfaces is not possible. These custom interfaces are, for example, 
> the checkpointing functions such as {{ListCheckpointed}} and 
> {{CheckpointedFunction}}.
> We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so 
> that it can correctly handle the case where wrapped user functions implement 
> on of these interfaces.
> Also, in the Scala API we have some custom functions that mimic 
> {{WrappingFunction}} behaviour. These should be moved to use 
> {{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3115: [FLINK-5250] Make AbstractUdfStreamOperator aware ...

[StefanRRichter]

Github user StefanRRichter closed the pull request at:

https://github.com/apache/flink/pull/3115


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5265) Introduce state handle replication mode for CheckpointCoordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822488#comment-15822488
 ] 

ASF GitHub Bot commented on FLINK-5265:
---

Github user StefanRRichter closed the pull request at:

https://github.com/apache/flink/pull/2948


> Introduce state handle replication mode for CheckpointCoordinator
> -
>
> Key: FLINK-5265
> URL: https://issues.apache.org/jira/browse/FLINK-5265
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> Currently, the {{CheckpointCoordinator}} only supports repartitioning of 
> {{OperatorStateHandle}}s based on a split-and-distribute strategy. For future 
> state types, such as broadcast or union state, we need a different 
> repartitioning method that allows for replicating state handles to all 
> subtasks.
> This is the first step on the way to implementing broadcast and union states.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822489#comment-15822489
 ] 

ASF GitHub Bot commented on FLINK-5250:
---

Github user StefanRRichter closed the pull request at:

https://github.com/apache/flink/pull/3115


> Make AbstractUdfStreamOperator aware of WrappingFunction
> 
>
> Key: FLINK-5250
> URL: https://issues.apache.org/jira/browse/FLINK-5250
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
>
> Right now, when using a {{WrappingFunction}}, as happens for 
> {{WindowFunction}} and also for some functions in the Scala API then using 
> custom interfaces is not possible. These custom interfaces are, for example, 
> the checkpointing functions such as {{ListCheckpointed}} and 
> {{CheckpointedFunction}}.
> We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so 
> that it can correctly handle the case where wrapped user functions implement 
> on of these interfaces.
> Also, in the Scala API we have some custom functions that mimic 
> {{WrappingFunction}} behaviour. These should be moved to use 
> {{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...

[StefanRRichter]

Github user StefanRRichter closed the pull request at:

https://github.com/apache/flink/pull/2948


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5489) maven release:prepare fails due to invalid JDOM comments in pom.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822486#comment-15822486
 ] 

ASF GitHub Bot commented on FLINK-5489:
---

GitHub user haohui opened a pull request:

https://github.com/apache/flink/pull/3123

[FLINK-5489] maven release:prepare fails due to invalid JDOM comments…

When I was trying to publish Flink to our internal artifactory, I found out 
that maven release:prepare has failed because the plugin complains about the 
some of the comments pom.xml do not conform with the JDOM format (More 
information on the JIRA).

This PR fixes the issue by making all comments conform to the JDOM format.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haohui/flink FLINK-5489

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3123.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3123


commit ba26954891b4ce1d968a90e7c89a04681dca6b65
Author: Haohui Mai 
Date:   2017-01-13T22:48:35Z

[FLINK-5489] maven release:prepare fails due to invalid JDOM comments in 
pom.xml.




> maven release:prepare fails due to invalid JDOM comments in pom.xml
> ---
>
> Key: FLINK-5489
> URL: https://issues.apache.org/jira/browse/FLINK-5489
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Haohui Mai
>Priority: Minor
>  Labels: newbie
>
> When I was trying to publish Flink to our internal artifactory, I found out 
> that {{maven release:prepare}} has failed because the plugin complains about 
> the some of the comments pom.xml do not conform with the JDOM format:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare (default-cli) on 
> project flink-parent: Execution default-cli of goal 
> org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare failed: The data 
> "-
> [ERROR] This module is used a dependency in the root pom. It activates 
> shading for all sub modules
> [ERROR] through an include rule in the shading configuration. This assures 
> that Maven always generates
> [ERROR] an effective pom for all modules, i.e. get rid of Maven properties. 
> In particular, this is needed
> [ERROR] to define the Scala version property in the root pom but not let the 
> root pom depend on Scala
> [ERROR] and thus be suffixed along with all other modules.
> [ERROR] " is not legal for a JDOM comment: Comment data cannot start with a 
> hyphen.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3123: [FLINK-5489] maven release:prepare fails due to in...

[haohui]

GitHub user haohui opened a pull request:

https://github.com/apache/flink/pull/3123

[FLINK-5489] maven release:prepare fails due to invalid JDOM comments…

When I was trying to publish Flink to our internal artifactory, I found out 
that maven release:prepare has failed because the plugin complains about the 
some of the comments pom.xml do not conform with the JDOM format (More 
information on the JIRA).

This PR fixes the issue by making all comments conform to the JDOM format.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haohui/flink FLINK-5489

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3123.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3123


commit ba26954891b4ce1d968a90e7c89a04681dca6b65
Author: Haohui Mai 
Date:   2017-01-13T22:48:35Z

[FLINK-5489] maven release:prepare fails due to invalid JDOM comments in 
pom.xml.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5489) maven release:prepare fails due to invalid JDOM comments in pom.xml

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-5489:
-

 Summary: maven release:prepare fails due to invalid JDOM comments 
in pom.xml
 Key: FLINK-5489
 URL: https://issues.apache.org/jira/browse/FLINK-5489
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.2.0, 1.3.0
Reporter: Haohui Mai
Priority: Minor


When I was trying to publish Flink to our internal artifactory, I found out 
that {{maven release:prepare}} has failed because the plugin complains about 
the some of the comments pom.xml do not conform with the JDOM format:

{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare (default-cli) on 
project flink-parent: Execution default-cli of goal 
org.apache.maven.plugins:maven-release-plugin:2.4.2:prepare failed: The data "-
[ERROR] This module is used a dependency in the root pom. It activates shading 
for all sub modules
[ERROR] through an include rule in the shading configuration. This assures that 
Maven always generates
[ERROR] an effective pom for all modules, i.e. get rid of Maven properties. In 
particular, this is needed
[ERROR] to define the Scala version property in the root pom but not let the 
root pom depend on Scala
[ERROR] and thus be suffixed along with all other modules.
[ERROR] " is not legal for a JDOM comment: Comment data cannot start with a 
hyphen.
{noformat}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3103: [FLINK-5464] [metrics] Prevent some NPEs

[zentol]

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3103


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5464) MetricQueryService throws NullPointerException on JobManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822472#comment-15822472
 ] 

ASF GitHub Bot commented on FLINK-5464:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3103
  
yeah I'll have to rethink this :/

I wanted to just ignore gauges returning null (since nothing in the 
web-interface accounts for that case), but did not adjust the count of metrics 
that are submitted. urgh.


> MetricQueryService throws NullPointerException on JobManager
> 
>
> Key: FLINK-5464
> URL: https://issues.apache.org/jira/browse/FLINK-5464
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> I'm using Flink 699f4b0.
> My JobManager log contains many of these log entries:
> {code}
> 2017-01-11 19:42:05,778 WARN  
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher - Fetching 
> metrics failed.
> akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/MetricQueryService#-970662317]] after [1 ms]
>   at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>   at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
>   at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-01-11 19:42:07,765 WARN  
> org.apache.flink.runtime.metrics.dump.MetricQueryService  - An exception 
> occurred while processing a message.
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:162)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$300(MetricDumpSerialization.java:47)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:90)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:109)
>   at 
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
>   at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3103: [FLINK-5464] [metrics] Prevent some NPEs

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3103
  
yeah I'll have to rethink this :/

I wanted to just ignore gauges returning null (since nothing in the 
web-interface accounts for that case), but did not adjust the count of metrics 
that are submitted. urgh.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5464) MetricQueryService throws NullPointerException on JobManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822473#comment-15822473
 ] 

ASF GitHub Bot commented on FLINK-5464:
---

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/3103


> MetricQueryService throws NullPointerException on JobManager
> 
>
> Key: FLINK-5464
> URL: https://issues.apache.org/jira/browse/FLINK-5464
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> I'm using Flink 699f4b0.
> My JobManager log contains many of these log entries:
> {code}
> 2017-01-11 19:42:05,778 WARN  
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher - Fetching 
> metrics failed.
> akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/MetricQueryService#-970662317]] after [1 ms]
>   at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>   at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
>   at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-01-11 19:42:07,765 WARN  
> org.apache.flink.runtime.metrics.dump.MetricQueryService  - An exception 
> occurred while processing a message.
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:162)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$300(MetricDumpSerialization.java:47)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:90)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:109)
>   at 
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
>   at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822415#comment-15822415
 ] 

ASF GitHub Bot commented on FLINK-5250:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3115
  
Thanks! Could you please close this PR? I merged it on master and am 
waiting for travis to merge to release-1.2 branch.


> Make AbstractUdfStreamOperator aware of WrappingFunction
> 
>
> Key: FLINK-5250
> URL: https://issues.apache.org/jira/browse/FLINK-5250
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
>
> Right now, when using a {{WrappingFunction}}, as happens for 
> {{WindowFunction}} and also for some functions in the Scala API then using 
> custom interfaces is not possible. These custom interfaces are, for example, 
> the checkpointing functions such as {{ListCheckpointed}} and 
> {{CheckpointedFunction}}.
> We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so 
> that it can correctly handle the case where wrapped user functions implement 
> on of these interfaces.
> Also, in the Scala API we have some custom functions that mimic 
> {{WrappingFunction}} behaviour. These should be moved to use 
> {{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3115: [FLINK-5250] Make AbstractUdfStreamOperator aware of Wrap...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3115
  
Thanks! Could you please close this PR? I merged it on master and am 
waiting for travis to merge to release-1.2 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5265) Introduce state handle replication mode for CheckpointCoordinator

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822414#comment-15822414
 ] 

Aljoscha Krettek commented on FLINK-5265:
-

Lower level code in master in:
1020ba2c9cfc1d01703e97c72e20a922bae0732d

> Introduce state handle replication mode for CheckpointCoordinator
> -
>
> Key: FLINK-5265
> URL: https://issues.apache.org/jira/browse/FLINK-5265
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> Currently, the {{CheckpointCoordinator}} only supports repartitioning of 
> {{OperatorStateHandle}}s based on a split-and-distribute strategy. For future 
> state types, such as broadcast or union state, we need a different 
> repartitioning method that allows for replicating state handles to all 
> subtasks.
> This is the first step on the way to implementing broadcast and union states.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5265) Introduce state handle replication mode for CheckpointCoordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822413#comment-15822413
 ] 

ASF GitHub Bot commented on FLINK-5265:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2948
  
Thanks for your work!  I merged, could you please close this PR?


> Introduce state handle replication mode for CheckpointCoordinator
> -
>
> Key: FLINK-5265
> URL: https://issues.apache.org/jira/browse/FLINK-5265
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> Currently, the {{CheckpointCoordinator}} only supports repartitioning of 
> {{OperatorStateHandle}}s based on a split-and-distribute strategy. For future 
> state types, such as broadcast or union state, we need a different 
> repartitioning method that allows for replicating state handles to all 
> subtasks.
> This is the first step on the way to implementing broadcast and union states.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822412#comment-15822412
 ] 

Aljoscha Krettek commented on FLINK-5250:
-

Implemented on master in:
ce023503f6f9601c80d41766c6d59836bcb0abb6
8492d9b7b92674db309f177e48782b11d9d1be5a

> Make AbstractUdfStreamOperator aware of WrappingFunction
> 
>
> Key: FLINK-5250
> URL: https://issues.apache.org/jira/browse/FLINK-5250
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
>
> Right now, when using a {{WrappingFunction}}, as happens for 
> {{WindowFunction}} and also for some functions in the Scala API then using 
> custom interfaces is not possible. These custom interfaces are, for example, 
> the checkpointing functions such as {{ListCheckpointed}} and 
> {{CheckpointedFunction}}.
> We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so 
> that it can correctly handle the case where wrapped user functions implement 
> on of these interfaces.
> Also, in the Scala API we have some custom functions that mimic 
> {{WrappingFunction}} behaviour. These should be moved to use 
> {{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2948: [FLINK-5265] Introduce state handle replication mode for ...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2948
  
Thanks for your work! 👍 I merged, could you please close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822356#comment-15822356
 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-5487 at 1/13/17 9:17 PM:
-

We should block this issue until FLINK-4988, which includes restructuring for 
the ES connectors as part of the fix, is resolved so at-least-once support can 
be simultaneously included in all ES versions.


was (Author: tzulitai):
Restructuring for the ES connectors is included as part of this issue. We 
should block this issue until FLINK-4988 is resolved so at-least-once support 
can be simultaneously included in all ES versions.

> Proper at-least-once support for ElasticsearchSink
> --
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Discussion in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
> message delivery.
> For proper support of at-least-once, the sink will need to participate in 
> Flink's checkpointing: when snapshotting is triggered at the 
> {{ElasticsearchSink}}, we need to synchronize on the pending ES requests by 
> flushing the internal bulk processor. For temporary ES failures (see 
> FLINK-5122) that may happen on the flush, we should retry them before 
> returning from snapshotting and acking the checkpoint. If there are 
> non-temporary ES failures on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822356#comment-15822356
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-5487:


Restructuring for the ES connectors is included as part of this issue. We 
should block this issue until FLINK-4988 is resolved so at-least-once support 
can be simultaneously included in all ES versions.

> Proper at-least-once support for ElasticsearchSink
> --
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Discussion in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
> message delivery.
> For proper support of at-least-once, the sink will need to participate in 
> Flink's checkpointing: when snapshotting is triggered at the 
> {{ElasticsearchSink}}, we need to synchronize on the pending ES requests by 
> flushing the internal bulk processor. For temporary ES failures (see 
> FLINK-5122) that may happen on the flush, we should retry them before 
> returning from snapshotting and acking the checkpoint. If there are 
> non-temporary ES failures on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5488) yarnClient should be closed in AbstractYarnClusterDescriptor for error conditions

[Ted Yu (JIRA)]

Ted Yu created FLINK-5488:
-

 Summary: yarnClient should be closed in 
AbstractYarnClusterDescriptor for error conditions
 Key: FLINK-5488
 URL: https://issues.apache.org/jira/browse/FLINK-5488
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Here is one example:
{code}
if(jobManagerMemoryMb > maxRes.getMemory() ) {
  failSessionDuringDeployment(yarnClient, yarnApplication);
  throw new YarnDeploymentException("The cluster does not have the 
requested resources for the JobManager available!\n"
+ "Maximum Memory: " + maxRes.getMemory() + "MB Requested: " + 
jobManagerMemoryMb + "MB. " + NOTE);
}
{code}
yarnClient implements Closeable.
It should be closed in situations where exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5487:
---
Description: 
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
{{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}}s 
by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) 
that may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.

  was:
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
`ElasticsearchSink`, we need to synchronize on the pending `ActionRequest`s by 
flushing the `BulkProcessor`. For temporary ES failures (see FLINK-5122) that 
may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.


> Proper at-least-once support for ElasticsearchSink
> --
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Discussion in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
> message delivery.
> For proper support of at-least-once, the sink will need to participate in 
> Flink's checkpointing: when snapshotting is triggered at the 
> {{ElasticsearchSink}}, we need to synchronize on the pending 
> {{ActionRequest}}s by flushing the {{BulkProcessor}}. For temporary ES 
> failures (see FLINK-5122) that may happen on the flush, we should retry them 
> before returning from snapshotting and acking the checkpoint. If there are 
> non-temporary ES failures on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5487:
---
Description: 
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
{{ElasticsearchSink}}, we need to synchronize on the pending ES requests by 
flushing the internal bulk processor. For temporary ES failures (see 
FLINK-5122) that may happen on the flush, we should retry them before returning 
from snapshotting and acking the checkpoint. If there are non-temporary ES 
failures on the flush, the current snapshot should fail.

  was:
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
{{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}} 
s by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) 
that may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.


> Proper at-least-once support for ElasticsearchSink
> --
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Discussion in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
> message delivery.
> For proper support of at-least-once, the sink will need to participate in 
> Flink's checkpointing: when snapshotting is triggered at the 
> {{ElasticsearchSink}}, we need to synchronize on the pending ES requests by 
> flushing the internal bulk processor. For temporary ES failures (see 
> FLINK-5122) that may happen on the flush, we should retry them before 
> returning from snapshotting and acking the checkpoint. If there are 
> non-temporary ES failures on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5487:
---
Description: 
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
{{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}} 
s by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) 
that may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.

  was:
Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
{{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}}s 
by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) 
that may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.


> Proper at-least-once support for ElasticsearchSink
> --
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Discussion in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
> message delivery.
> For proper support of at-least-once, the sink will need to participate in 
> Flink's checkpointing: when snapshotting is triggered at the 
> {{ElasticsearchSink}}, we need to synchronize on the pending 
> {{ActionRequest}} s by flushing the {{BulkProcessor}}. For temporary ES 
> failures (see FLINK-5122) that may happen on the flush, we should retry them 
> before returning from snapshotting and acking the checkpoint. If there are 
> non-temporary ES failures on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-5487:
--

 Summary: Proper at-least-once support for ElasticsearchSink
 Key: FLINK-5487
 URL: https://issues.apache.org/jira/browse/FLINK-5487
 Project: Flink
  Issue Type: Bug
  Components: Streaming Connectors
Reporter: Tzu-Li (Gordon) Tai
Priority: Critical


Discussion in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for 
message delivery.

For proper support of at-least-once, the sink will need to participate in 
Flink's checkpointing: when snapshotting is triggered at the 
`ElasticsearchSink`, we need to synchronize on the pending `ActionRequest`s by 
flushing the `BulkProcessor`. For temporary ES failures (see FLINK-5122) that 
may happen on the flush, we should retry them before returning from 
snapshotting and acking the checkpoint. If there are non-temporary ES failures 
on the flush, the current snapshot should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822346#comment-15822346
 ] 

Ted Yu commented on FLINK-5486:
---

Lock on State.bucketStates should be held in the following method:
{code}
  private void handleRestoredBucketState(State restoredState) {
Preconditions.checkNotNull(restoredState);

for (BucketState bucketState : restoredState.bucketStates.values()) {
{code}

> Lack of synchronization in BucketingSink#handleRestoredBucketState()
> 
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
>   
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
>   synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
>   }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside 
> the synchronization block. Otherwise during the processing of 
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be 
> cleared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822251#comment-15822251
 ] 

Fabian Hueske commented on FLINK-5434:
--

Hi [~gallenvara_bg], projection was not added to the Scala API on purpose. The 
problem with project is that, it is not type-safe, i.e., the API cannot 
automatically infer the result type of project. Moreover, projects are super 
straightforward with Scala Lambda functions.

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Vasia Kalavri
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()

[Ted Yu (JIRA)]

Ted Yu created FLINK-5486:
-

 Summary: Lack of synchronization in 
BucketingSink#handleRestoredBucketState()
 Key: FLINK-5486
 URL: https://issues.apache.org/jira/browse/FLINK-5486
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
  
handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);

  synchronized (bucketState.pendingFilesPerCheckpoint) {
bucketState.pendingFilesPerCheckpoint.clear();
  }
{code}
The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside 
the synchronization block. Otherwise during the processing of 
handlePendingFilesForPreviousCheckpoints(), some entries of the map may be 
cleared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5485) Mark compiled web frontend files as binary when processed by git diff

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822127#comment-15822127
 ] 

ASF GitHub Bot commented on FLINK-5485:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/3122

[FLINK-5485] [webfrontend] Mark compiled web frontend files as binary when 
processed by git diff

Particularly beneficial now that javascript is minified, we can mark 
compiled web frontend files as binary when processed by git diff.
  https://linux.die.net/man/5/gitattributes

This does not affect how files are displayed by github.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
5485_mark_compiled_web_frontend_files_as_binary_when_processed_by_git_diff

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3122


commit ace0272f9c3abc60645f2119744e88afa975a731
Author: Greg Hogan 
Date:   2017-01-13T18:33:29Z

[FLINK-5485] [webfrontend] Mark compiled web frontend files as binary when 
processed by git diff

Particularly beneficial now that javascript is minified, we can mark
compiled web frontend files as binary when processed by git diff.
  https://linux.die.net/man/5/gitattributes

This does not affect how files are displayed by github.




> Mark compiled web frontend files as binary when processed by git diff
> -
>
> Key: FLINK-5485
> URL: https://issues.apache.org/jira/browse/FLINK-5485
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.3.0
>
>
> Particularly beneficial now that javascript is minified, we can mark compiled 
> web frontend files as binary when processed by git diff.
>   https://linux.die.net/man/5/gitattributes
> This does not affect how files are displayed by github.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3122: [FLINK-5485] [webfrontend] Mark compiled web front...

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/3122

[FLINK-5485] [webfrontend] Mark compiled web frontend files as binary when 
processed by git diff

Particularly beneficial now that javascript is minified, we can mark 
compiled web frontend files as binary when processed by git diff.
  https://linux.die.net/man/5/gitattributes

This does not affect how files are displayed by github.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
5485_mark_compiled_web_frontend_files_as_binary_when_processed_by_git_diff

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3122


commit ace0272f9c3abc60645f2119744e88afa975a731
Author: Greg Hogan 
Date:   2017-01-13T18:33:29Z

[FLINK-5485] [webfrontend] Mark compiled web frontend files as binary when 
processed by git diff

Particularly beneficial now that javascript is minified, we can mark
compiled web frontend files as binary when processed by git diff.
  https://linux.die.net/man/5/gitattributes

This does not affect how files are displayed by github.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5485) Mark compiled web frontend files as binary when processed by git diff

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-5485:
-

 Summary: Mark compiled web frontend files as binary when processed 
by git diff
 Key: FLINK-5485
 URL: https://issues.apache.org/jira/browse/FLINK-5485
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Affects Versions: 1.3.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Trivial
 Fix For: 1.3.0


Particularly beneficial now that javascript is minified, we can mark compiled 
web frontend files as binary when processed by git diff.
  https://linux.die.net/man/5/gitattributes

This does not affect how files are displayed by github.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3086: Improve docker setup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3086


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3086: Improve docker setup

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/3086
  
No, looks good. Thanks. As Greg mentioned, please open a JIRA issue next 
time. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-5413) Convert TableEnvironmentITCases to unit tests

[GaoLun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

GaoLun reassigned FLINK-5413:
-

Assignee: GaoLun

> Convert TableEnvironmentITCases to unit tests
> -
>
> Key: FLINK-5413
> URL: https://issues.apache.org/jira/browse/FLINK-5413
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: GaoLun
>
> The following IT cases could be converted into unit tests:
> - {{org.apache.flink.table.api.scala.batch.TableEnvironmentITCase}}
> - {{org.apache.flink.table.api.java.batch.TableEnvironmentITCase}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822050#comment-15822050
 ] 

ASF GitHub Bot commented on FLINK-5434:
---

GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/3121

[FLINK-5434] Remove unsupported project() transformation from Scala 
DataStream docs.

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [X] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink flink-5434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3121


commit 370d48a94d3984cfc0461a8d2c256714be962018
Author: gaolun.gl 
Date:   2017-01-13T17:41:39Z

Remove unsupported project() transformation from Scala DataStream docs.




> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Vasia Kalavri
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3121: [FLINK-5434] Remove unsupported project() transfor...

[gallenvara]

GitHub user gallenvara opened a pull request:

https://github.com/apache/flink/pull/3121

[FLINK-5434] Remove unsupported project() transformation from Scala 
DataStream docs.

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [X] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gallenvara/flink flink-5434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3121


commit 370d48a94d3984cfc0461a8d2c256714be962018
Author: gaolun.gl 
Date:   2017-01-13T17:41:39Z

Remove unsupported project() transformation from Scala DataStream docs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[GaoLun (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822045#comment-15822045
 ] 

GaoLun commented on FLINK-5434:
---

Just remove from the doc as the following pr. Should we create another jira to 
support the operator for the scala data stream api ?

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Vasia Kalavri
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2720) Add Storm-CountMetric in flink-stormcompatibility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822025#comment-15822025
 ] 

ASF GitHub Bot commented on FLINK-2720:
---

Github user mjsax commented on the issue:

https://github.com/apache/flink/pull/1157
  
Did not have a chance to work on this either -- I am in a new job, too :)

You can pick this up again if you are still interested.


> Add Storm-CountMetric in flink-stormcompatibility
> -
>
> Key: FLINK-2720
> URL: https://issues.apache.org/jira/browse/FLINK-2720
> Project: Flink
>  Issue Type: New Feature
>  Components: Storm Compatibility
>Reporter: Huang Wei
>Assignee: Matthias J. Sax
> Fix For: 1.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Add the CountMetric for the first step of storm metrics:
> 1.Do a wrapper FlinkCountMetric for CountMetric
> 2.push the RuntimeContext in FlinkTopologyContext to use `addAccumulator` 
> method for registering the metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1157: [FLINK-2720][storm-compatibility]Add Storm-CountMetric fo...

[mjsax]

Github user mjsax commented on the issue:

https://github.com/apache/flink/pull/1157
  
Did not have a chance to work on this either -- I am in a new job, too :)

You can pick this up again if you are still interested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5484) Kryo serialization changed between 1.1 and 1.2

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-5484:
--

 Summary: Kryo serialization changed between 1.1 and 1.2
 Key: FLINK-5484
 URL: https://issues.apache.org/jira/browse/FLINK-5484
 Project: Flink
  Issue Type: Bug
  Components: Type Serialization System
Reporter: Ufuk Celebi


I think the way that Kryo serializes data changed between 1.1 and 1.2.

I have a generic Object that is serialized as part of a 1.1 savepoint that I 
cannot resume from with 1.2:

{code}
org.apache.flink.client.program.ProgramInvocationException: The program 
execution failed: Job execution failed.
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at 
org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:101)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:68)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1486)
at com.dataartisans.DidKryoChange.main(DidKryoChange.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
at 
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1117)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution 
failed.
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:900)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalStateException: Could not initialize keyed state 
backend.
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:286)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:199)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:649)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:636)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:654)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: 
Unable to find class: f
at 

[jira] [Commented] (FLINK-5464) MetricQueryService throws NullPointerException on JobManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821920#comment-15821920
 ] 

ASF GitHub Bot commented on FLINK-5464:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3103
  
While testing this PR, I found that the jobmanager.log is now full with 
exceptions like this one:

```
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeString(MetricDumpSerialization.java:230)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeMetricInfo(MetricDumpSerialization.java:278)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeGauge(MetricDumpSerialization.java:243)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$800(MetricDumpSerialization.java:47)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpDeserializer.deserialize(MetricDumpSerialization.java:214)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.addMetrics(MetricFetcher.java:196)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.access$500(MetricFetcher.java:58)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher$4.onSuccess(MetricFetcher.java:188)
at akka.dispatch.OnSuccess.internal(Future.scala:212)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at 
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:25)
at 
scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:117)
at 
scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:115)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at 
java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.java:1265)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
at 
java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:604)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:398)
```



> MetricQueryService throws NullPointerException on JobManager
> 
>
> Key: FLINK-5464
> URL: https://issues.apache.org/jira/browse/FLINK-5464
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> I'm using Flink 699f4b0.
> My JobManager log contains many of these log entries:
> {code}
> 2017-01-11 19:42:05,778 WARN  
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher - Fetching 
> metrics failed.
> akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/MetricQueryService#-970662317]] after [1 ms]
>   at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>   at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
>   at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-01-11 19:42:07,765 WARN  
> org.apache.flink.runtime.metrics.dump.MetricQueryService  - An exception 
> occurred while processing a message.
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:162)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$300(MetricDumpSerialization.java:47)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:90)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:109)
>   at 
> 

[GitHub] flink issue #3103: [FLINK-5464] [metrics] Prevent some NPEs

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3103
  
While testing this PR, I found that the jobmanager.log is now full with 
exceptions like this one:

```
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeString(MetricDumpSerialization.java:230)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeMetricInfo(MetricDumpSerialization.java:278)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.deserializeGauge(MetricDumpSerialization.java:243)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$800(MetricDumpSerialization.java:47)
at 
org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpDeserializer.deserialize(MetricDumpSerialization.java:214)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.addMetrics(MetricFetcher.java:196)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.access$500(MetricFetcher.java:58)
at 
org.apache.flink.runtime.webmonitor.metrics.MetricFetcher$4.onSuccess(MetricFetcher.java:188)
at akka.dispatch.OnSuccess.internal(Future.scala:212)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at 
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:25)
at 
scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:117)
at 
scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:115)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at 
java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.java:1265)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
at 
java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:604)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:398)
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-3133) Introduce collect()/count()/print() methods in DataStream API

[Evgeny Kincharov (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821899#comment-15821899
 ] 

Evgeny Kincharov edited comment on FLINK-3133 at 1/13/17 3:42 PM:
--

Hi [~mxm], I assign this issue to me again. I try to explain how I implemented 
this and some problems that I met with.
1. I planned to make a new sink that only save records passing through. It is 
not hard.
2. We need to have the possibility to execute the stream pipeline in the 
nonblocking mode. You propose add the method executeWithControl() instead the 
execute(). I know this is described in subtasks of FLINK-4272 but I implemented 
executeWithControl() in this issue only for debugging the sink from 1 (of 
course I delete it from here before PR)
3. We need to insert this sink into the pipeline. It is the hardest part of the 
implementation. If I add this sink after executeWithControl() - addSink doesn't 
change the pipeline due to some conversion transformations during 
executeWithControl.
My current state here: https://github.com/kenmy/flink/tree/FLINK-3133_temp
The commit is here: 
https://github.com/kenmy/flink/commit/136d51c79008126ed586f2b50a997b03c84b21b3
As a result, I can't see any simple possibility to add "sample" without 
changing pipeline before start executing, It may be a problem. Possible 
solutions (IMHO) :
1 Change the design to
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment(); 
DataStream printSink = env.addSource(..).print(); 
ResultQueryable queryObject = env.executeWithResultQueryable(); 
List sampled = queryObject.retrieve(printSink, Time.seconds(5)); 
{code}
The main idea is to change the pipeline before the starting execution, not 
after.
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment();
Sampler sampler = new Sampler();
DataStream streamData = env.addSource(..).map(..).sample(sampler);
JobClient jobClient = env.executeWithControl();
Iterable sampled = sampler(Time.seconds(5));
{code}
2. Don't use sink, use another mechanism to intercept DataStream (like 
extending DataStream by method getSampler that will return object which allows 
to enable/disable storing transferred data for any DataStream). IMHO "sink" 
approach looks more lightweight.
What solution do you prefer, I recommend first but I may not know all bussiness 
needs. Or may be you know better solution?
Thanks.


was (Author: kenmy):
Hi [~mxm], I assign this issue to me again. I try to explain how I implemented 
this and some problems that I met with.
1. I planned to make the new sink that only save records passing through. It is 
not hard.
2. We need to have the possibility to execute the stream pipeline in the 
nonblocking mode. You propose add the method executeWithControl() instead the 
execute(). I know this is described in subtasks of FLINK-4272 but I implemented 
executeWithControl() in this issue only for debugging the sink from 1 (of 
course I delete it from here before PR)
3. We need to insert this sink into the pipeline. It is the hardest part of the 
implementation. If I add this sink after executeWithControl() - addSink doesn't 
change the pipeline due to some conversion transformations during 
executeWithControl.
The my current state here: https://github.com/kenmy/flink/tree/FLINK-3133_temp
The commit is here: 
https://github.com/kenmy/flink/commit/136d51c79008126ed586f2b50a997b03c84b21b3
As a result, I can't see the simple possibility to add "sample" without 
changing pipeline before start executing, It may be a problem. Possible 
solutions (IMHO) :
1 Change the design to
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment(); 
DataStream printSink = env.addSource(..).print(); 
ResultQueryable queryObject = env.executeWithResultQueryable(); 
List sampled = queryObject.retrieve(printSink, Time.seconds(5)); 
{code}
The main idea is to change the pipeline before the starting execution, not 
after.
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment();
Sampler sampler = new Sampler();
DataStream streamData = env.addSource(..).map(..).sample(sampler);
JobClient jobClient = env.executeWithControl();
Iterable sampled = sampler(Time.seconds(5));
{code}
2. Don't use sink, use another mechanism to intercept DataStream (like 
extending DataStream by method getSampler that will return object which allows 
to enable/disable storing transferred data for any DataStream). IMHO "sink" 
approach looks more lightweight.
What solution do you prefer, I recommend first but I may not know all bussiness 
needs. Or may be you know better solution?
Thanks.

> Introduce collect()/count()/print() methods in DataStream API
> -
>
> Key: FLINK-3133
> URL: 

[jira] [Assigned] (FLINK-5375) Fix watermark documentation

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai reassigned FLINK-5375:
--

Assignee: Tzu-Li (Gordon) Tai

> Fix watermark documentation
> ---
>
> Key: FLINK-5375
> URL: https://issues.apache.org/jira/browse/FLINK-5375
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation, Project Website
>Affects Versions: 1.2.0, 1.1.3, 1.3.0
>Reporter: Fabian Hueske
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Fix For: 1.2.0, 1.3.0, 1.1.5
>
>
> The [documentation of 
> watermarks|https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_time.html#event-time-and-watermarks]
>  is not correct. It states 
> {quote}
> A Watermark(t) declares that event time has reached time t in that stream, 
> meaning that all events with a timestamps t’ < t have occurred.
> {quote}
> whereas the JavaDocs which is aligned with implementation says
> {quote}
> A Watermark tells operators that receive it that no elements with a
> timestamp older or equal to the watermark timestamp should arrive at the
> operator.
> {quote}
> The documentation needs to be updated. Moreover, we need to carefully check 
> that the watermark semantics are correctly described in other pages of the 
> documentation and blog posts published on the Flink website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5483) Link to "linking modules not contained in binary distribution" broken in all connector documentations

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-5483:
--

 Summary: Link to "linking modules not contained in binary 
distribution" broken in all connector documentations
 Key: FLINK-5483
 URL: https://issues.apache.org/jira/browse/FLINK-5483
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
 Fix For: 1.2.0


All connector documentations have this message: "Note that the streaming 
connectors are currently not part of the binary distribution. See how to link 
with them for cluster execution `here`."

The link at the end is broken in all the connector documentations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5482) QueryableStateClient does not recover from a failed lookup due to a non-running job

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821907#comment-15821907
 ] 

ASF GitHub Bot commented on FLINK-5482:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/3120

[FLINK-5482] QueryableStateClient does not recover from a failed lookup due 
to a non-running job

This PR checks each cached lookup query whether it is complete and removes 
any failed lookup from the cache in favour of a retry.

An appropriate unit test is added based on existing test code.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-5482

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3120


commit 086aca06674618cb3be962a883efa62c77aa1c66
Author: Nico Kruber 
Date:   2017-01-12T15:41:30Z

[FLINK-5482] share more code in QueryableStateITCase

commit a50e155cf2a1e7e04b160d5226f16f017509799e
Author: Nico Kruber 
Date:   2017-01-12T15:48:27Z

[FLINK-5482] fix QueryableStateClient not re-issuing a lookup upon failure

Any failing lookup, e.g. in case the job has not been started yet, 
previously
remained in the lookup cache and thus future queries did not retry the 
lookup
and failed. This commit changes the lookup caching code so that completed
and failed futures are removed from the cache and replaced by new lookups.




> QueryableStateClient does not recover from a failed lookup due to a 
> non-running job
> ---
>
> Key: FLINK-5482
> URL: https://issues.apache.org/jira/browse/FLINK-5482
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> When the QueryableStateClient is used to issue a query but the job is not 
> running yet, its internal lookup result is cached with an 
> IllegalStateException that the job was not found. It does, however, never 
> recover from that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3120: [FLINK-5482] QueryableStateClient does not recover...

[NicoK]

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/3120

[FLINK-5482] QueryableStateClient does not recover from a failed lookup due 
to a non-running job

This PR checks each cached lookup query whether it is complete and removes 
any failed lookup from the cache in favour of a retry.

An appropriate unit test is added based on existing test code.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-5482

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3120


commit 086aca06674618cb3be962a883efa62c77aa1c66
Author: Nico Kruber 
Date:   2017-01-12T15:41:30Z

[FLINK-5482] share more code in QueryableStateITCase

commit a50e155cf2a1e7e04b160d5226f16f017509799e
Author: Nico Kruber 
Date:   2017-01-12T15:48:27Z

[FLINK-5482] fix QueryableStateClient not re-issuing a lookup upon failure

Any failing lookup, e.g. in case the job has not been started yet, 
previously
remained in the lookup cache and thus future queries did not retry the 
lookup
and failed. This commit changes the lookup caching code so that completed
and failed futures are removed from the cache and replaced by new lookups.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-5481) Cannot create Collection of Row

[Anton Solovev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Solovev updated FLINK-5481:
-
 Priority: Trivial  (was: Major)
Affects Version/s: 1.2.0

> Cannot create Collection of Row 
> 
>
> Key: FLINK-5481
> URL: https://issues.apache.org/jira/browse/FLINK-5481
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Anton Solovev
>Assignee: Anton Solovev
>Priority: Trivial
>
> When we use {{ExecutionEnvironment#fromElements(X... data)}} it takes first 
> element of {{data}} to define a type. If first Row in collection has wrong 
> number of fields (there are nulls) TypeExtractor returns not RowTypeInfo, but 
> GenericType



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-5465) RocksDB fails with segfault while calling AbstractRocksDBState.clear()

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-5465.
---
Resolution: Won't Fix

Thank you guys for looking into it. I'm closing the issue as won't fix.

> RocksDB fails with segfault while calling AbstractRocksDBState.clear()
> --
>
> Key: FLINK-5465
> URL: https://issues.apache.org/jira/browse/FLINK-5465
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
> Attachments: hs-err-pid26662.log
>
>
> I'm using Flink 699f4b0.
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f91a0d49b78, pid=26662, tid=140263356024576
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni-linux64.so+0x1aeb78]  
> rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /yarn/nm/usercache/robert/appcache/application_1484132267957_0007/container_1484132267957_0007_01_10/hs_err_pid26662.log
> Compiled method (nm) 1869778  903 n   org.rocksdb.RocksDB::remove 
> (native)
>  total in heap  [0x7f91b40b9dd0,0x7f91b40ba150] = 896
>  relocation [0x7f91b40b9ef0,0x7f91b40b9f48] = 88
>  main code  [0x7f91b40b9f60,0x7f91b40ba150] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.sun.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5482) QueryableStateClient does not recover from a failed lookup due to a non-running job

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-5482:
--

 Summary: QueryableStateClient does not recover from a failed 
lookup due to a non-running job
 Key: FLINK-5482
 URL: https://issues.apache.org/jira/browse/FLINK-5482
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Nico Kruber
Assignee: Nico Kruber


When the QueryableStateClient is used to issue a query but the job is not 
running yet, its internal lookup result is cached with an IllegalStateException 
that the job was not found. It does, however, never recover from that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3133) Introduce collect()/count()/print() methods in DataStream API

[Evgeny Kincharov (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821899#comment-15821899
 ] 

Evgeny Kincharov edited comment on FLINK-3133 at 1/13/17 3:20 PM:
--

Hi [~mxm], I assign this issue to me again. I try to explain how I implemented 
this and some problems that I met with.
1. I planned to make the new sink that only save records passing through. It is 
not hard.
2. We need to have the possibility to execute the stream pipeline in the 
nonblocking mode. You propose add the method executeWithControl() instead the 
execute(). I know this is described in subtasks of FLINK-4272 but I implemented 
executeWithControl() in this issue only for debugging the sink from 1 (of 
course I delete it from here before PR)
3. We need to insert this sink into the pipeline. It is the hardest part of the 
implementation. If I add this sink after executeWithControl() - addSink doesn't 
change the pipeline due to some conversion transformations during 
executeWithControl.
The my current state here: https://github.com/kenmy/flink/tree/FLINK-3133_temp
The commit is here: 
https://github.com/kenmy/flink/commit/136d51c79008126ed586f2b50a997b03c84b21b3
As a result, I can't see the simple possibility to add "sample" without 
changing pipeline before start executing, It may be a problem. Possible 
solutions (IMHO) :
1 Change the design to
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment(); 
DataStream printSink = env.addSource(..).print(); 
ResultQueryable queryObject = env.executeWithResultQueryable(); 
List sampled = queryObject.retrieve(printSink, Time.seconds(5)); 
{code}
The main idea is to change the pipeline before the starting execution, not 
after.
{code:java}
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment();
Sampler sampler = new Sampler();
DataStream streamData = env.addSource(..).map(..).sample(sampler);
JobClient jobClient = env.executeWithControl();
Iterable sampled = sampler(Time.seconds(5));
{code}
2. Don't use sink, use another mechanism to intercept DataStream (like 
extending DataStream by method getSampler that will return object which allows 
to enable/disable storing transferred data for any DataStream). IMHO "sink" 
approach looks more lightweight.
What solution do you prefer, I recommend first but I may not know all bussiness 
needs. Or may be you know better solution?
Thanks.


was (Author: kenmy):
Hi [~mxm], I assign this issue to me again. I try to explain how I implemented 
this and some problems that I met with.
1. I planned to make the new sink that only save records passing through. It is 
not hard.
2. We need to have the possibility to execute the stream pipeline in the 
nonblocking mode. You propose add the method executeWithControl() instead the 
execute(). I know this is described in subtasks of FLINK-4272 but I implemented 
executeWithControl() in this issue only for debugging the sink from 1 (of 
course I delete it from here before PR)
3. We need to insert this sink into the pipeline. It is the hardest part of the 
implementation. If I add this sink after executeWithControl() - addSink doesn't 
change the pipeline due to some conversion transformations during 
executeWithControl.
The my current state here: https://github.com/kenmy/flink/tree/FLINK-3133_temp
The commit is here: 
https://github.com/kenmy/flink/commit/136d51c79008126ed586f2b50a997b03c84b21b3
As a result, I can't see the simple possibility to add "sample" without 
changing pipeline before start executing, It may be a problem. Possible 
solutions (IMHO) :
1 Change the design to
{code:java} 
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment(); 
DataStream printSink = env.addSource(..).print(); 
ResultQueryable queryObject = env.executeWithResultQueryable(); 
List sampled = queryObject.retrieve(printSink, Time.seconds(5)); 
{code}
The main idea is to change the pipeline before the starting execution, not 
after.
{code:java} 
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment();
Sampler sampler = new Sampler();
DataStream streamData = env.addSource(..).map(..).sample(sampler);
JobClient jobClient = env.executeWithControl();
Iterable sampled = sampler(Time.seconds(5));
{code}
2. Don't use sink, use another mechanism to intercept DataStream (like 
extending DataStream by method getSampler that will return object which allows 
to enable/disable storing transferred data for any DataStream). IMHO "sink" 
approach looks more lightweight.
What solution do you prefer, I recommend [1] but I may not know all bussiness 
needs. Or may be you know better solution?
Thanks.

> Introduce collect()/count()/print() methods in DataStream API
> -
>
> Key: FLINK-3133
> URL: 

[jira] [Commented] (FLINK-3133) Introduce collect()/count()/print() methods in DataStream API

[Evgeny Kincharov (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821899#comment-15821899
 ] 

Evgeny Kincharov commented on FLINK-3133:
-

Hi [~mxm], I assign this issue to me again. I try to explain how I implemented 
this and some problems that I met with.
1. I planned to make the new sink that only save records passing through. It is 
not hard.
2. We need to have the possibility to execute the stream pipeline in the 
nonblocking mode. You propose add the method executeWithControl() instead the 
execute(). I know this is described in subtasks of FLINK-4272 but I implemented 
executeWithControl() in this issue only for debugging the sink from 1 (of 
course I delete it from here before PR)
3. We need to insert this sink into the pipeline. It is the hardest part of the 
implementation. If I add this sink after executeWithControl() - addSink doesn't 
change the pipeline due to some conversion transformations during 
executeWithControl.
The my current state here: https://github.com/kenmy/flink/tree/FLINK-3133_temp
The commit is here: 
https://github.com/kenmy/flink/commit/136d51c79008126ed586f2b50a997b03c84b21b3
As a result, I can't see the simple possibility to add "sample" without 
changing pipeline before start executing, It may be a problem. Possible 
solutions (IMHO) :
1 Change the design to
{code:java} 
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment(); 
DataStream printSink = env.addSource(..).print(); 
ResultQueryable queryObject = env.executeWithResultQueryable(); 
List sampled = queryObject.retrieve(printSink, Time.seconds(5)); 
{code}
The main idea is to change the pipeline before the starting execution, not 
after.
{code:java} 
StreamExecutionEnvironment env = 
StramEnvironment.getStreamExecutionEnvironment();
Sampler sampler = new Sampler();
DataStream streamData = env.addSource(..).map(..).sample(sampler);
JobClient jobClient = env.executeWithControl();
Iterable sampled = sampler(Time.seconds(5));
{code}
2. Don't use sink, use another mechanism to intercept DataStream (like 
extending DataStream by method getSampler that will return object which allows 
to enable/disable storing transferred data for any DataStream). IMHO "sink" 
approach looks more lightweight.
What solution do you prefer, I recommend [1] but I may not know all bussiness 
needs. Or may be you know better solution?
Thanks.

> Introduce collect()/count()/print() methods in DataStream API
> -
>
> Key: FLINK-3133
> URL: https://issues.apache.org/jira/browse/FLINK-3133
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API, Streaming
>Affects Versions: 0.10.0, 1.0.0, 0.10.1
>Reporter: Maximilian Michels
>Assignee: Evgeny Kincharov
> Fix For: 1.0.0
>
>
> The DataSet API's methods {{collect()}}, {{count()}}, and {{print()}} should 
> be mirrored to the DataStream API. 
> The semantics of the calls are different. We need to be able to sample parts 
> of a stream, e.g. by supplying a time period in the arguments to the methods. 
> Users should use the {{JobClient}} to retrieve the results.
> {code:java}
> StreamExecutionEnvironment env = 
> StramEnvironment.getStreamExecutionEnvironment();
> DataStream streamData = env.addSource(..).map(..);
> JobClient jobClient = env.executeWithControl();
> Iterable sampled = jobClient.sampleStream(streamData, 
> Time.seconds(5));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5481) Cannot create Collection of Row

[Anton Solovev (JIRA)]

Anton Solovev created FLINK-5481:


 Summary: Cannot create Collection of Row 
 Key: FLINK-5481
 URL: https://issues.apache.org/jira/browse/FLINK-5481
 Project: Flink
  Issue Type: Bug
Reporter: Anton Solovev
Assignee: Anton Solovev


When we use {{ExecutionEnvironment#fromElements(X... data)}} it takes first 
element of {{data}} to define a type. If first Row in collection has wrong 
number of fields (there are nulls) TypeExtractor returns not RowTypeInfo, but 
GenericType



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-5268) Split TableProgramsTestBase into TableProgramsCollectionTestBase and TableProgramsClusterTestBase

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821889#comment-15821889
 ] 

Jark Wu edited comment on FLINK-5268 at 1/13/17 3:03 PM:
-

Hi [~fhueske] , do we have the case that an IT case should run both on 
collection environment and cluster environment ? 

And when should we use cluster environment ?  To cover the combine phase is the 
only case I know. I think we should add some comments to remind developers when 
to use {{TableProgramsClusterTestBase}} .


was (Author: jark):
Hi [~fhueske] , do we have the case that an IT case should run both on 
collection environment and cluster environment ? 

And when should we use cluster environment ?   I think we should add some 
comments to remind developers when to use {{TableProgramsClusterTestBase}} .

> Split TableProgramsTestBase into TableProgramsCollectionTestBase and 
> TableProgramsClusterTestBase
> -
>
> Key: FLINK-5268
> URL: https://issues.apache.org/jira/browse/FLINK-5268
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Priority: Minor
>
> Currently the {{TableProgramsTestBase}} allows to run tests on a collection 
> environment and a MiniCluster by setting a testing parameter. This was done 
> to cover different execution path. However, testing on a MiniCluster is quite 
> expensive and should only be done in rare cases.
> I propose to split the {{TableProgramsTestBase}} into 
> * {{TableProgramsCollectionTestBase}} and
> * {{TableProgramsClusterTestBase}}
> to have the separation of both execution backends more clear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3150) Make YARN container invocation configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821890#comment-15821890
 ] 

ASF GitHub Bot commented on FLINK-3150:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3056
  
+1 to merge. The cluster test was successful.


> Make YARN container invocation configurable
> ---
>
> Key: FLINK-3150
> URL: https://issues.apache.org/jira/browse/FLINK-3150
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN
>Reporter: Robert Metzger
>Assignee: Nico Kruber
>  Labels: qa
>
> Currently, the JVM invocation call of YARN containers is hardcoded.
> With this change, I would like to make the call configurable, using a string 
> such as
> "%java% %memopts% %jvmopts% ..."
> Also, we should respect the {{java.env.home}} if its set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5268) Split TableProgramsTestBase into TableProgramsCollectionTestBase and TableProgramsClusterTestBase

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821889#comment-15821889
 ] 

Jark Wu commented on FLINK-5268:


Hi [~fhueske] , do we have the case that an IT case should run both on 
collection environment and cluster environment ? 

And when should we use cluster environment ?   I think we should add some 
comments to remind developers when to use {{TableProgramsClusterTestBase}} .

> Split TableProgramsTestBase into TableProgramsCollectionTestBase and 
> TableProgramsClusterTestBase
> -
>
> Key: FLINK-5268
> URL: https://issues.apache.org/jira/browse/FLINK-5268
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Priority: Minor
>
> Currently the {{TableProgramsTestBase}} allows to run tests on a collection 
> environment and a MiniCluster by setting a testing parameter. This was done 
> to cover different execution path. However, testing on a MiniCluster is quite 
> expensive and should only be done in rare cases.
> I propose to split the {{TableProgramsTestBase}} into 
> * {{TableProgramsCollectionTestBase}} and
> * {{TableProgramsClusterTestBase}}
> to have the separation of both execution backends more clear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3056: [FLINK-3150] make YARN container invocation configurable

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3056
  
+1 to merge. The cluster test was successful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5441) Directly allow SQL queries on a Table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821880#comment-15821880
 ] 

ASF GitHub Bot commented on FLINK-5441:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3107
  
Hi @KurtYoung , thank your for reviewing. I'm not sure about that. Do you 
mean omitting the "FROM" syntax ?  But it will be hard to validate SQL syntax. 
Because the query can follow a WHERE clause or JOIN/UNION with other tables.  
For example, it's wired and hard to validate the query like this:

```
mytable.sql("SELECT * WHERE a > 13 UNION SELECT * FROM TABLE_B")
```

I think allowing query on other tables is fine here.  What do you think ? 


> Directly allow SQL queries on a Table
> -
>
> Key: FLINK-5441
> URL: https://issues.apache.org/jira/browse/FLINK-5441
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Right now a user has to register a table before it can be used in SQL 
> queries. In order to allow more fluent programming we propose calling SQL 
> directly on a table. An underscore can be used to reference the current table:
> {code}
> myTable.sql("SELECT a, b, c FROM _ WHERE d = 12")
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3107: [FLINK-5441] [table] Directly allow SQL queries on a Tabl...

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3107
  
Hi @KurtYoung , thank your for reviewing. I'm not sure about that. Do you 
mean omitting the "FROM" syntax ?  But it will be hard to validate SQL syntax. 
Because the query can follow a WHERE clause or JOIN/UNION with other tables.  
For example, it's wired and hard to validate the query like this:

```
mytable.sql("SELECT * WHERE a > 13 UNION SELECT * FROM TABLE_B")
```

I think allowing query on other tables is fine here.  What do you think ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3119: [FLINK-5468] Improved error message for migrating semi as...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3119
  
cc @rmetzger 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5468) Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with ClassNotFoundException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821850#comment-15821850
 ] 

ASF GitHub Bot commented on FLINK-5468:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3119
  
cc @rmetzger 


> Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with 
> ClassNotFoundException
> ---
>
> Key: FLINK-5468
> URL: https://issues.apache.org/jira/browse/FLINK-5468
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Stefan Richter
>
> I think we should catch this exception and explain what's going on and how 
> users can resolve the issue.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>   at 
> org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
>   at 
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>   at com.dataartisans.eventwindow.Generator.main(Generator.java:60)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
>   at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:328)
>   at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:382)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
>   ... 22 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend$FinalSemiAsyncSnapshot
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:162)
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:70)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepoint(SavepointStore.java:138)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:64)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1348)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> 

[jira] [Commented] (FLINK-5468) Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with ClassNotFoundException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821849#comment-15821849
 ] 

ASF GitHub Bot commented on FLINK-5468:
---

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/3119

[FLINK-5468] Improved error message for migrating semi async snapshot

This PR addresses [FLINK-5468].

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink 
FLINK-5468-restoring-from-semi-async

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3119


commit 537166e1ff510d4f84783b22f8cc6f8f0ee2f753
Author: Stefan Richter 
Date:   2017-01-13T14:19:37Z

[FLINK-5468] Improved error message for migrating semi async RocksDB 
snapshot




> Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with 
> ClassNotFoundException
> ---
>
> Key: FLINK-5468
> URL: https://issues.apache.org/jira/browse/FLINK-5468
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Stefan Richter
>
> I think we should catch this exception and explain what's going on and how 
> users can resolve the issue.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>   at 
> org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
>   at 
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>   at com.dataartisans.eventwindow.Generator.main(Generator.java:60)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
>   at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:328)
>   at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:382)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
>   ... 22 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend$FinalSemiAsyncSnapshot
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:162)
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:70)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepoint(SavepointStore.java:138)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:64)
>   at 
> 

[GitHub] flink pull request #3119: [FLINK-5468] Improved error message for migrating ...

[StefanRRichter]

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/3119

[FLINK-5468] Improved error message for migrating semi async snapshot

This PR addresses [FLINK-5468].

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink 
FLINK-5468-restoring-from-semi-async

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3119


commit 537166e1ff510d4f84783b22f8cc6f8f0ee2f753
Author: Stefan Richter 
Date:   2017-01-13T14:19:37Z

[FLINK-5468] Improved error message for migrating semi async RocksDB 
snapshot




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5224) Improve UDTF: emit rows directly instead of buffering them

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821841#comment-15821841
 ] 

ASF GitHub Bot commented on FLINK-5224:
---

GitHub user wuchong opened a pull request:

https://github.com/apache/flink/pull/3118

[FLINK-5224] [table] Improve UDTF: emit rows directly instead of buffering 
them

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


This PR is an improvement for user-defined table function in Flink TableAPI 
& SQL. I didn't add new tests because the existing tests are sufficient.

The core design is code generating a `Collector` and register it into the 
instance of `TableFunction`, and emit the rows generated by the UDTF directly 
instead of buffering them.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wuchong/flink udtf-collector-FLINK-5224

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3118


commit 5e126a97bb26654048efa735c21e0daff136a6a2
Author: Jark Wu 
Date:   2017-01-13T13:53:49Z

[FLINK-5224] [table] Improve UDTF: emit rows directly instead of buffering 
them




> Improve UDTF: emit rows directly instead of buffering them
> --
>
> Key: FLINK-5224
> URL: https://issues.apache.org/jira/browse/FLINK-5224
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Jark Wu
>Assignee: Jark Wu
>
> This needs to code generate a `Collector` and register it into instance of 
> {{TableFunction}}, and emit the rows generated by the UDTF directly instead 
> of buffering them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3118: [FLINK-5224] [table] Improve UDTF: emit rows direc...

[wuchong]

GitHub user wuchong opened a pull request:

https://github.com/apache/flink/pull/3118

[FLINK-5224] [table] Improve UDTF: emit rows directly instead of buffering 
them

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


This PR is an improvement for user-defined table function in Flink TableAPI 
& SQL. I didn't add new tests because the existing tests are sufficient.

The core design is code generating a `Collector` and register it into the 
instance of `TableFunction`, and emit the rows generated by the UDTF directly 
instead of buffering them.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wuchong/flink udtf-collector-FLINK-5224

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3118


commit 5e126a97bb26654048efa735c21e0daff136a6a2
Author: Jark Wu 
Date:   2017-01-13T13:53:49Z

[FLINK-5224] [table] Improve UDTF: emit rows directly instead of buffering 
them




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5465) RocksDB fails with segfault while calling AbstractRocksDBState.clear()

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821827#comment-15821827
 ] 

Stephan Ewen commented on FLINK-5465:
-

+1 for setting it to won't fix for now and properly fixing it in the next 
release when we get rid of multi-threading in the StreamTask

> RocksDB fails with segfault while calling AbstractRocksDBState.clear()
> --
>
> Key: FLINK-5465
> URL: https://issues.apache.org/jira/browse/FLINK-5465
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
> Attachments: hs-err-pid26662.log
>
>
> I'm using Flink 699f4b0.
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f91a0d49b78, pid=26662, tid=140263356024576
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni-linux64.so+0x1aeb78]  
> rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /yarn/nm/usercache/robert/appcache/application_1484132267957_0007/container_1484132267957_0007_01_10/hs_err_pid26662.log
> Compiled method (nm) 1869778  903 n   org.rocksdb.RocksDB::remove 
> (native)
>  total in heap  [0x7f91b40b9dd0,0x7f91b40ba150] = 896
>  relocation [0x7f91b40b9ef0,0x7f91b40b9f48] = 88
>  main code  [0x7f91b40b9f60,0x7f91b40ba150] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.sun.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-5468) Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with ClassNotFoundException

[Stefan Richter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Richter reassigned FLINK-5468:
-

Assignee: Stefan Richter

> Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with 
> ClassNotFoundException
> ---
>
> Key: FLINK-5468
> URL: https://issues.apache.org/jira/browse/FLINK-5468
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Stefan Richter
>
> I think we should catch this exception and explain what's going on and how 
> users can resolve the issue.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>   at 
> org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
>   at 
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>   at com.dataartisans.eventwindow.Generator.main(Generator.java:60)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
>   at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:328)
>   at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:382)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
>   ... 22 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend$FinalSemiAsyncSnapshot
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:162)
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:70)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepoint(SavepointStore.java:138)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:64)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1348)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at 

[jira] [Commented] (FLINK-5465) RocksDB fails with segfault while calling AbstractRocksDBState.clear()

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821808#comment-15821808
 ] 

Stefan Richter commented on FLINK-5465:
---

After talking to [~StephanEwen], we came to the conclusion that this can be 
tolerated as known issue as it can only happen at the very end of a task's 
lifecycle and is very hard to change without potential problems for fast task 
cancelation or general state performance.

Can I set this to won't fix?

> RocksDB fails with segfault while calling AbstractRocksDBState.clear()
> --
>
> Key: FLINK-5465
> URL: https://issues.apache.org/jira/browse/FLINK-5465
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
> Attachments: hs-err-pid26662.log
>
>
> I'm using Flink 699f4b0.
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f91a0d49b78, pid=26662, tid=140263356024576
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni-linux64.so+0x1aeb78]  
> rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /yarn/nm/usercache/robert/appcache/application_1484132267957_0007/container_1484132267957_0007_01_10/hs_err_pid26662.log
> Compiled method (nm) 1869778  903 n   org.rocksdb.RocksDB::remove 
> (native)
>  total in heap  [0x7f91b40b9dd0,0x7f91b40ba150] = 896
>  relocation [0x7f91b40b9ef0,0x7f91b40b9f48] = 88
>  main code  [0x7f91b40b9f60,0x7f91b40ba150] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.sun.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5468) Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with ClassNotFoundException

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821807#comment-15821807
 ] 

Stefan Richter commented on FLINK-5468:
---

Migrating from semi async snapshotting will not be supported in Flink 1.2. This 
is a known issue that was also communicated in release notes. I agree, the 
error message could be improved here.

> Restoring from a semi async rocksdb statebackend (1.1) to 1.2 fails with 
> ClassNotFoundException
> ---
>
> Key: FLINK-5468
> URL: https://issues.apache.org/jira/browse/FLINK-5468
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>
> I think we should catch this exception and explain what's going on and how 
> users can resolve the issue.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>   at 
> org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
>   at 
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>   at com.dataartisans.eventwindow.Generator.main(Generator.java:60)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed
>   at 
> org.apache.flink.runtime.client.JobClient.awaitJobResult(JobClient.java:328)
>   at 
> org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:382)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
>   ... 22 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend$FinalSemiAsyncSnapshot
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:162)
>   at 
> org.apache.flink.migration.runtime.checkpoint.savepoint.SavepointV0Serializer.deserialize(SavepointV0Serializer.java:70)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.loadSavepoint(SavepointStore.java:138)
>   at 
> org.apache.flink.runtime.checkpoint.savepoint.SavepointLoader.loadAndValidateSavepoint(SavepointLoader.java:64)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:1348)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:1330)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> 

[jira] [Commented] (FLINK-2662) CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators."

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821800#comment-15821800
 ] 

ASF GitHub Bot commented on FLINK-2662:
---

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/3083
  
Seems there is a long story under this issue. I will try to catch up all 
the histories and have a look at this pr.


> CompilerException: "Bug: Plan generation for Unions picked a ship strategy 
> between binary plan operators."
> --
>
> Key: FLINK-2662
> URL: https://issues.apache.org/jira/browse/FLINK-2662
> Project: Flink
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 0.9.1, 0.10.0
>Reporter: Gabor Gevay
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.0.0, 1.2.0, 1.3.0, 1.1.5
>
> Attachments: Bug.java, FlinkBug.scala
>
>
> I have a Flink program which throws the exception in the jira title. Full 
> text:
> Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: 
> Plan generation for Unions picked a ship strategy between binary plan 
> operators.
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>   at 
> org.apache.flink.optimizer.plan.WorksetIterationPlanNode.acceptForStepFunction(WorksetIterationPlanNode.java:194)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:49)
>   at 
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.preVisit(BinaryUnionReplacer.java:41)
>   at 
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:162)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>   at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>   at 
> org.apache.flink.client.LocalExecutor.getOptimizerPlanAsJSON(LocalExecutor.java:202)
>   at 
> org.apache.flink.api.java.LocalEnvironment.getExecutionPlan(LocalEnvironment.java:63)
>   at malom.Solver.main(Solver.java:66)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> The execution plan:
> http://compalg.inf.elte.hu/~ggevay/flink/plan_3_4_0_0_without_verif.txt
> (I obtained this by commenting out the line that throws the exception)
> The code is here:
> https://github.com/ggevay/flink/tree/plan-generation-bug
> The class to run is "Solver". It needs a command line argument, which is a 
> directory where it would write output. (On first run, it generates some 
> lookuptables for a few minutes, which are then placed to /tmp/movegen)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3083: [FLINK-2662] [optimizer] Fix translation of broadcasted u...

[KurtYoung]

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/3083
  
Seems there is a long story under this issue. I will try to catch up all 
the histories and have a look at this pr.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5438) Typo in JobGraph generator Exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821763#comment-15821763
 ] 

ASF GitHub Bot commented on FLINK-5438:
---

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/3098
  
Hi @mtunique , thanks for your effort.
Can't see any reason we shouldn't merge this.


> Typo in JobGraph generator Exception 
> -
>
> Key: FLINK-5438
> URL: https://issues.apache.org/jira/browse/FLINK-5438
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Reporter: Ufuk Celebi
>Priority: Trivial
>
> When trying to run a job with parallelism  > max parallelism there is a typo 
> in the error message:
> {code}
> Caused by: java.lang.IllegalStateException: The maximum parallelism (1) of 
> the stream node Flat Map-3 is smaller than the parallelism (18). Increase the 
> maximum parallelism or decrease the parallelism >>>ofthis<<< operator.
> at 
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobVertex(StreamingJobGraphGenerator.java:318)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3098: [FLINK-5438] Typo in JobGraph generator Exception

[KurtYoung]

Github user KurtYoung commented on the issue:

https://github.com/apache/flink/pull/3098
  
Hi @mtunique , thanks for your effort.
Can't see any reason we shouldn't merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-5283) InputStream should be closed in RocksDBKeyedStateBackend#restoreOldSavepointKeyedState

[Stefan Richter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Richter closed FLINK-5283.
-
Resolution: Fixed

> InputStream should be closed in 
> RocksDBKeyedStateBackend#restoreOldSavepointKeyedState
> --
>
> Key: FLINK-5283
> URL: https://issues.apache.org/jira/browse/FLINK-5283
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stefan Richter
>Priority: Minor
>
> {code}
>HashMap namedStates =
> 
> InstantiationUtil.deserializeObject(restoreState.iterator().next().openInputStream(),
>  userCodeClassLoader);
> {code}
> The InputStream returned by openInputStream() should be closed upon return 
> from the method.
> Similar problem exists in HeapKeyedStateBackend#restoreOldSavepointKeyedState



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4956) Introduce convenient methods for testing checkpoint/restore in StreamTaskTestHarness

[Stefan Richter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Richter closed FLINK-4956.
-
Resolution: Implemented

> Introduce convenient methods for testing checkpoint/restore in 
> StreamTaskTestHarness
> 
>
> Key: FLINK-4956
> URL: https://issues.apache.org/jira/browse/FLINK-4956
> Project: Flink
>  Issue Type: Test
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> Methods for testing of checkpoint/restore cycles in StreamTaskTestHarness 
> could make some tests easier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5265) Introduce state handle replication mode for CheckpointCoordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821749#comment-15821749
 ] 

ASF GitHub Bot commented on FLINK-5265:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/2948#discussion_r95991932
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateCheckpointOutputStream.java
 ---
@@ -66,8 +66,10 @@ OperatorStateHandle closeAndGetHandle() throws 
IOException {
startNewPartition();
}
 
-   Map offsetsMap = new HashMap<>(1);
-   
offsetsMap.put(DefaultOperatorStateBackend.DEFAULT_OPERATOR_STATE_NAME, 
partitionOffsets.toArray());
+   Map offsetsMap = new 
HashMap<>(1);
+   OperatorStateHandle.StateMetaInfo metaInfo =
+   new 
OperatorStateHandle.StateMetaInfo(partitionOffsets.toArray(), 
OperatorStateHandle.Mode.BROADCAST);
--- End diff --

Yes, that is correct :-). Will change this as suggested.


> Introduce state handle replication mode for CheckpointCoordinator
> -
>
> Key: FLINK-5265
> URL: https://issues.apache.org/jira/browse/FLINK-5265
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> Currently, the {{CheckpointCoordinator}} only supports repartitioning of 
> {{OperatorStateHandle}}s based on a split-and-distribute strategy. For future 
> state types, such as broadcast or union state, we need a different 
> repartitioning method that allows for replicating state handles to all 
> subtasks.
> This is the first step on the way to implementing broadcast and union states.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3086: Improve docker setup

[kaelumania]

Github user kaelumania commented on the issue:

https://github.com/apache/flink/pull/3086
  
@mxm thanks :) needs there anything to be done, until this PR can be merged?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...

[StefanRRichter]

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/2948#discussion_r95991932
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateCheckpointOutputStream.java
 ---
@@ -66,8 +66,10 @@ OperatorStateHandle closeAndGetHandle() throws 
IOException {
startNewPartition();
}
 
-   Map offsetsMap = new HashMap<>(1);
-   
offsetsMap.put(DefaultOperatorStateBackend.DEFAULT_OPERATOR_STATE_NAME, 
partitionOffsets.toArray());
+   Map offsetsMap = new 
HashMap<>(1);
+   OperatorStateHandle.StateMetaInfo metaInfo =
+   new 
OperatorStateHandle.StateMetaInfo(partitionOffsets.toArray(), 
OperatorStateHandle.Mode.BROADCAST);
--- End diff --

Yes, that is correct :-). Will change this as suggested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5480) User-provided hashes for operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821745#comment-15821745
 ] 

ASF GitHub Bot commented on FLINK-5480:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3117
  
cc @uce 


> User-provided hashes for operators
> --
>
> Key: FLINK-5480
> URL: https://issues.apache.org/jira/browse/FLINK-5480
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> We could allow users to provided (alternative) hashes for operators in a 
> StreamGraph. This can make migration between Flink versions easier, in case 
> the automatically produced hashes between versions are incompatible. For 
> example, users could just copy the old hashes from the web ui to their job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3117: [FLINK-5480] Introduce user-provided hash for JobV...

[StefanRRichter]

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/3117

[FLINK-5480] Introduce user-provided hash for JobVertexes

This PR allows users to provided (alternative) hashes for operators in a 
StreamGraph. This can make migration between Flink versions easier, in case the 
automatically produced hashes between versions are incompatible. For example, 
users could just copy the old hashes from the web ui to their job.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink UserProvidedHash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3117


commit 96ef2c041bfe33462600d72b8ec1472f53c852f2
Author: Stefan Richter 
Date:   2017-01-12T17:57:52Z

[FLINK-5480] Introduce user-provided hash for JobVertexes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5480) User-provided hashes for operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821744#comment-15821744
 ] 

ASF GitHub Bot commented on FLINK-5480:
---

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/3117

[FLINK-5480] Introduce user-provided hash for JobVertexes

This PR allows users to provided (alternative) hashes for operators in a 
StreamGraph. This can make migration between Flink versions easier, in case the 
automatically produced hashes between versions are incompatible. For example, 
users could just copy the old hashes from the web ui to their job.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink UserProvidedHash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3117


commit 96ef2c041bfe33462600d72b8ec1472f53c852f2
Author: Stefan Richter 
Date:   2017-01-12T17:57:52Z

[FLINK-5480] Introduce user-provided hash for JobVertexes




> User-provided hashes for operators
> --
>
> Key: FLINK-5480
> URL: https://issues.apache.org/jira/browse/FLINK-5480
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> We could allow users to provided (alternative) hashes for operators in a 
> StreamGraph. This can make migration between Flink versions easier, in case 
> the automatically produced hashes between versions are incompatible. For 
> example, users could just copy the old hashes from the web ui to their job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3117: [FLINK-5480] Introduce user-provided hash for JobVertexes

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3117
  
cc @uce 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5480) User-provided hashes for operators

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-5480:
-

 Summary: User-provided hashes for operators
 Key: FLINK-5480
 URL: https://issues.apache.org/jira/browse/FLINK-5480
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Affects Versions: 1.2.0
Reporter: Stefan Richter
Assignee: Stefan Richter


We could allow users to provided (alternative) hashes for operators in a 
StreamGraph. This can make migration between Flink versions easier, in case the 
automatically produced hashes between versions are incompatible. For example, 
users could just copy the old hashes from the web ui to their job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5464) MetricQueryService throws NullPointerException on JobManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821684#comment-15821684
 ] 

ASF GitHub Bot commented on FLINK-5464:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3103
  
cc @rmetzger 


> MetricQueryService throws NullPointerException on JobManager
> 
>
> Key: FLINK-5464
> URL: https://issues.apache.org/jira/browse/FLINK-5464
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> I'm using Flink 699f4b0.
> My JobManager log contains many of these log entries:
> {code}
> 2017-01-11 19:42:05,778 WARN  
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher - Fetching 
> metrics failed.
> akka.pattern.AskTimeoutException: Ask timed out on 
> [Actor[akka://flink/user/MetricQueryService#-970662317]] after [1 ms]
>   at 
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>   at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
>   at 
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
>   at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-01-11 19:42:07,765 WARN  
> org.apache.flink.runtime.metrics.dump.MetricQueryService  - An exception 
> occurred while processing a message.
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:162)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$300(MetricDumpSerialization.java:47)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:90)
>   at 
> org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:109)
>   at 
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
>   at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3103: [FLINK-5464] [metrics] Prevent some NPEs

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3103
  
cc @rmetzger 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-3849) Add FilterableTableSource interface and translation rule

[Anton Solovev (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821676#comment-15821676
 ] 

Anton Solovev edited comment on FLINK-3849 at 1/13/17 11:41 AM:


Hi [~fhueske], so {{setPredicate(predicate: Expression)}} returns unsupported 
expression. What if predicate looks like {{id > 2 || (amount * 2) > anydata}} 
and {{id > 2}} part may be pushed into a source. Should we make sure that a 
source wouldn't do any logic mistakes before? And how,maybe add another method 
ensuring filtering or try to set predicate and a source make all checks.


was (Author: tonycox):
Hi [~fhueske], so {{setPredicate(predicate: Expression)}} returns unsupported 
expression. What if predicate looks like {{id > 2 || (amount * 2) > anydata}} 
and {{id > 2}} part may be pushed into a source ? Should we make sure that a 
source wouldn't do any logic mistakes before? And how, add another method 
ensuring filtering or try to set predicate and a source make all checks?

> Add FilterableTableSource interface and translation rule
> 
>
> Key: FLINK-3849
> URL: https://issues.apache.org/jira/browse/FLINK-3849
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Anton Solovev
>
> Add a {{FilterableTableSource}} interface for {{TableSource}} implementations 
> which support filter push-down.
> The interface could look as follows
> {code}
> def trait FilterableTableSource {
>   // returns unsupported predicate expression
>   def setPredicate(predicate: Expression): Expression
> }
> {code}
> In addition we need Calcite rules to push a predicate (or parts of it) into a 
> TableScan that refers to a {{FilterableTableSource}}. We might need to tweak 
> the cost model as well to push the optimizer in the right direction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3849) Add FilterableTableSource interface and translation rule

[Anton Solovev (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821676#comment-15821676
 ] 

Anton Solovev commented on FLINK-3849:
--

Hi [~fhueske], so {{setPredicate(predicate: Expression)}} returns unsupported 
expression. What if predicate looks like {{id > 2 || (amount * 2) > anydata}} 
and {{id > 2}} part may be pushed into a source ? Should we make sure that a 
source wouldn't do any logic mistakes before? And how, add another method 
ensuring filtering or try to set predicate and a source make all checks?

> Add FilterableTableSource interface and translation rule
> 
>
> Key: FLINK-3849
> URL: https://issues.apache.org/jira/browse/FLINK-3849
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Anton Solovev
>
> Add a {{FilterableTableSource}} interface for {{TableSource}} implementations 
> which support filter push-down.
> The interface could look as follows
> {code}
> def trait FilterableTableSource {
>   // returns unsupported predicate expression
>   def setPredicate(predicate: Expression): Expression
> }
> {code}
> In addition we need Calcite rules to push a predicate (or parts of it) into a 
> TableScan that refers to a {{FilterableTableSource}}. We might need to tweak 
> the cost model as well to push the optimizer in the right direction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821673#comment-15821673
 ] 

ASF GitHub Bot commented on FLINK-5432:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3090
  
looks good to me, adding it to my next batch,


> ContinuousFileMonitoringFunction is not monitoring nested files
> ---
>
> Key: FLINK-5432
> URL: https://issues.apache.org/jira/browse/FLINK-5432
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.2.0
>Reporter: Yassine Marzougui
>Assignee: Yassine Marzougui
> Fix For: 1.2.0, 1.3.0
>
>
> The {{ContinuousFileMonitoringFunction}} does not monitor nested files even 
> if the inputformat has NestedFileEnumeration set to true. This can be fixed 
> by enabling a recursive scan of the directories in the {{listEligibleFiles}} 
> method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3090: [FLINK-5432] Fix nested files enumeration in ContinuousFi...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3090
  
looks good to me, adding it to my next batch,


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4396) GraphiteReporter class not found at startup of jobmanager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821668#comment-15821668
 ] 

ASF GitHub Bot commented on FLINK-4396:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/3116

[docs} [metrics] Update metrics documentation

This PR makes several changes to the metrics documentation.

Changes include
* clearly document that reporter jars must be placed in the /lib folder, 
and document the /opt folder containing jars for flink reporters (FLINK-4396)
* document that multiple reporters exist
* extend the JMX documentation regarding domain and key-property list 
(FLINK-5451)
* document metrics visualiuation in web-frontend
* rework the system metrics section to be more readable, and add missing 
metrics (FLINK-5446)

Here's a before/after of the system-metrics section:
Before:

![before](https://cloud.githubusercontent.com/assets/5725237/21928711/9f47ef08-d98c-11e6-8f22-9029f4f4c10f.png)

After:

![after](https://cloud.githubusercontent.com/assets/5725237/21928715/a318b2ac-d98c-11e6-8d54-792b80a351c7.png)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink docs_metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3116


commit 9437a073f988eda789cb992d05bfe83520ce72d7
Author: zentol 
Date:   2017-01-11T14:09:26Z

[FLINK-4396] [docs] Document /opt reporter jars

commit 395693c50a09a2575d9325e3fd99ca625856c4b6
Author: zentol 
Date:   2017-01-13T11:16:29Z

[docs] Document reporter plurality

commit f0a6abf30555d8c60cec3324f0f93c8d966df508
Author: zentol 
Date:   2017-01-13T11:17:20Z

[FLINK-5451] Extend JMX reporter section

commit 03ac8f602af8bcb30a59b29a3a7255e65e2bcd31
Author: zentol 
Date:   2017-01-13T11:18:01Z

[docs] Document metrics visualization in web-frontend

commit ca9c3bad269d3b1764ff5ef4686065fce5b14898
Author: zentol 
Date:   2017-01-13T11:18:34Z

[FLINK-5446] [docs] Rework system-metrics section




> GraphiteReporter class not found at startup of jobmanager
> -
>
> Key: FLINK-4396
> URL: https://issues.apache.org/jira/browse/FLINK-4396
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System, Metrics
>Affects Versions: 1.1.1
> Environment: Windows and Unix
>Reporter: RWenden
>Assignee: Chesnay Schepler
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> For Flink 1.1.1 we configured Graphite metrics settings on the 
> flink-conf.yaml (for job manager (and taskmanager)).
> We see the following error in the log:
> 2016-08-15 14:20:34,167 ERROR org.apache.flink.runtime.metrics.MetricRegistry 
>   - Could not instantiate metrics reportermy_reporter. Metrics 
> might not be exposed/reported.
> java.lang.ClassNotFoundException: 
> org.apache.flink.metrics.graphite.GraphiteReporter
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.flink.runtime.metrics.MetricRegistry.(MetricRegistry.java:119)
> We found out that this class is not packaged inside flink-dist_2.11-1.1.1.jar.
> Long story short: we had to install/provide the following jars into the lib 
> folder to make Graphite metrics to work:
> flink-metrics-graphite-1.1.1.jar
> flink-metrics-dropwizard-1.1.1.jar
> metrics-graphite-3.1.0.jar (from dropwizard)
> We think these libraries (and the ones for Ganglia,StatsD,...) should be 
> included in flink-dist_2.11-1.1.1.jar, for these are needed at manager 
> startup time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3116: [docs} [metrics] Update metrics documentation

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/3116

[docs} [metrics] Update metrics documentation

This PR makes several changes to the metrics documentation.

Changes include
* clearly document that reporter jars must be placed in the /lib folder, 
and document the /opt folder containing jars for flink reporters (FLINK-4396)
* document that multiple reporters exist
* extend the JMX documentation regarding domain and key-property list 
(FLINK-5451)
* document metrics visualiuation in web-frontend
* rework the system metrics section to be more readable, and add missing 
metrics (FLINK-5446)

Here's a before/after of the system-metrics section:
Before:

![before](https://cloud.githubusercontent.com/assets/5725237/21928711/9f47ef08-d98c-11e6-8f22-9029f4f4c10f.png)

After:

![after](https://cloud.githubusercontent.com/assets/5725237/21928715/a318b2ac-d98c-11e6-8d54-792b80a351c7.png)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink docs_metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3116


commit 9437a073f988eda789cb992d05bfe83520ce72d7
Author: zentol 
Date:   2017-01-11T14:09:26Z

[FLINK-4396] [docs] Document /opt reporter jars

commit 395693c50a09a2575d9325e3fd99ca625856c4b6
Author: zentol 
Date:   2017-01-13T11:16:29Z

[docs] Document reporter plurality

commit f0a6abf30555d8c60cec3324f0f93c8d966df508
Author: zentol 
Date:   2017-01-13T11:17:20Z

[FLINK-5451] Extend JMX reporter section

commit 03ac8f602af8bcb30a59b29a3a7255e65e2bcd31
Author: zentol 
Date:   2017-01-13T11:18:01Z

[docs] Document metrics visualization in web-frontend

commit ca9c3bad269d3b1764ff5ef4686065fce5b14898
Author: zentol 
Date:   2017-01-13T11:18:34Z

[FLINK-5446] [docs] Rework system-metrics section




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4988) Elasticsearch 5.x support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821662#comment-15821662
 ] 

ASF GitHub Bot commented on FLINK-4988:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3112#discussion_r95979696
  
--- Diff: .travis.yml ---
@@ -16,14 +16,14 @@ matrix:
   include:
   # Always run test groups A and B together
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
--- End diff --

With the automatic activation, you don't need these changes (see my other 
comment at the profile def)


> Elasticsearch 5.x support
> -
>
> Key: FLINK-4988
> URL: https://issues.apache.org/jira/browse/FLINK-4988
> Project: Flink
>  Issue Type: New Feature
>Reporter: Mike Dias
>
> Elasticsearch 5.x was released: 
> https://www.elastic.co/blog/elasticsearch-5-0-0-released



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4988) Elasticsearch 5.x support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821661#comment-15821661
 ] 

ASF GitHub Bot commented on FLINK-4988:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3112#discussion_r95979634
  
--- Diff: flink-connectors/pom.xml ---
@@ -85,6 +86,17 @@ under the License.
flink-connector-kinesis


+
+   
+   
+   include-elasticsearch5
--- End diff --

Can you make this a profile that activates itself automatically when java 8 
is available?
http://maven.apache.org/guides/introduction/introduction-to-profiles.html


> Elasticsearch 5.x support
> -
>
> Key: FLINK-4988
> URL: https://issues.apache.org/jira/browse/FLINK-4988
> Project: Flink
>  Issue Type: New Feature
>Reporter: Mike Dias
>
> Elasticsearch 5.x was released: 
> https://www.elastic.co/blog/elasticsearch-5-0-0-released



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

[rmetzger]

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3112#discussion_r95979696
  
--- Diff: .travis.yml ---
@@ -16,14 +16,14 @@ matrix:
   include:
   # Always run test groups A and B together
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.7.2 -Dscala-2.11 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-a,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
 - jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis 
-Dmaven.javadoc.skip=true"
+  env: PROFILE="-Dhadoop.version=2.6.3 
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis,include-elasticsearch5 
-Dmaven.javadoc.skip=true"
--- End diff --

With the automatic activation, you don't need these changes (see my other 
comment at the profile def)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

[rmetzger]

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3112#discussion_r95979634
  
--- Diff: flink-connectors/pom.xml ---
@@ -85,6 +86,17 @@ under the License.
flink-connector-kinesis


+
+   
+   
+   include-elasticsearch5
--- End diff --

Can you make this a profile that activates itself automatically when java 8 
is available?
http://maven.apache.org/guides/introduction/introduction-to-profiles.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821660#comment-15821660
 ] 

ASF GitHub Bot commented on FLINK-5432:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3090
  
@zentol do you want to recheck and merge if good?


> ContinuousFileMonitoringFunction is not monitoring nested files
> ---
>
> Key: FLINK-5432
> URL: https://issues.apache.org/jira/browse/FLINK-5432
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.2.0
>Reporter: Yassine Marzougui
>Assignee: Yassine Marzougui
> Fix For: 1.2.0, 1.3.0
>
>
> The {{ContinuousFileMonitoringFunction}} does not monitor nested files even 
> if the inputformat has NestedFileEnumeration set to true. This can be fixed 
> by enabling a recursive scan of the directories in the {{listEligibleFiles}} 
> method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3090: [FLINK-5432] Fix nested files enumeration in ContinuousFi...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3090
  
@zentol do you want to recheck and merge if good?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---