[jira] [Commented] (FLINK-3667) Generalize client<->cluster communication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305118#comment-15305118
 ] 

ASF GitHub Bot commented on FLINK-3667:
---

Github user EronWright commented on the pull request:

https://github.com/apache/flink/pull/1978#issuecomment-85524
  
This PR dovetails nicely with the Mesos work and I'll be sure to build on 
it. Here's a few suggestions to align it even further.

The problem of _managing_ a Flink cluster is mostly independent from 
_using_ a cluster to submit and manage jobs.I would like to see the two 
concerns be cleanly separated.In this PR, the `ClusterDescriptor` handles 
creating the cluster, then produces a `Client` with which to manage jobs and to 
handle shutdown.I suggest that a new component - the `YarnDispatcher` - be 
introduced to handle all lifecycle operations for a cluster.   Make the 
`ClusterDescriptor` be an entity class that is given to the dispatcher.  

A related issue is that its only possible to use the `YarnClusterClient` to 
interact with a newly-created YARN session, not a pre-existing one.   When 
submitting a job to an existing YARN session, seems the 
`StandaloneClusterClient` is used (by supplying a JM endpoint) - is that true?

Eventually the CLI should provide a nice way to discover and use existing 
YARN sessions.

The `detached` flags could use clarification.   In the `Client` context, 
the detached concept seems related to interactivity with the job (tailing the 
status messages, etc).   I don't think it should imply anything about the 
lifecycle of the cluster; leave that to the dispatcher.   The `stopAfterJob` 
method should move accordingly to the dispatcher.

How this relates to Mesos is, the `MesosDispatcher` component will run in 
the Mesos cluster and be accessed remotely by the CLI.The 
`ClusterDescriptor` will be passed via REST to it.   Everything will fit 
nicely.  :)



> Generalize client<->cluster communication
> -
>
> Key: FLINK-3667
> URL: https://issues.apache.org/jira/browse/FLINK-3667
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with 
> regard to future integration of other resource management frameworks in 
> addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
> 
> 1.1 Status Quo
> ──
> 1.1.1 FlinkYarnClient
> ╌
>   • Holds the cluster configuration (Flink-specific and Yarn-specific)
>   • Contains the deploy() method to deploy the cluster
>   • Creates the Hadoop Yarn client
>   • Receives the initial job manager address
>   • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌
>   • Wrapper around the Hadoop Yarn client
>   • Queries cluster for status updates
>   • Life time methods to start and shutdown the cluster
>   • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌
>   • Acts as a middle-man for asynchronous cluster communication
>   • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌
>   • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
>   • Constantly distinguishes between Yarn and Standalone mode
>   • Would be nice to have a general abstraction in place
> 1.1.5 Client
> 
>   • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
> 
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌
>   • Extensible cluster-agnostic config
>   • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌
>   • Deals with cluster (RM) specific communication
>   • Exposes framework agnostic information
>   • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌
>   • Basic interface to communicate with a running cluster
>   • Receives the ClusterClient for cluster-specific communication
>   • Should not have to care about the specific implementations of the
> client
> 1.2.4 ApplicationClient
> ╌╌╌
>   • Can be changed to work cluster-agnostic (first steps already in
> FLINK-3543)
> 1.2.5 CliFrontend
> ╌
>   • CliFrontend does never have to differentiate between different
> cluster types after it has determined which cluster class to load.
>   • Base class 

[GitHub] flink pull request: [FLINK-3667] refactor client communication

[EronWright]

Github user EronWright commented on the pull request:

https://github.com/apache/flink/pull/1978#issuecomment-85524
  
This PR dovetails nicely with the Mesos work and I'll be sure to build on 
it. Here's a few suggestions to align it even further.

The problem of _managing_ a Flink cluster is mostly independent from 
_using_ a cluster to submit and manage jobs.I would like to see the two 
concerns be cleanly separated.In this PR, the `ClusterDescriptor` handles 
creating the cluster, then produces a `Client` with which to manage jobs and to 
handle shutdown.I suggest that a new component - the `YarnDispatcher` - be 
introduced to handle all lifecycle operations for a cluster.   Make the 
`ClusterDescriptor` be an entity class that is given to the dispatcher.  

A related issue is that its only possible to use the `YarnClusterClient` to 
interact with a newly-created YARN session, not a pre-existing one.   When 
submitting a job to an existing YARN session, seems the 
`StandaloneClusterClient` is used (by supplying a JM endpoint) - is that true?

Eventually the CLI should provide a nice way to discover and use existing 
YARN sessions.

The `detached` flags could use clarification.   In the `Client` context, 
the detached concept seems related to interactivity with the job (tailing the 
status messages, etc).   I don't think it should imply anything about the 
lifecycle of the cluster; leave that to the dispatcher.   The `stopAfterJob` 
method should move accordingly to the dispatcher.

How this relates to Mesos is, the `MesosDispatcher` component will run in 
the Mesos cluster and be accessed remotely by the CLI.The 
`ClusterDescriptor` will be passed via REST to it.   Everything will fit 
nicely.  :)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3984) Event time of stream transformations is undocumented

[Elias Levy (JIRA)]

Elias Levy created FLINK-3984:
-

 Summary: Event time of stream transformations is undocumented
 Key: FLINK-3984
 URL: https://issues.apache.org/jira/browse/FLINK-3984
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.0.3
Reporter: Elias Levy


The Event Time, Windowing, and DataStream Transformation documentation section 
fail to state what event time, if any, the output of transformations have on a 
stream that is configured to use event time and that has timestamp assigners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3715) Move Accumulating/Discarding from Trigger to WindowOperator

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305082#comment-15305082
 ] 

Elias Levy commented on FLINK-3715:
---

As mentioned on the Google doc, this may not be flexible enough.  If you have 
windows where you are implementing early firing using multiple triggers, you 
will want to keep the elements in the window during early firing, but purge 
them at the end of the window firing.

> Move Accumulating/Discarding from Trigger to WindowOperator
> ---
>
> Key: FLINK-3715
> URL: https://issues.apache.org/jira/browse/FLINK-3715
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> As mentioned in 
> https://docs.google.com/document/d/1Xp-YBf87vLTduYSivgqWVEMjYUmkA-hyb4muX3KRl08/edit#heading=h.pyk1sn8f33q2
>  we should move the decision of whether to {{PURGE}} a window upon firing 
> from the {{Trigger}} to the {{WindowOperator}}. This also requires to add API 
> so that the user can specify whether windows should be purged upon trigger 
> firing (discarding) or kept (accumulating).
> As mentioned in the above doc, the {{Trigger}} can react with 4 results right 
> now: {{CONTINUE}}, {{FIRE}}, {{PURGE}}, {{FIRE_AND_PURGE}}. The behavior of a 
> trigger is not apparent if not looking at the code of the trigger, this has 
> confused a number of users. With the new regime, a {{Trigger}} can just 
> decide whether to {{CONTINUE}} or {{FIRE}}. The setting of 
> accumulating/discarding decides whether to purge the window or keep it.
> This depends on FLINK-3714 where we introduce an "allowed lateness" setting. 
> Having a choice between accumulating and discarding only makes sense with an 
> allowed lateness greater zero. Otherwise no late elements could ever arrive 
> that would go into the kept windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3667) Generalize client<->cluster communication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304946#comment-15304946
 ] 

ASF GitHub Bot commented on FLINK-3667:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/1978#discussion_r64975479
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -393,7 +373,7 @@ protected AbstractFlinkYarnCluster deployInternal() 
throws Exception {
}
}
 
-   // -- Check if the YARN Cluster has the 
requested resources --
+   // -- Check if the YARN ClusterClient has the 
requested resources --
--- End diff --

should remain "Cluster"


> Generalize client<->cluster communication
> -
>
> Key: FLINK-3667
> URL: https://issues.apache.org/jira/browse/FLINK-3667
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with 
> regard to future integration of other resource management frameworks in 
> addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
> 
> 1.1 Status Quo
> ──
> 1.1.1 FlinkYarnClient
> ╌
>   • Holds the cluster configuration (Flink-specific and Yarn-specific)
>   • Contains the deploy() method to deploy the cluster
>   • Creates the Hadoop Yarn client
>   • Receives the initial job manager address
>   • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌
>   • Wrapper around the Hadoop Yarn client
>   • Queries cluster for status updates
>   • Life time methods to start and shutdown the cluster
>   • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌
>   • Acts as a middle-man for asynchronous cluster communication
>   • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌
>   • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
>   • Constantly distinguishes between Yarn and Standalone mode
>   • Would be nice to have a general abstraction in place
> 1.1.5 Client
> 
>   • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
> 
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌
>   • Extensible cluster-agnostic config
>   • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌
>   • Deals with cluster (RM) specific communication
>   • Exposes framework agnostic information
>   • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌
>   • Basic interface to communicate with a running cluster
>   • Receives the ClusterClient for cluster-specific communication
>   • Should not have to care about the specific implementations of the
> client
> 1.2.4 ApplicationClient
> ╌╌╌
>   • Can be changed to work cluster-agnostic (first steps already in
> FLINK-3543)
> 1.2.5 CliFrontend
> ╌
>   • CliFrontend does never have to differentiate between different
> cluster types after it has determined which cluster class to load.
>   • Base class handles framework agnostic command line arguments
>   • Pluggables for Yarn, Mesos handle specific commands
> {noformat}
> I would like to create/refactor the affected classes to set us up for a more 
> flexible client side resource management abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3667] refactor client communication

[EronWright]

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/1978#discussion_r64975479
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -393,7 +373,7 @@ protected AbstractFlinkYarnCluster deployInternal() 
throws Exception {
}
}
 
-   // -- Check if the YARN Cluster has the 
requested resources --
+   // -- Check if the YARN ClusterClient has the 
requested resources --
--- End diff --

should remain "Cluster"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304799#comment-15304799
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-55574
  
That was my concern actually, Let's wait for it then :)


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-55208
  
Okay, I've filed https://issues.apache.org/jira/browse/FLINK-3983. However, 
before you jump onto it: While I wrote the JIRA, I was actually wondering 
whether it makes sense now to address this, if we are going to drop the the KPL 
in the next pull request.
I hope I can make up my mind until Monday.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304798#comment-15304798
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-55208
  
Okay, I've filed https://issues.apache.org/jira/browse/FLINK-3983. However, 
before you jump onto it: While I wrote the JIRA, I was actually wondering 
whether it makes sense now to address this, if we are going to drop the the KPL 
in the next pull request.
I hope I can make up my mind until Monday.


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[aozturk]

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-55079
  
Repeated lines were terrible, it looks neater now. But it seems the if 
checks for max count configurations in FlinkKinesisProducer.java are redundant 
now. 
Adding all configurations sounds good. I thought about it but was not sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304796#comment-15304796
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-55079
  
Repeated lines were terrible, it looks neater now. But it seems the if 
checks for max count configurations in FlinkKinesisProducer.java are redundant 
now. 
Adding all configurations sounds good. I thought about it but was not sure.


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3983) Allow users to set any (relevant) configuration parameter of the KinesisProducerConfiguration

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3983:
-

 Summary: Allow users to set any (relevant) configuration parameter 
of the KinesisProducerConfiguration
 Key: FLINK-3983
 URL: https://issues.apache.org/jira/browse/FLINK-3983
 Project: Flink
  Issue Type: Sub-task
  Components: Kinesis Connector
Affects Versions: 1.1.0
Reporter: Robert Metzger


Currently, users can only set some of the configuration parameters in the 
{{KinesisProducerConfiguration}} through Properties.
It would be good to introduce configuration keys for these keys so that users 
can change the producer configuration.

I think these and most of the other variables in the 
KinesisProducerConfiguration should be exposed via properties:
- aggregationEnabled
- collectionMaxCount
- collectionMaxSize
- connectTimeout
- credentialsRefreshDelay
- failIfThrottled
- logLevel
- metricsGranularity
- metricsLevel
- metricsNamespace
- metricsUploadDelay





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304765#comment-15304765
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-52000
  
I have to admit that these changes were not necessary: 
https://github.com/apache/flink/commit/23d1cba72859339bd3ee8f877b031353380c87fb#diff-d34eb8418ccff3e2d59a3e660fed6cec
I wanted to change it so that the code won't throw any exceptions if the 
provided value is not correctly parseable. I didn't see that the verify method 
in the constructor is already doing that.

I'll add a follow up JIRA for adding configuration keys for all relevant 
fields in `KinesisProducerConfiguration`


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-52000
  
I have to admit that these changes were not necessary: 
https://github.com/apache/flink/commit/23d1cba72859339bd3ee8f877b031353380c87fb#diff-d34eb8418ccff3e2d59a3e660fed6cec
I wanted to change it so that the code won't throw any exceptions if the 
provided value is not correctly parseable. I didn't see that the verify method 
in the constructor is already doing that.

I'll add a follow up JIRA for adding configuration keys for all relevant 
fields in `KinesisProducerConfiguration`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[aozturk]

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-50998
  
Thanks for the merge! Your latest enhancements look great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304755#comment-15304755
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-50998
  
Thanks for the merge! Your latest enhancements look great.


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304743#comment-15304743
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-48940
  
Okay, let me explain:

I decided to remove the field, even though its public, and strictly 
speaking a breaking API change. However, it was a mistake to mark the field as 
public (back then, when it was added). It was only used internally as a 
configuration key for storing the serialized execution config (so it was 
something only used by flink).
It doesn't have any use for our users at all, so I think its absolutely 
safe to remove it. Its also not mentioned in the documentation.

I've added this exclude to the API check to ignore the breaking change:

https://github.com/apache/flink/commit/b0acd97935cd21843bac3b9b5afa3662b52bb95d#diff-40616c4678c3fbfe07c0701505ce0567


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.1.0
>
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-48940
  
Okay, let me explain:

I decided to remove the field, even though its public, and strictly 
speaking a breaking API change. However, it was a mistake to mark the field as 
public (back then, when it was added). It was only used internally as a 
configuration key for storing the serialized execution config (so it was 
something only used by flink).
It doesn't have any use for our users at all, so I think its absolutely 
safe to remove it. Its also not mentioned in the documentation.

I've added this exclude to the API check to ignore the breaking change:

https://github.com/apache/flink/commit/b0acd97935cd21843bac3b9b5afa3662b52bb95d#diff-40616c4678c3fbfe07c0701505ce0567


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3982) Multiple ResourceManagers register at JobManager in standalone HA mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304740#comment-15304740
 ] 

ASF GitHub Bot commented on FLINK-3982:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2046

[FLINK-3982] let only ResourceManager of leading JobManager register

In HA mode, multiple ResourceManagers may register at the leading
JobManager. They register one after another at the JobManager. The last
registering ResourceManager stays registered with the JobManager. This
only applies to Standalone mode and doesn't affect functionality.

To prevent duplicate registration for the standalone ResourceManager,
the easiest solution is to only start registration when the leading
JobManager runs in the same ActorSystem as theResourceManager. Other
ResourceManager implementations may also run independently of the
JobManager.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3982

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2046


commit 09cbb90d2a93ba0c9efb82e16f421641c10fec32
Author: Maximilian Michels 
Date:   2016-05-27T19:03:04Z

[FLINK-3982] let only ResourceManager of leading JobManager register

In HA mode, multiple ResourceManagers may register at the leading
JobManager. They register one after another at the JobManager. The last
registering ResourceManager stays registered with the JobManager. This
only applies to Standalone mode and doesn't affect functionality.

To prevent duplicate registration for the standalone ResourceManager,
the easiest solution is to only start registration when the leading
JobManager runs in the same ActorSystem as theResourceManager. Other
ResourceManager implementations may also run independently of the
JobManager.

commit 30e58fdcdbbf211b1c0908bb2467464c1f83f07c
Author: Maximilian Michels 
Date:   2016-05-27T19:33:29Z

correct entity in String




> Multiple ResourceManagers register at JobManager in standalone HA mode
> --
>
> Key: FLINK-3982
> URL: https://issues.apache.org/jira/browse/FLINK-3982
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.1.0
>
>
> In HA mode, multiple ResourceManagers may register at the leading JobManager. 
> They register one after another at the JobManager. The last registering 
> ResourceManager stays registered with the JobManager. This only applies to 
> Standalone mode and doesn't affect functionality.
> To prevent duplicate registration for the standalone ResourceManager, the 
> easiest solution is to only start registration when the leading JobManager 
> runs in the same ActorSystem as the ResourceManager. Other ResourceManager 
> implementations may also run independently of the JobManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3982] let only ResourceManager of leadi...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2046

[FLINK-3982] let only ResourceManager of leading JobManager register

In HA mode, multiple ResourceManagers may register at the leading
JobManager. They register one after another at the JobManager. The last
registering ResourceManager stays registered with the JobManager. This
only applies to Standalone mode and doesn't affect functionality.

To prevent duplicate registration for the standalone ResourceManager,
the easiest solution is to only start registration when the leading
JobManager runs in the same ActorSystem as theResourceManager. Other
ResourceManager implementations may also run independently of the
JobManager.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3982

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2046


commit 09cbb90d2a93ba0c9efb82e16f421641c10fec32
Author: Maximilian Michels 
Date:   2016-05-27T19:03:04Z

[FLINK-3982] let only ResourceManager of leading JobManager register

In HA mode, multiple ResourceManagers may register at the leading
JobManager. They register one after another at the JobManager. The last
registering ResourceManager stays registered with the JobManager. This
only applies to Standalone mode and doesn't affect functionality.

To prevent duplicate registration for the standalone ResourceManager,
the easiest solution is to only start registration when the leading
JobManager runs in the same ActorSystem as theResourceManager. Other
ResourceManager implementations may also run independently of the
JobManager.

commit 30e58fdcdbbf211b1c0908bb2467464c1f83f07c
Author: Maximilian Michels 
Date:   2016-05-27T19:33:29Z

correct entity in String




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3981) Don't log duplicate TaskManager registrations as exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304733#comment-15304733
 ] 

ASF GitHub Bot commented on FLINK-3981:
---

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2045

[FLINK-3981] don't log duplicate TaskManager registrations

Duplicate TaskManager registrations shouldn't be logged with Exceptions
in the ResourceManager. Duplicate registrations can happen if the
TaskManager sends out registration messages too fast when the actual
reply is not lost but still in transit.

The ResourceManager should simply acknowledge the duplicate
registrations, leaving it up to the JobManager to decide how to treat
the duplicate registrations (currently it will send an AlreadyRegistered
to the TaskManager).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3981

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2045.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2045


commit 8106ab460920ccea2a96e90c3a778d7acac0c672
Author: Maximilian Michels 
Date:   2016-05-27T19:02:38Z

[FLINK-3981] don't log duplicate TaskManager registrations

Duplicate TaskManager registrations shouldn't be logged with Exceptions
in the ResourceManager. Duplicate registrations can happen if the
TaskManager sends out registration messages too fast when the actual
reply is not lost but still in transit.

The ResourceManager should simply acknowledge the duplicate
registrations, leaving it up to the JobManager to decide how to treat
the duplicate registrations (currently it will send an AlreadyRegistered
to the TaskManager).




> Don't log duplicate TaskManager registrations as exceptions
> ---
>
> Key: FLINK-3981
> URL: https://issues.apache.org/jira/browse/FLINK-3981
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> Duplicate TaskManager registrations shouldn't be logged with Exceptions in 
> the ResourceManager. Duplicate registrations can happen if the TaskManager 
> sends out registration messages too fast when the actual reply is not lost 
> but still in transit.
> The ResourceManager should simply acknowledge the duplicate registrations, 
> leaving it up to the JobManager to decide how to treat the duplicate 
> registrations (currently it will send an AlreadyRegistered to the 
> TaskManager).
> This change also affects our test stability because the Yarn tests check the 
> logs for exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304731#comment-15304731
 ] 

Robert Metzger edited comment on FLINK-3923 at 5/27/16 8:38 PM:


This has been resolved in 
http://git-wip-us.apache.org/repos/asf/flink/commit/23d1cba7 and 
http://git-wip-us.apache.org/repos/asf/flink/commit/fc6dc487


was (Author: rmetzger):
This has been resolved in 
http://git-wip-us.apache.org/repos/asf/flink/commit/23d1cba7

> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304732#comment-15304732
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2016


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3923.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

This has been resolved in 
http://git-wip-us.apache.org/repos/asf/flink/commit/23d1cba7

> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
> Fix For: 1.1.0
>
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3981] don't log duplicate TaskManager r...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/flink/pull/2045

[FLINK-3981] don't log duplicate TaskManager registrations

Duplicate TaskManager registrations shouldn't be logged with Exceptions
in the ResourceManager. Duplicate registrations can happen if the
TaskManager sends out registration messages too fast when the actual
reply is not lost but still in transit.

The ResourceManager should simply acknowledge the duplicate
registrations, leaving it up to the JobManager to decide how to treat
the duplicate registrations (currently it will send an AlreadyRegistered
to the TaskManager).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/flink FLINK-3981

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2045.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2045


commit 8106ab460920ccea2a96e90c3a778d7acac0c672
Author: Maximilian Michels 
Date:   2016-05-27T19:02:38Z

[FLINK-3981] don't log duplicate TaskManager registrations

Duplicate TaskManager registrations shouldn't be logged with Exceptions
in the ResourceManager. Duplicate registrations can happen if the
TaskManager sends out registration messages too fast when the actual
reply is not lost but still in transit.

The ResourceManager should simply acknowledge the duplicate
registrations, leaving it up to the JobManager to decide how to treat
the duplicate registrations (currently it will send an AlreadyRegistered
to the TaskManager).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2016


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3982) Multiple ResourceManagers register at JobManager in standalone HA mode

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-3982:
-

 Summary: Multiple ResourceManagers register at JobManager in 
standalone HA mode
 Key: FLINK-3982
 URL: https://issues.apache.org/jira/browse/FLINK-3982
 Project: Flink
  Issue Type: Bug
  Components: ResourceManager
Affects Versions: 1.1.0
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Priority: Minor
 Fix For: 1.1.0


In HA mode, multiple ResourceManagers may register at the leading JobManager. 
They register one after another at the JobManager. The last registering 
ResourceManager stays registered with the JobManager. This only applies to 
Standalone mode and doesn't affect functionality.

To prevent duplicate registration for the standalone ResourceManager, the 
easiest solution is to only start registration when the leading JobManager runs 
in the same ActorSystem as the ResourceManager. Other ResourceManager 
implementations may also run independently of the JobManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3981) Don't log duplicate TaskManager registrations as exceptions

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-3981:
-

 Summary: Don't log duplicate TaskManager registrations as 
exceptions
 Key: FLINK-3981
 URL: https://issues.apache.org/jira/browse/FLINK-3981
 Project: Flink
  Issue Type: Bug
  Components: ResourceManager
Affects Versions: 1.1.0
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Priority: Minor
 Fix For: 1.1.0


Duplicate TaskManager registrations shouldn't be logged with Exceptions in the 
ResourceManager. Duplicate registrations can happen if the TaskManager sends 
out registration messages too fast when the actual reply is not lost but still 
in transit.

The ResourceManager should simply acknowledge the duplicate registrations, 
leaving it up to the JobManager to decide how to treat the duplicate 
registrations (currently it will send an AlreadyRegistered to the TaskManager).

This change also affects our test stability because the Yarn tests check the 
logs for exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304422#comment-15304422
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-08597
  
no my concern wasn't even remotely addressed.
The public static String CONFIG_KEY was removed in commit 
5a7f4e3395bfb06da496584be88501c328f6ac1d, violating the interface stability.

In commit 6c079360c60f47a336eda2f8936a6bddfb948058 you add CONFIG_KEY in 
the ExecutionConfig, while the subsequent commit 
b0acd97935cd21843bac3b9b5afa3662b52bb95d removes it again.

so now the public static key is still removed which shouldn't be the case, 
no?


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.1.0
>
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[zentol]

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-08597
  
no my concern wasn't even remotely addressed.
The public static String CONFIG_KEY was removed in commit 
5a7f4e3395bfb06da496584be88501c328f6ac1d, violating the interface stability.

In commit 6c079360c60f47a336eda2f8936a6bddfb948058 you add CONFIG_KEY in 
the ExecutionConfig, while the subsequent commit 
b0acd97935cd21843bac3b9b5afa3662b52bb95d removes it again.

so now the public static key is still removed which shouldn't be the case, 
no?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2915) JobManagerProcessFailureBatchRecoveryITCase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304332#comment-15304332
 ] 

ASF GitHub Bot commented on FLINK-2915:
---

Github user rekhajoshm commented on the pull request:

https://github.com/apache/flink/pull/2027#issuecomment-222198043
  
Thanks @StephanEwen for review. Closing PR.


> JobManagerProcessFailureBatchRecoveryITCase
> ---
>
> Key: FLINK-2915
> URL: https://issues.apache.org/jira/browse/FLINK-2915
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Matthias J. Sax
>Assignee: Rekha Joshi
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> https://travis-ci.org/apache/flink/jobs/87193692
> {noformat}
> Failed tests:   
> JobManagerProcessFailureBatchRecoveryITCase>AbstractJobManagerProcessFailureRecoveryITCase.testJobManagerProcessFailure:259
>  JobManager did not start up within 291736881301 nanoseconds.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2915] Fix for JobManagerProcessFailureB...

[rekhajoshm]

Github user rekhajoshm closed the pull request at:

https://github.com/apache/flink/pull/2027


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2915) JobManagerProcessFailureBatchRecoveryITCase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304333#comment-15304333
 ] 

ASF GitHub Bot commented on FLINK-2915:
---

Github user rekhajoshm closed the pull request at:

https://github.com/apache/flink/pull/2027


> JobManagerProcessFailureBatchRecoveryITCase
> ---
>
> Key: FLINK-2915
> URL: https://issues.apache.org/jira/browse/FLINK-2915
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Matthias J. Sax
>Assignee: Rekha Joshi
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> https://travis-ci.org/apache/flink/jobs/87193692
> {noformat}
> Failed tests:   
> JobManagerProcessFailureBatchRecoveryITCase>AbstractJobManagerProcessFailureRecoveryITCase.testJobManagerProcessFailure:259
>  JobManager did not start up within 291736881301 nanoseconds.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2915] Fix for JobManagerProcessFailureB...

[rekhajoshm]

Github user rekhajoshm commented on the pull request:

https://github.com/apache/flink/pull/2027#issuecomment-222198043
  
Thanks @StephanEwen for review. Closing PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2771] Fix for IterateTest.testSimpleIte...

[rekhajoshm]

Github user rekhajoshm commented on the pull request:

https://github.com/apache/flink/pull/2028#issuecomment-222197851
  
Yes, glad to @StephanEwen . Looking forward to your design for 
timeout-based termination to a proper "end-of-stream-event" based termination. 
thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3972) Subclasses of ResourceID may not to be serializable

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels closed FLINK-3972.
-
Resolution: Fixed

Fixed via 4d41bd8fa787315d203bb34973f8608d84c5b6ac

> Subclasses of ResourceID may not to be serializable
> ---
>
> Key: FLINK-3972
> URL: https://issues.apache.org/jira/browse/FLINK-3972
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.1.0
>
>
> WorkerTypes are currently subclasses of ResourceID. ResourceID has to be 
> Serializable but its subclasses don't. This may lead to problems when these 
> subclasses are used as ResourceIDs, i.e. serialization may fail with 
> NotSerializableExceptions. Currently, subclasses are never send over the wire 
> but they might be in the future.
> Instead of relying on subclasses of ResourceID for the WorkerTypes, we can 
> let them implement an interface to retrieve the ResourceID of a WorkerType.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3972) Subclasses of ResourceID may not to be serializable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304291#comment-15304291
 ] 

ASF GitHub Bot commented on FLINK-3972:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2037


> Subclasses of ResourceID may not to be serializable
> ---
>
> Key: FLINK-3972
> URL: https://issues.apache.org/jira/browse/FLINK-3972
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.1.0
>
>
> WorkerTypes are currently subclasses of ResourceID. ResourceID has to be 
> Serializable but its subclasses don't. This may lead to problems when these 
> subclasses are used as ResourceIDs, i.e. serialization may fail with 
> NotSerializableExceptions. Currently, subclasses are never send over the wire 
> but they might be in the future.
> Instead of relying on subclasses of ResourceID for the WorkerTypes, we can 
> let them implement an interface to retrieve the ResourceID of a WorkerType.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3972] subclasses of ResourceID may not ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2037


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304243#comment-15304243
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222185242
  
@zentol I think your concern was addressed here: 
https://github.com/apache/flink/pull/2042/commits/c20efc6cda6ff705c72924213c90f618f7f40e51

Thanks all for the review!


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.1.0
>
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222185242
  
@zentol I think your concern was addressed here: 
https://github.com/apache/flink/pull/2042/commits/c20efc6cda6ff705c72924213c90f618f7f40e51

Thanks all for the review!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2042


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-3129) Add tooling to ensure interface stability

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3129.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

Resolved in current master in 
http://git-wip-us.apache.org/repos/asf/flink/commit/08eca761 and 
http://git-wip-us.apache.org/repos/asf/flink/commit/6c079360 and 
http://git-wip-us.apache.org/repos/asf/flink/commit/b0acd979

> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.1.0
>
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304238#comment-15304238
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2042


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.1.0
>
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3291) Object reuse bug in MergeIterator.HeadStream.nextHead

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304234#comment-15304234
 ] 

Greg Hogan commented on FLINK-3291:
---

The {{inputStream}} field causing the {{IllegalArgumentException}} was changed 
from package-private to protected in [0]. Could your system be using an older 
version of Kryo?

[0] 
https://github.com/EsotericSoftware/kryo/commit/642972d532e832aabfe8c9be515b8b2560f093ce#diff-d6b5559e1180e131a85fa781a469f0b2L19

> Object reuse bug in MergeIterator.HeadStream.nextHead
> -
>
> Key: FLINK-3291
> URL: https://issues.apache.org/jira/browse/FLINK-3291
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: Gabor Gevay
>Priority: Critical
>
> MergeIterator.HeadStream.nextHead saves a reference into `this.head` of the 
> `reuse` object that it got as an argument. This object might be modified 
> later by the caller.
> This actually happens when ReduceDriver.run calls input.next (which will 
> actually be MergeIterator.next(E reuse)) in the inner while loop of the 
> objectReuseEnabled branch, and that calls top.nextHead with the reference 
> that it got from ReduceDriver, which erroneously saves the reference, and 
> then ReduceDriver later uses that same object for doing the reduce.
> Another way in which this fails is when MergeIterator.next(E reuse) gives 
> `reuse` to different `top`s in different calls, and then the heads end up 
> being the same object.
> You can observe the latter situation in action by running ReducePerformance 
> here:
> https://github.com/ggevay/flink/tree/merge-iterator-object-reuse-bug
> Set memory to -Xmx200m (so that the MergeIterator actually has merging to 
> do), put a breakpoint at the beginning of MergeIterator.next(reuse), and then 
> watch `reuse`, and the heads of the first two elements of `this.heap` in the 
> debugger. They will get to be the same object after hitting continue about 6 
> times.
> You can also look at the count that is printed at the end, which shouldn't be 
> larger than the key range. Also, if you look into the output file 
> /tmp/xxxobjectreusebug, for example the key 77 appears twice.
> The good news is that I think I can see an easy fix that doesn't affect 
> performance: MergeIterator.HeadStream could have a reuse object of its own as 
> a member, and give that to iterator.next in nextHead(E reuse). And then we 
> wouldn't need the overload of nextHead that has the reuse parameter, and 
> MergeIterator.next(E reuse) could just call its other overload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK1919] add HCatOutputFormat

[fpompermaier]

Github user fpompermaier commented on the pull request:

https://github.com/apache/flink/pull/1079#issuecomment-222177257
  
Any update on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-2971) Add outer joins to the Table API

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske closed FLINK-2971.

   Resolution: Implemented
Fix Version/s: 1.1.0

Implemented with cbc5f8303ec0e0ebd5cf5c9927d108f52719c160

Thanks for the contribution [~dawidwys]!

> Add outer joins to the Table API
> 
>
> Key: FLINK-2971
> URL: https://issues.apache.org/jira/browse/FLINK-2971
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API
>Reporter: Timo Walther
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Fix For: 1.1.0
>
>
> Since Flink now supports outer joins, the Table API can also support left, 
> right and full outer joins.
> Given that null values are properly supported by RowSerializer etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1981


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3887) Improve dependency management for building docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304163#comment-15304163
 ] 

ASF GitHub Bot commented on FLINK-3887:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/2033#issuecomment-222171685
  
`-p` didn't work, talked to Max offline. After the fix +1 to merge. Very 
nice change!


> Improve dependency management for building docs
> ---
>
> Key: FLINK-3887
> URL: https://issues.apache.org/jira/browse/FLINK-3887
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> Our nightly docs builds currently fail: 
> https://ci.apache.org/builders/flink-docs-master/
> I will file an issue with JIRA to fix it. The root cause is that we rely on a 
> couple of dependencies to be installed. We could circumvent this by providing 
> a Ruby Gemfile that we can then use to load necessary dependencies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3887] improve dependency management for...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/2033#issuecomment-222171685
  
`-p` didn't work, talked to Max offline. After the fix +1 to merge. Very 
nice change!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3836] Add LongHistogram accumulator

[greghogan]

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1966#issuecomment-222161038
  
I was interested to see what happened here and a simple rebase and force 
push corrects the problem.

Make sure local master is up-to-date
$ git checkout master
$ git pull apache

Fetch this PR and checkout the branch
$ git fetch github pull/1966/head:pr1966
$ git checkout pr1966

Move the new commits after the last commit on master
$ git rebase master

Push the changes to your repo
$ git push -f pr1966 origin


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3836) Change Histogram to enable Long counters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304121#comment-15304121
 ] 

ASF GitHub Bot commented on FLINK-3836:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1966#issuecomment-222161038
  
I was interested to see what happened here and a simple rebase and force 
push corrects the problem.

Make sure local master is up-to-date
$ git checkout master
$ git pull apache

Fetch this PR and checkout the branch
$ git fetch github pull/1966/head:pr1966
$ git checkout pr1966

Move the new commits after the last commit on master
$ git rebase master

Push the changes to your repo
$ git push -f pr1966 origin


> Change Histogram to enable Long counters
> 
>
> Key: FLINK-3836
> URL: https://issues.apache.org/jira/browse/FLINK-3836
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.2
>Reporter: Maximilian Bode
>Assignee: Maximilian Bode
>Priority: Minor
>
> Change 
> flink/flink-core/src/main/java/org/apache/flink/api/common/accumulators/Histogram.java
>  to enable Long counts instead of Integer. In particular, change the TreeMap 
> to be .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3836) Change Histogram to enable Long counters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304094#comment-15304094
 ] 

ASF GitHub Bot commented on FLINK-3836:
---

Github user mbode closed the pull request at:

https://github.com/apache/flink/pull/1966


> Change Histogram to enable Long counters
> 
>
> Key: FLINK-3836
> URL: https://issues.apache.org/jira/browse/FLINK-3836
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.2
>Reporter: Maximilian Bode
>Assignee: Maximilian Bode
>Priority: Minor
>
> Change 
> flink/flink-core/src/main/java/org/apache/flink/api/common/accumulators/Histogram.java
>  to enable Long counts instead of Integer. In particular, change the TreeMap 
> to be .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3836] Add LongHistogram accumulator

[mbode]

Github user mbode commented on the pull request:

https://github.com/apache/flink/pull/1966#issuecomment-222155560
  
Sorry guys, botched the PR :/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3836] Add LongHistogram accumulator

[mbode]

Github user mbode closed the pull request at:

https://github.com/apache/flink/pull/1966


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3974) enableObjectReuse fails when an operator chains to multiple downstream operators

[B Wyatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304062#comment-15304062
 ] 

B Wyatt commented on FLINK-3974:


In the topologies I've been building object reuse has been a pretty significant 
CPU win.  It would be a shame to lose that capability for topos where care is 
taken to actually maintain the application level requirements of object reuse: 
Don't modify your inputs.

Disabling chaining is a fine option if this is not a bottleneck in your 
topology.  If unchained-splits become the default, I'd like to see the ability 
to chain them remain as an option.  Sometimes splitting into multiple 
chain/threads is good: cpu heavy operators that benefit from parallelism.  
Sometimes maintaining the chain to avoid the de/serialization costs is good 
(cpu light operators with high throughput).

The patch attached essentially gives each collector its own dedicated stream 
record with a reference to the value. You are correct it doesn't solve the 
problems of mutable operator inputs but, that has always been the primary 
concession of object reuse and it doesn't seem like a bad requirement to put on 
the application level.  I don't know that there is a way to avoid the cost of 
cloning *and* protect against operators mutating input in a language like Java.

> enableObjectReuse fails when an operator chains to multiple downstream 
> operators
> 
>
> Key: FLINK-3974
> URL: https://issues.apache.org/jira/browse/FLINK-3974
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.0.3
>Reporter: B Wyatt
> Attachments: ReproFLINK3974.java, bwyatt-FLINK3974.1.patch
>
>
> Given a topology that looks like this:
> {code:java}
> DataStream input = ...
> input
> .map(MapFunction...)
> .addSink(...);
> input
> .map(MapFunction...)
> ​.addSink(...);
> {code}
> enableObjectReuse() will cause an exception in the form of 
> {{"java.lang.ClassCastException: B cannot be cast to A"}} to be thrown.
> It looks like the input operator calls {{Output.collect}} 
> which attempts to loop over the downstream operators and process them.
> However, the first map operation will call {{StreamRecord<>.replace}} which 
> mutates the value stored in the StreamRecord<>.  
> As a result, when the {{Output.collect}} call passes the 
> {{StreamRecord}} to the second map operation it is actually a 
> {{StreamRecord}} and behaves as if the two map operations were serial 
> instead of parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304052#comment-15304052
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2042#discussion_r64902741
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -82,6 +82,10 @@
 
private static final long DEFAULT_RESTART_DELAY = 1L;
 
+   // This field was used as a key for storing the EC in the Job 
Configuration
+   @Deprecated
+   public static final String CONFIG_KEY = "runtime.config";
--- End diff --

this change belongs in the "Fix breaking changes in flink-core" commit, 
unless they will be squashed.


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2042#discussion_r64902741
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -82,6 +82,10 @@
 
private static final long DEFAULT_RESTART_DELAY = 1L;
 
+   // This field was used as a key for storing the EC in the Job 
Configuration
+   @Deprecated
+   public static final String CONFIG_KEY = "runtime.config";
--- End diff --

this change belongs in the "Fix breaking changes in flink-core" commit, 
unless they will be squashed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304029#comment-15304029
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222139308
  
Thank you for the review.

Its very low overhead since the 1.0.0 jar's are cached in the local .m2 
directory and the tool is doing the check really fast.


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222139308
  
Thank you for the review.

Its very low overhead since the 1.0.0 jar's are cached in the local .m2 
directory and the tool is doing the check really fast.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304025#comment-15304025
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222138890
  
+1 Looks good to me. 

Only wondering, how much will it affect our build time? We might want to 
run this in a dedicated profile.


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[mxm]

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222138890
  
+1 Looks good to me. 

Only wondering, how much will it affect our build time? We might want to 
run this in a dedicated profile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[dawidwys]

Github user dawidwys commented on the pull request:

https://github.com/apache/flink/pull/1981#issuecomment-222138623
  
Really sorry for those mistakes :( Thanks for fixing them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304015#comment-15304015
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-222136775
  
Thanks a lot for addressing the comments.
I'll push the changes to travis, once its green I'll merge them.


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3923] [connector-kinesis] Unify configu...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-222136775
  
Thanks a lot for addressing the comments.
I'll push the changes to travis, once its green I'll merge them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303998#comment-15303998
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222134657
  
The plugin runs with every mvn verify call.

For each maven module, it will download the last stable release (in this 
case 1.0.0) and check it against the build (currently the 1.1-SNAPSHOT).


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222134657
  
The plugin runs with every mvn verify call.

For each maven module, it will download the last stable release (in this 
case 1.0.0) and check it against the build (currently the 1.1-SNAPSHOT).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3291) Object reuse bug in MergeIterator.HeadStream.nextHead

[Gabor Gevay (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303994#comment-15303994
 ] 

Gabor Gevay commented on FLINK-3291:


This bug occurred when object reuse is enabled. This is disabled by default, 
and can be enabled by calling ExecutionConfig.enableObjectReuse. I don't know 
Beam, but I would guess that this option is also not enabled there by default.

> Object reuse bug in MergeIterator.HeadStream.nextHead
> -
>
> Key: FLINK-3291
> URL: https://issues.apache.org/jira/browse/FLINK-3291
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: Gabor Gevay
>Priority: Critical
>
> MergeIterator.HeadStream.nextHead saves a reference into `this.head` of the 
> `reuse` object that it got as an argument. This object might be modified 
> later by the caller.
> This actually happens when ReduceDriver.run calls input.next (which will 
> actually be MergeIterator.next(E reuse)) in the inner while loop of the 
> objectReuseEnabled branch, and that calls top.nextHead with the reference 
> that it got from ReduceDriver, which erroneously saves the reference, and 
> then ReduceDriver later uses that same object for doing the reduce.
> Another way in which this fails is when MergeIterator.next(E reuse) gives 
> `reuse` to different `top`s in different calls, and then the heads end up 
> being the same object.
> You can observe the latter situation in action by running ReducePerformance 
> here:
> https://github.com/ggevay/flink/tree/merge-iterator-object-reuse-bug
> Set memory to -Xmx200m (so that the MergeIterator actually has merging to 
> do), put a breakpoint at the beginning of MergeIterator.next(reuse), and then 
> watch `reuse`, and the heads of the first two elements of `this.heap` in the 
> debugger. They will get to be the same object after hitting continue about 6 
> times.
> You can also look at the count that is printed at the end, which shouldn't be 
> larger than the key range. Also, if you look into the output file 
> /tmp/xxxobjectreusebug, for example the key 77 appears twice.
> The good news is that I think I can see an easy fix that doesn't affect 
> performance: MergeIterator.HeadStream could have a reuse object of its own as 
> a member, and give that to iterator.next in nextHead(E reuse). And then we 
> wouldn't need the overload of nextHead that has the reuse parameter, and 
> MergeIterator.next(E reuse) could just call its other overload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3944) Add optimization rules to reorder Cartesian products and joins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303992#comment-15303992
 ] 

ASF GitHub Bot commented on FLINK-3944:
---

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/2044

[FLINK-3944] [tableAPI] Add rewrite rules to reorder Cartesian products and 
joins.

The Table API / SQL interface do not reorder joins due to missing stats and 
estimates.

This can lead to situations where a query with two inner equi joins cannot 
be executed due to missing support for Cartesian products. For instance the 
following query would not be executed:

```
SELECT * 
FROM t1, t2, t3
WHERE t1.a = t3.a AND t2.b = t3.b
```
because the plan would first compute the Cartesian product of  `t1` and 
`t2` because both are listed first in the `FROM` clause.

This PR adds optimization rules to reorder joins and Cartesian products to 
resolve such situation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink tableCrossOrder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2044.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2044


commit 760ffe29bc1e2f4462f453f378c06c7b1f980dee
Author: Fabian Hueske 
Date:   2016-05-27T11:04:12Z

[FLINK-3944] [tableAPI] Add rewrite rules to reorder Cartesian products and 
joins.

These rules are necessary to resolve join orders that initially contain 
Cartesian products
due to the order in which base relations are added in the FROM clause (SQL) 
or joined (Table API).




> Add optimization rules to reorder Cartesian products and joins
> --
>
> Key: FLINK-3944
> URL: https://issues.apache.org/jira/browse/FLINK-3944
> Project: Flink
>  Issue Type: Bug
>  Components: Table API
>Affects Versions: 1.1.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.1.0
>
>
> Currently, we do not support the execution of Cartesian products. 
> Because we do not optimize the order of joins (due to missing statistics), 
> joins are executed in the order in which they are specified. This works well 
> for the Table API, however it can be problematic in case of SQL queries where 
> the order of tables in the FROM clause should not matter.
> In case of SQL queries, it can happen that the optimized plan contains 
> Cartesian products because joins are not reordered. If we add optimization 
> rules that switch Cartesian products and joins, such situations can be 
> resolved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3944] [tableAPI] Add rewrite rules to r...

[fhueske]

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/2044

[FLINK-3944] [tableAPI] Add rewrite rules to reorder Cartesian products and 
joins.

The Table API / SQL interface do not reorder joins due to missing stats and 
estimates.

This can lead to situations where a query with two inner equi joins cannot 
be executed due to missing support for Cartesian products. For instance the 
following query would not be executed:

```
SELECT * 
FROM t1, t2, t3
WHERE t1.a = t3.a AND t2.b = t3.b
```
because the plan would first compute the Cartesian product of  `t1` and 
`t2` because both are listed first in the `FROM` clause.

This PR adds optimization rules to reorder joins and Cartesian products to 
resolve such situation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink tableCrossOrder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2044.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2044


commit 760ffe29bc1e2f4462f453f378c06c7b1f980dee
Author: Fabian Hueske 
Date:   2016-05-27T11:04:12Z

[FLINK-3944] [tableAPI] Add rewrite rules to reorder Cartesian products and 
joins.

These rules are necessary to resolve join orders that initially contain 
Cartesian products
due to the order in which base relations are added in the FROM clause (SQL) 
or joined (Table API).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303991#comment-15303991
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222132940
  
Thanks for the pull request! Could you elaborate on how and where this will 
be integrated with our CI or release process?


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[mxm]

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2042#issuecomment-222132940
  
Thanks for the pull request! Could you elaborate on how and where this will 
be integrated with our CI or release process?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-3960) Disable, fix and re-enable EventTimeWindowCheckpointingITCase

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3960:
--
Summary: Disable, fix and re-enable EventTimeWindowCheckpointingITCase  
(was: EventTimeWindowCheckpointingITCase fails with a segmentation fault)

> Disable, fix and re-enable EventTimeWindowCheckpointingITCase
> -
>
> Key: FLINK-3960
> URL: https://issues.apache.org/jira/browse/FLINK-3960
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Tests
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> As a follow-up issue of FLINK-3909, our tests fail with the following. I 
> believe [~aljoscha] is working on a fix.
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fae0c62a264, pid=72720, tid=140385528268544
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_76-b13) (build 
> 1.7.0_76-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.76-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni78704726610339516..so+0x13c264]  
> rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, 
> rocksdb::ColumnFamilyHandle*)+0x4
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/travis/build/mxm/flink/flink-tests/target/hs_err_pid72720.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> Aborted (core dumped)
> {noformat}
> I propose to disable the test case in the meantime because it is blocking our 
> test execution which we need for pull requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (FLINK-3960) EventTimeWindowCheckpointingITCase fails with a segmentation fault

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reopened FLINK-3960:
---

Keeping this for the record to revert 98a939552e12fc699ff39111bbe877e112460ceb.

> EventTimeWindowCheckpointingITCase fails with a segmentation fault
> --
>
> Key: FLINK-3960
> URL: https://issues.apache.org/jira/browse/FLINK-3960
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Tests
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> As a follow-up issue of FLINK-3909, our tests fail with the following. I 
> believe [~aljoscha] is working on a fix.
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fae0c62a264, pid=72720, tid=140385528268544
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_76-b13) (build 
> 1.7.0_76-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.76-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni78704726610339516..so+0x13c264]  
> rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, 
> rocksdb::ColumnFamilyHandle*)+0x4
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/travis/build/mxm/flink/flink-tests/target/hs_err_pid72720.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> Aborted (core dumped)
> {noformat}
> I propose to disable the test case in the meantime because it is blocking our 
> test execution which we need for pull requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1981#issuecomment-222131071
  
Thanks for the update @dawidwys. I found a few minor issues in the docs. 
Will fix them and merge this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895580
  
--- Diff: docs/apis/table.md ---
@@ -542,6 +578,42 @@ val result = left.join(right).where('a === 
'd).select('a, 'b, 'e);
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.rightOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
+{% endhighlight %}
+  
+
+
+
+  LeftOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.leftOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
--- End diff --

`=` -> `===`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895561
  
--- Diff: docs/apis/table.md ---
@@ -542,6 +578,42 @@ val result = left.join(right).where('a === 
'd).select('a, 'b, 'e);
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.rightOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
+{% endhighlight %}
+  
+
+
+
+  LeftOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
--- End diff --

`FULL OUTER JOIN` should be `LEFT OUTER JOIN`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895590
  
--- Diff: docs/apis/table.md ---
@@ -542,6 +578,42 @@ val result = left.join(right).where('a === 
'd).select('a, 'b, 'e);
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.rightOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
+{% endhighlight %}
+  
+
+
+
+  LeftOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.leftOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
+{% endhighlight %}
+  
+
+
+
+  FullOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.fullOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
--- End diff --

`=` -> `===`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895548
  
--- Diff: docs/apis/table.md ---
@@ -542,6 +578,42 @@ val result = left.join(right).where('a === 
'd).select('a, 'b, 'e);
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.rightOuterJoin(right, 'a = 'd).select('a, 'b, 'e)
--- End diff --

`=` should be `===`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895471
  
--- Diff: docs/apis/table.md ---
@@ -423,14 +423,50 @@ Table result = in.groupBy("a").select("a, b.sum as 
d");
 
   Join
   
-Similar to a SQL JOIN clause. Joins two tables. Both tables 
must have distinct field names and an equality join predicate must be defined 
using a where or filter operator.
+Similar to a SQL JOIN clause. Joins two tables. Both tables 
must have distinct field names and at least one equality join predicate must be 
defined through join operator or using a where or filter operator.
 {% highlight java %}
 Table left = tableEnv.fromDataSet(ds1, "a, b, c");
 Table right = tableEnv.fromDataSet(ds2, "d, e, f");
 Table result = left.join(right).where("a = d").select("a, b, e");
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
--- End diff --

`FULL OUTER JOIN` should be `RIGHT OUTER JOIN`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895489
  
--- Diff: docs/apis/table.md ---
@@ -423,14 +423,50 @@ Table result = in.groupBy("a").select("a, b.sum as 
d");
 
   Join
   
-Similar to a SQL JOIN clause. Joins two tables. Both tables 
must have distinct field names and an equality join predicate must be defined 
using a where or filter operator.
+Similar to a SQL JOIN clause. Joins two tables. Both tables 
must have distinct field names and at least one equality join predicate must be 
defined through join operator or using a where or filter operator.
 {% highlight java %}
 Table left = tableEnv.fromDataSet(ds1, "a, b, c");
 Table right = tableEnv.fromDataSet(ds2, "d, e, f");
 Table result = left.join(right).where("a = d").select("a, b, e");
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.rightOuterJoin(right, "a = d").select("a, b, e");
+{% endhighlight %}
+  
+
+
+
+  LeftOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
--- End diff --

`FULL OUTER JOIN` should be `LEFT OUTER JOIN`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-2971][table] Add outer joins to the Tab...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1981#discussion_r64895504
  
--- Diff: docs/apis/table.md ---
@@ -542,6 +578,42 @@ val result = left.join(right).where('a === 
'd).select('a, 'b, 'e);
 {% endhighlight %}
   
 
+
+
+  RightOuterJoin
+  
+Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both 
tables must have distinct field names and at least one equality join predicate 
must be defined.
--- End diff --

`FULL OUTER JOIN` should be `RIGHT OUTER JOIN`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3886) Give a better error when the application Main class is not public.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303970#comment-15303970
 ] 

ASF GitHub Bot commented on FLINK-3886:
---

GitHub user nielsbasjes opened a pull request:

https://github.com/apache/flink/pull/2043

[FLINK-3886] Give a better error when the application Main class is not 
public.

A simple fix that reduces the time needed to find the cause of this simple 
developer error.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nielsbasjes/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2043


commit 062879e0e74d32caa6aa09d770f8e67230181cf9
Author: Niels Basjes 
Date:   2016-05-27T11:51:04Z

[FLINK-3886] Give a better error when the application Main class is not 
public.




> Give a better error when the application Main class is not public.
> --
>
> Key: FLINK-3886
> URL: https://issues.apache.org/jira/browse/FLINK-3886
> Project: Flink
>  Issue Type: Improvement
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: FLINK-3886-20160509.patch
>
>
> I wrote a Flink application and made the simple mistake of making the Main 
> class 'package private' by simply writing it as (note: I simply forgot the 
> 'public' keyword)
> {code}
> class Main {
>...
> }
> {code}
> The error you get is:
> {code}
> Caused by: java.lang.IllegalAccessException: Class 
> org.apache.flink.client.program.PackagedProgram can not access a member of 
> class com.bol.experiment.flink.Main with modifiers "public static"
>   at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>   at 
> java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
>   at 
> java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
>   at java.lang.reflect.Method.invoke(Method.java:490)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
> {code}
> This took me 30 minutes to figure out what I did wrong.
> I think the error message should be more explanatory to the developer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3886] Give a better error when the appl...

[nielsbasjes]

GitHub user nielsbasjes opened a pull request:

https://github.com/apache/flink/pull/2043

[FLINK-3886] Give a better error when the application Main class is not 
public.

A simple fix that reduces the time needed to find the cause of this simple 
developer error.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nielsbasjes/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2043


commit 062879e0e74d32caa6aa09d770f8e67230181cf9
Author: Niels Basjes 
Date:   2016-05-27T11:51:04Z

[FLINK-3886] Give a better error when the application Main class is not 
public.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3960) EventTimeWindowCheckpointingITCase fails with a segmentation fault

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-3960.
---
Resolution: Duplicate

This is a duplicate of FLINK-3948.

> EventTimeWindowCheckpointingITCase fails with a segmentation fault
> --
>
> Key: FLINK-3960
> URL: https://issues.apache.org/jira/browse/FLINK-3960
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Tests
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
>  Labels: test-stability
> Fix For: 1.1.0
>
>
> As a follow-up issue of FLINK-3909, our tests fail with the following. I 
> believe [~aljoscha] is working on a fix.
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fae0c62a264, pid=72720, tid=140385528268544
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_76-b13) (build 
> 1.7.0_76-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.76-b04 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [librocksdbjni78704726610339516..so+0x13c264]  
> rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, 
> rocksdb::ColumnFamilyHandle*)+0x4
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/travis/build/mxm/flink/flink-tests/target/hs_err_pid72720.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> Aborted (core dumped)
> {noformat}
> I propose to disable the test case in the meantime because it is blocking our 
> test execution which we need for pull requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request:

[mxm]

Github user mxm commented on the pull request:


https://github.com/apache/flink/commit/bf256c7fbe05accdadc8470013879f567341d1aa#commitcomment-17644775
  
In 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceID.java:
In 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceID.java
 on line 52:
No worries, that stuff can happen. Fixing this with #2037 (underlying issue 
why this bug could occur).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-3944) Add optimization rules to reorder Cartesian products and joins

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske reassigned FLINK-3944:


Assignee: Fabian Hueske

> Add optimization rules to reorder Cartesian products and joins
> --
>
> Key: FLINK-3944
> URL: https://issues.apache.org/jira/browse/FLINK-3944
> Project: Flink
>  Issue Type: Bug
>  Components: Table API
>Affects Versions: 1.1.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.1.0
>
>
> Currently, we do not support the execution of Cartesian products. 
> Because we do not optimize the order of joins (due to missing statistics), 
> joins are executed in the order in which they are specified. This works well 
> for the Table API, however it can be problematic in case of SQL queries where 
> the order of tables in the FROM clause should not matter.
> In case of SQL queries, it can happen that the optimized plan contains 
> Cartesian products because joins are not reordered. If we add optimization 
> rules that switch Cartesian products and joins, such situations can be 
> resolved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3948) EventTimeWindowCheckpointingITCase Fails with Core Dump

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-3948:

Affects Version/s: 1.1.0

> EventTimeWindowCheckpointingITCase Fails with Core Dump
> ---
>
> Key: FLINK-3948
> URL: https://issues.apache.org/jira/browse/FLINK-3948
> Project: Flink
>  Issue Type: Bug
>  Components: state backends
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Critical
>
> It fails because of a core dump in RocksDB. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request:

[StephanEwen]

Github user StephanEwen commented on the pull request:


https://github.com/apache/flink/commit/bf256c7fbe05accdadc8470013879f567341d1aa#commitcomment-17644300
  
In 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceID.java:
In 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceID.java
 on line 52:
Sorry, this was a partial change that accidentally sneaked into this patch.
My bad :-(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3129] Add tooling to ensure interface s...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/2042

[FLINK-3129] Add tooling to ensure interface stability

This pull request adds a maven module for ensuring interface stability 
(japicmp).

I needed to revert some of the changes since 1.0:
- I added the `public static final String CONFIG_KEY` in the 
`ExecutionConfig` again.
- I made the `getMetricGroup()` in the `RuntimeContext` an evolving API.
- I added two deprecated config constants again.
- I reintroduced the deprecated `Key` interface.

Once this has been merged, I'll file JIRAs for 2.0 to remove these changes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3129-third

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2042.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2042


commit fd0e81558953f04b07dd5edaec58e9fe73903a7f
Author: Robert Metzger 
Date:   2015-12-07T14:45:57Z

[FLINK-3129] Bring infrastructure for ensuring interface stability in place

commit b15d5b84dd714fb6660b51ce72f03f1f25c134b8
Author: Robert Metzger 
Date:   2016-05-27T08:53:35Z

[FLINK-3129] Fix breaking changes in flink-core

commit b53b4805a70f2c0ae678bf41148c82e04b531417
Author: Robert Metzger 
Date:   2016-05-27T10:09:45Z

[FLINK-3129] Fix ES2 issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303904#comment-15303904
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/2042

[FLINK-3129] Add tooling to ensure interface stability

This pull request adds a maven module for ensuring interface stability 
(japicmp).

I needed to revert some of the changes since 1.0:
- I added the `public static final String CONFIG_KEY` in the 
`ExecutionConfig` again.
- I made the `getMetricGroup()` in the `RuntimeContext` an evolving API.
- I added two deprecated config constants again.
- I reintroduced the deprecated `Key` interface.

Once this has been merged, I'll file JIRAs for 2.0 to remove these changes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3129-third

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2042.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2042


commit fd0e81558953f04b07dd5edaec58e9fe73903a7f
Author: Robert Metzger 
Date:   2015-12-07T14:45:57Z

[FLINK-3129] Bring infrastructure for ensuring interface stability in place

commit b15d5b84dd714fb6660b51ce72f03f1f25c134b8
Author: Robert Metzger 
Date:   2016-05-27T08:53:35Z

[FLINK-3129] Fix breaking changes in flink-core

commit b53b4805a70f2c0ae678bf41148c82e04b531417
Author: Robert Metzger 
Date:   2016-05-27T10:09:45Z

[FLINK-3129] Fix ES2 issue




> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add japicmp-maven-plugin to ensur...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1791#issuecomment-222113705
  
Closing this PR in favor of a new one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303895#comment-15303895
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger closed the pull request at:

https://github.com/apache/flink/pull/1791


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3129) Add tooling to ensure interface stability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303894#comment-15303894
 ] 

ASF GitHub Bot commented on FLINK-3129:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1791#issuecomment-222113705
  
Closing this PR in favor of a new one.


> Add tooling to ensure interface stability
> -
>
> Key: FLINK-3129
> URL: https://issues.apache.org/jira/browse/FLINK-3129
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> I would like to use this maven plugin: https://github.com/siom79/japicmp to 
> automatically ensure interface stability across minor releases.
> Ideally we have the plugin in place after Flink 1.0 is out, so that maven 
> builds break if a breaking change has been made.
> The plugin already supports downloading a reference release, checking the 
> build and breaking it.
> Not yet supported are class/method inclusions based on annotations, but I've 
> opened a pull request for adding it.
> There are also issues with the resolution of the dependency with the 
> annotations, but I'm working on resolving those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3129] Add japicmp-maven-plugin to ensur...

[rmetzger]

Github user rmetzger closed the pull request at:

https://github.com/apache/flink/pull/1791


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3972) Subclasses of ResourceID may not to be serializable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303888#comment-15303888
 ] 

ASF GitHub Bot commented on FLINK-3972:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2037#issuecomment-222113016
  
Rebased to the latest master. Will merge once the tests pass.


> Subclasses of ResourceID may not to be serializable
> ---
>
> Key: FLINK-3972
> URL: https://issues.apache.org/jira/browse/FLINK-3972
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.1.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.1.0
>
>
> WorkerTypes are currently subclasses of ResourceID. ResourceID has to be 
> Serializable but its subclasses don't. This may lead to problems when these 
> subclasses are used as ResourceIDs, i.e. serialization may fail with 
> NotSerializableExceptions. Currently, subclasses are never send over the wire 
> but they might be in the future.
> Instead of relying on subclasses of ResourceID for the WorkerTypes, we can 
> let them implement an interface to retrieve the ResourceID of a WorkerType.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3972] subclasses of ResourceID may not ...

[mxm]

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/2037#issuecomment-222113016
  
Rebased to the latest master. Will merge once the tests pass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3291) Object reuse bug in MergeIterator.HeadStream.nextHead

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303884#comment-15303884
 ] 

Pawel Szczur commented on FLINK-3291:
-

I've browsed through a Flink issues to find anything related to this: 
http://stackoverflow.com/questions/37473682/items-not-groupped-correctly-cogroupbykey

Basing on the description and some comments, is it possible my problem is 
related to this bug?

> Object reuse bug in MergeIterator.HeadStream.nextHead
> -
>
> Key: FLINK-3291
> URL: https://issues.apache.org/jira/browse/FLINK-3291
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: Gabor Gevay
>Priority: Critical
>
> MergeIterator.HeadStream.nextHead saves a reference into `this.head` of the 
> `reuse` object that it got as an argument. This object might be modified 
> later by the caller.
> This actually happens when ReduceDriver.run calls input.next (which will 
> actually be MergeIterator.next(E reuse)) in the inner while loop of the 
> objectReuseEnabled branch, and that calls top.nextHead with the reference 
> that it got from ReduceDriver, which erroneously saves the reference, and 
> then ReduceDriver later uses that same object for doing the reduce.
> Another way in which this fails is when MergeIterator.next(E reuse) gives 
> `reuse` to different `top`s in different calls, and then the heads end up 
> being the same object.
> You can observe the latter situation in action by running ReducePerformance 
> here:
> https://github.com/ggevay/flink/tree/merge-iterator-object-reuse-bug
> Set memory to -Xmx200m (so that the MergeIterator actually has merging to 
> do), put a breakpoint at the beginning of MergeIterator.next(reuse), and then 
> watch `reuse`, and the heads of the first two elements of `this.heap` in the 
> debugger. They will get to be the same object after hitting continue about 6 
> times.
> You can also look at the count that is printed at the end, which shouldn't be 
> larger than the key range. Also, if you look into the output file 
> /tmp/xxxobjectreusebug, for example the key 77 appears twice.
> The good news is that I think I can see an easy fix that doesn't affect 
> performance: MergeIterator.HeadStream could have a reuse object of its own as 
> a member, and give that to iterator.next in nextHead(E reuse). And then we 
> wouldn't need the overload of nextHead that has the reuse parameter, and 
> MergeIterator.next(E reuse) could just call its other overload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3317) Add timeout handler to CEP operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303879#comment-15303879
 ] 

ASF GitHub Bot commented on FLINK-3317:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2041

[FLINK-3317] [cep] Introduce timeout handler to CEP operator

This PR adds timeout handling functionality to the CEP library. Whenever an 
event pattern is discarded because it exceeds the window length, a user 
specified timeout function can be called. The timeout function receives the all 
matched events of the partial pattern as well as the timestamp when the timeout 
has occurred. This allows to create timeout events to which down stream 
operators can react.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink cepTimeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2041.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2041


commit d8ccf9bc734ceeaf7a3f897e3c3a5ebda427ccd0
Author: Till Rohrmann 
Date:   2016-05-26T09:34:16Z

[FLINK-3317] [cep] Introduce timeout handler to CEP operator

Introduce timeout handling flag for the NFACompiler

Expose timeout handling via Java API

Update documentation of PatternStream and CEP

Introduce timeout select function to CEP Scala API

Add select and flatSelect with timeout support to CEP Scala API

Add test cases for timeout handling

Update documentation

commit 2d57d8016cedc997ca18b34406dd9a98e93de496
Author: Till Rohrmann 
Date:   2016-05-27T10:13:09Z

Fix CEP Scala API completeness test




> Add timeout handler to CEP operator
> ---
>
> Key: FLINK-3317
> URL: https://issues.apache.org/jira/browse/FLINK-3317
> Project: Flink
>  Issue Type: Improvement
>  Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, event sequences which exceed the defined pattern timeout will be 
> discarded. However, in some cases the user might be interested in getting to 
> know when such a timeout occurred to return a default value for these event 
> sequences.
> Thus, the pattern API should be extended to be able to define a timeout 
> handler. Furthermore, the {{NFA}} has to be extended to also return the 
> discarded event sequences. The {{CEPOperator}} would then call for every 
> discarded event sequence the timeout handler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3923) Unify configuration conventions of the Kinesis producer to the same as the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303878#comment-15303878
 ] 

ASF GitHub Bot commented on FLINK-3923:
---

Github user aozturk commented on the pull request:

https://github.com/apache/flink/pull/2016#issuecomment-222111542
  
Pushed updates according to your comments.


> Unify configuration conventions of the Kinesis producer to the same as the 
> consumer
> ---
>
> Key: FLINK-3923
> URL: https://issues.apache.org/jira/browse/FLINK-3923
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kinesis Connector, Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Robert Metzger
>Assignee: Abdullah Ozturk
>
> Currently, the Kinesis consumer and producer are configured differently.
> The producer expects a list of arguments for the access key, secret, region, 
> stream. The consumer is accepting properties (similar to the Kafka connector).
> The objective of this issue is to change the producer so that it is also 
> using a properties-based configuration (including an input validation step)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3317] [cep] Introduce timeout handler t...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2041

[FLINK-3317] [cep] Introduce timeout handler to CEP operator

This PR adds timeout handling functionality to the CEP library. Whenever an 
event pattern is discarded because it exceeds the window length, a user 
specified timeout function can be called. The timeout function receives the all 
matched events of the partial pattern as well as the timestamp when the timeout 
has occurred. This allows to create timeout events to which down stream 
operators can react.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink cepTimeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2041.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2041


commit d8ccf9bc734ceeaf7a3f897e3c3a5ebda427ccd0
Author: Till Rohrmann 
Date:   2016-05-26T09:34:16Z

[FLINK-3317] [cep] Introduce timeout handler to CEP operator

Introduce timeout handling flag for the NFACompiler

Expose timeout handling via Java API

Update documentation of PatternStream and CEP

Introduce timeout select function to CEP Scala API

Add select and flatSelect with timeout support to CEP Scala API

Add test cases for timeout handling

Update documentation

commit 2d57d8016cedc997ca18b34406dd9a98e93de496
Author: Till Rohrmann 
Date:   2016-05-27T10:13:09Z

Fix CEP Scala API completeness test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---