[jira] [Updated] (FLINK-4375) Introduce rpc protocols implemented by job manager

[Wenlong Lyu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenlong Lyu updated FLINK-4375:
---
Summary: Introduce rpc protocols implemented by job manager  (was: 
Introduce rpc protocols provided by job manager)

> Introduce rpc protocols implemented by job manager
> --
>
> Key: FLINK-4375
> URL: https://issues.apache.org/jira/browse/FLINK-4375
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Wenlong Lyu
>Assignee: Wenlong Lyu
>
>  job manager RPC server needs to  implement a job control protocol, resource 
> user protocol, task control protocol,
>   1. job controller: cancelJob, suspendJob, etc.
>   2. resource user: slotFailed(notify slot failure), 
> slotAvailable(offer slot), etc.
>   3. task controller: updateTaskState, updateResultPartitionInfo, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418355#comment-15418355
 ] 

ASF GitHub Bot commented on FLINK-4253:
---

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/2342
  
Just a query,
If there are both old and new configs available -  we should give priority 
to the new one right?  Even if the value for the old and new configs are 
different?


> Rename "recovery.mode" config key to "high-availability"
> 
>
> Key: FLINK-4253
> URL: https://issues.apache.org/jira/browse/FLINK-4253
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ufuk Celebi
>Assignee: ramkrishna.s.vasudevan
>
> Currently, HA is configured via the following configuration keys:
> {code}
> recovery.mode: STANDALONE // No high availability (HA)
> recovery.mode: ZOOKEEPER // HA
> {code}
> This could be more straight forward by simply renaming the key to 
> {{high-availability}}. Furthermore, the term {{STANDALONE}} is overloaded. We 
> already have standalone cluster mode.
> {code}
> high-availability: NONE // No HA
> high-availability: ZOOKEEPER // HA via ZooKeeper
> {code}
> The {{recovery.mode}} configuration keys would have to be deprecated before 
> completely removing them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418349#comment-15418349
 ] 

ASF GitHub Bot commented on FLINK-4253:
---

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/2342
  
> Can you add such a test?

Sure I can. The existing test cases helped to ensure that if there are no 
old configs we are able to fetch from the new config. I can add a test case to 
ensure both the configs are accepted.

> The other variables should match, e.g.
recovery.jobmanager.port => high-availability.jobmanager.port and also 
recovery.zookeeper.* => recovery.zookeeper.* etc.

I see. I had this doubt but since the PR was talking about this specific 
param I thought that would be enough. +1 to change all relevant ones. I can 
update it in the next PR. 

Thanks all for the comments. 


> Rename "recovery.mode" config key to "high-availability"
> 
>
> Key: FLINK-4253
> URL: https://issues.apache.org/jira/browse/FLINK-4253
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ufuk Celebi
>Assignee: ramkrishna.s.vasudevan
>
> Currently, HA is configured via the following configuration keys:
> {code}
> recovery.mode: STANDALONE // No high availability (HA)
> recovery.mode: ZOOKEEPER // HA
> {code}
> This could be more straight forward by simply renaming the key to 
> {{high-availability}}. Furthermore, the term {{STANDALONE}} is overloaded. We 
> already have standalone cluster mode.
> {code}
> high-availability: NONE // No HA
> high-availability: ZOOKEEPER // HA via ZooKeeper
> {code}
> The {{recovery.mode}} configuration keys would have to be deprecated before 
> completely removing them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3870) Add IntelliJ code style file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418303#comment-15418303
 ] 

ASF GitHub Bot commented on FLINK-3870:
---

Github user wuchong commented on a diff in the pull request:

https://github.com/apache/flink/pull/1963#discussion_r74537594
  
--- Diff: tools/FlinkCodeStyle.xml ---
@@ -0,0 +1,75 @@
+
+
+  
+  
+  
+  
+  
+  
+
+
+
+  
+  
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+
+  
+  
+
--- End diff --

How about not align multiline chained methods? 

It will align like this:

```scala
fieldNames
.zip(fieldTypes)
.foreach(...
```


> Add IntelliJ code style file
> 
>
> Key: FLINK-3870
> URL: https://issues.apache.org/jira/browse/FLINK-3870
> Project: Flink
>  Issue Type: New Feature
>  Components: Documentation
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>
> Attach intellij code style file to code base and reference it from How to 
> contribute site.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4355) Implement TaskManager side of registration at ResourceManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418264#comment-15418264
 ] 

ASF GitHub Bot commented on FLINK-4355:
---

Github user wenlong88 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2353#discussion_r74533793
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/taskexecutor/TaskExecutorToResourceManagerConnection.java
 ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.taskexecutor;
+
+import org.apache.flink.runtime.rpc.resourcemanager.ResourceManagerGateway;
+
+import java.util.UUID;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+public class TaskExecutorToResourceManagerConnection {
+
+   private final TaskExecutor taskExecutor;
+
+   private final ResourceManagerGateway resourceManager;
+
+   private final UUID resourceManagerLeaderId;
+
+   private final String resourceManagerAddress;
+
+   public TaskExecutorToResourceManagerConnection(
--- End diff --

I think we can have a HARPCGateway extends HAService which can be reused 
for other components


> Implement TaskManager side of registration at ResourceManager
> -
>
> Key: FLINK-4355
> URL: https://issues.apache.org/jira/browse/FLINK-4355
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Stephan Ewen
>
> If the {{TaskManager}} is unregistered, it should try and register at the 
> {{ResourceManager}} leader. The registration messages are fenced via the 
> {{RmLeaderID}}.
> The ResourceManager may acknowledge the registration (or respond that the 
> TaskManager is AlreadyRegistered) or refuse the registration.
> Upon registration refusal, the TaskManager may have to kill itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4355) Implement TaskManager side of registration at ResourceManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418260#comment-15418260
 ] 

ASF GitHub Bot commented on FLINK-4355:
---

Github user wenlong88 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2353#discussion_r74533629
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/taskexecutor/TaskExecutorToResourceManagerConnection.java
 ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.taskexecutor;
+
+import org.apache.flink.runtime.rpc.resourcemanager.ResourceManagerGateway;
+
+import java.util.UUID;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+public class TaskExecutorToResourceManagerConnection {
--- End diff --

I think we can have a HARPCGateway extends HAService which can be reused 
for other components


> Implement TaskManager side of registration at ResourceManager
> -
>
> Key: FLINK-4355
> URL: https://issues.apache.org/jira/browse/FLINK-4355
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Stephan Ewen
>
> If the {{TaskManager}} is unregistered, it should try and register at the 
> {{ResourceManager}} leader. The registration messages are fenced via the 
> {{RmLeaderID}}.
> The ResourceManager may acknowledge the registration (or respond that the 
> TaskManager is AlreadyRegistered) or refuse the registration.
> Upon registration refusal, the TaskManager may have to kill itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1963: [FLINK-3870][docs] Added IntelliJ code style

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/1963
  
Hi @danielblazevski , I find a way to solve this. Just copy the 
`FlinkCodeStyle.xml` to  the config/codestyles directory (for me is 
`~/Library/Preferences/IntelliJIdea15/codestyles`) and restarting the IDE and 
go to IntelliJ code style settings (File -> Settings -> Editor -> Code Style) 
and choose `Flink` schema.

Hope this can help you. My version is IntelliJ IDEA 15.0.6 (ultimate 
edition).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4355) Implement TaskManager side of registration at ResourceManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418259#comment-15418259
 ] 

ASF GitHub Bot commented on FLINK-4355:
---

Github user wenlong88 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2353#discussion_r74533628
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/taskexecutor/TaskExecutorToResourceManagerConnection.java
 ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.taskexecutor;
+
+import org.apache.flink.runtime.rpc.resourcemanager.ResourceManagerGateway;
+
+import java.util.UUID;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+public class TaskExecutorToResourceManagerConnection {
--- End diff --

I think we can have a HARPCGateway extends HAService which can be reused 
for other components


> Implement TaskManager side of registration at ResourceManager
> -
>
> Key: FLINK-4355
> URL: https://issues.apache.org/jira/browse/FLINK-4355
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Stephan Ewen
>
> If the {{TaskManager}} is unregistered, it should try and register at the 
> {{ResourceManager}} leader. The registration messages are fenced via the 
> {{RmLeaderID}}.
> The ResourceManager may acknowledge the registration (or respond that the 
> TaskManager is AlreadyRegistered) or refuse the registration.
> Upon registration refusal, the TaskManager may have to kill itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418104#comment-15418104
 ] 

ASF GitHub Bot commented on FLINK-3874:
---

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2244
  
@twalthr Thank you for your detailed review. I am very new to Apache Flink 
at the moment so I made some absurd changes. I'll try to avoid this in the 
future.

I'll update the PR according to your comments in a day or two.


> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3318) Add support for quantifiers to CEP's pattern API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417952#comment-15417952
 ] 

ASF GitHub Bot commented on FLINK-3318:
---

Github user mushketyk closed the pull request at:

https://github.com/apache/flink/pull/2361


> Add support for quantifiers to CEP's pattern API
> 
>
> Key: FLINK-3318
> URL: https://issues.apache.org/jira/browse/FLINK-3318
> Project: Flink
>  Issue Type: Improvement
>  Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> It would be a good addition to extend the pattern API to support quantifiers 
> known from regular expressions (e.g. Kleene star, ?, +, or count bounds). 
> This would considerably enrich the set of supported patterns.
> Implementing the count bounds could be done by unrolling the pattern state. 
> In order to support the Kleene star operator, the {{NFACompiler}} has to be 
> extended to insert epsilon-transition between a Kleene start state and the 
> succeeding pattern state. In order to support {{?}}, one could insert two 
> paths from the preceding state, one which accepts the event and another which 
> directly goes into the next pattern state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3318) Add support for quantifiers to CEP's pattern API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417954#comment-15417954
 ] 

ASF GitHub Bot commented on FLINK-3318:
---

GitHub user mushketyk reopened a pull request:

https://github.com/apache/flink/pull/2361

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink cep-operators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2361


commit 96c077a4c1c2c1ccba678ec863775402554d6dcf
Author: Ivan Mushketyk 
Date:   2016-08-05T20:05:39Z

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

commit 425fa3d94d15ea6b1396e7bf7a901f7318f107b0
Author: Ivan Mushketyk 
Date:   2016-08-11T21:10:43Z

[FLINK-3318][cep] Add documentation about pattern quantifiers




> Add support for quantifiers to CEP's pattern API
> 
>
> Key: FLINK-3318
> URL: https://issues.apache.org/jira/browse/FLINK-3318
> Project: Flink
>  Issue Type: Improvement
>  Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> It would be a good addition to extend the pattern API to support quantifiers 
> known from regular expressions (e.g. Kleene star, ?, +, or count bounds). 
> This would considerably enrich the set of supported patterns.
> Implementing the count bounds could be done by unrolling the pattern state. 
> In order to support the Kleene star operator, the {{NFACompiler}} has to be 
> extended to insert epsilon-transition between a Kleene start state and the 
> succeeding pattern state. In order to support {{?}}, one could insert two 
> paths from the preceding state, one which accepts the event and another which 
> directly goes into the next pattern state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2361: [FLINK-3318][cep] Add support for quantifiers to C...

[mushketyk]

Github user mushketyk closed the pull request at:

https://github.com/apache/flink/pull/2361


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3318) Add support for quantifiers to CEP's pattern API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417947#comment-15417947
 ] 

ASF GitHub Bot commented on FLINK-3318:
---

GitHub user mushketyk opened a pull request:

https://github.com/apache/flink/pull/2361

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink cep-operators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2361


commit 96c077a4c1c2c1ccba678ec863775402554d6dcf
Author: Ivan Mushketyk 
Date:   2016-08-05T20:05:39Z

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

commit 425fa3d94d15ea6b1396e7bf7a901f7318f107b0
Author: Ivan Mushketyk 
Date:   2016-08-11T21:10:43Z

[FLINK-3318][cep] Add documentation about pattern quantifiers




> Add support for quantifiers to CEP's pattern API
> 
>
> Key: FLINK-3318
> URL: https://issues.apache.org/jira/browse/FLINK-3318
> Project: Flink
>  Issue Type: Improvement
>  Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> It would be a good addition to extend the pattern API to support quantifiers 
> known from regular expressions (e.g. Kleene star, ?, +, or count bounds). 
> This would considerably enrich the set of supported patterns.
> Implementing the count bounds could be done by unrolling the pattern state. 
> In order to support the Kleene star operator, the {{NFACompiler}} has to be 
> extended to insert epsilon-transition between a Kleene start state and the 
> succeeding pattern state. In order to support {{?}}, one could insert two 
> paths from the preceding state, one which accepts the event and another which 
> directly goes into the next pattern state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2361: [FLINK-3318][cep] Add support for quantifiers to C...

[mushketyk]

GitHub user mushketyk opened a pull request:

https://github.com/apache/flink/pull/2361

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink cep-operators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2361


commit 96c077a4c1c2c1ccba678ec863775402554d6dcf
Author: Ivan Mushketyk 
Date:   2016-08-05T20:05:39Z

[FLINK-3318][cep] Add support for quantifiers to CEP's pattern API

commit 425fa3d94d15ea6b1396e7bf7a901f7318f107b0
Author: Ivan Mushketyk 
Date:   2016-08-11T21:10:43Z

[FLINK-3318][cep] Add documentation about pattern quantifiers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417880#comment-15417880
 ] 

Elias Levy commented on FLINK-4035:
---

FWIW I generated a flink-connector-kafka-0.9_2.11-1.1.1.jar that uses 
kaka-clients 0.10.0.1 (it required hacking around some issues in one of the 
tests which I largely ignore).  I've tested it in a Flink 1.1.1 cluster against 
a 0.10.0.1 Kafka cluster without any issues.  Making use of the 0.10.0.1 
clients dropped the CPU usage on the Kafka brokers from 100% to 2%, as 
previously the broker had to transcode the messages from the 0.10 format to the 
0.9 format, whereas with the 0.10 client it can make use of zero copy from disk 
to the socket.

It is really too bad that the Kafka clients are not backwards compatible with 
older brokers.  If they were, that would obviate the need to support multiple 
Kafka client version concurrently in Flink and similar system.  We'd just have 
to keep up with the latest version of the client.

> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Assignee: Robert Metzger
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4198) Replace org.apache.flink.streaming.api.windowing.time.Time with org.apache.flink.api.common.time.Time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417764#comment-15417764
 ] 

ASF GitHub Bot commented on FLINK-4198:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2359
  
I think this breaks the API compatibility.
This issue was scheduled as part of the API breaking changes for FLINK 2.0

Compatibility for `@Public` annotated methods is something we promised in 
the 1.0 release, so we have to stick with it.


> Replace org.apache.flink.streaming.api.windowing.time.Time with 
> org.apache.flink.api.common.time.Time
> -
>
> Key: FLINK-4198
> URL: https://issues.apache.org/jira/browse/FLINK-4198
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
> Fix For: 2.0.0
>
>
> Remove {{org.apache.flink.streaming.api.windowing.time.Time}} and replace it 
> with {{org.apache.flink.api.common.time.Time}} which resides in 
> {{flink-core}}. The latter is basically the copy of the former which has been 
> moved to {{flink-core}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2359: [FLINK-4198] Replace org.apache.flink.streaming.api.windo...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2359
  
I think this breaks the API compatibility.
This issue was scheduled as part of the API breaking changes for FLINK 2.0

Compatibility for `@Public` annotated methods is something we promised in 
the 1.0 release, so we have to stick with it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4386) Add as way to assert that code runs in the RpcEndpoint's Main Thread

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4386:
---

 Summary: Add as way to assert that code runs in the RpcEndpoint's 
Main Thread
 Key: FLINK-4386
 URL: https://issues.apache.org/jira/browse/FLINK-4386
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
 Environment: FLIP-6 feature branch
Reporter: Stephan Ewen
 Fix For: 1.2.0


It would greatly help stability if we were able to add assertions to the code, 
like

{code}
private void someCallbackHandler() {
assert isRunningInMainThread()
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4385) Union on Timestamp fields does not work

[Timo Walther (JIRA)]

Timo Walther created FLINK-4385:
---

 Summary: Union on Timestamp fields does not work
 Key: FLINK-4385
 URL: https://issues.apache.org/jira/browse/FLINK-4385
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: Timo Walther


The following does not work:

{code}
public static class SDF {
public Timestamp t = Timestamp.valueOf("1990-10-10 12:10:10");
}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet dataSet1 = env.fromElements(new SDF());
DataSet dataSet2 = env.fromElements(new SDF());

BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
tableEnv.registerDataSet( "table0", dataSet1 );
tableEnv.registerDataSet( "table1", dataSet2 );
Table table = tableEnv.sql( "select t from table0 union select t from table1" );
DataSet d = tableEnv.toDataSet(table, Row.class);
d.print();
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4382) Buffer rpc calls until RpcEndpoint is properly started

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417632#comment-15417632
 ] 

ASF GitHub Bot commented on FLINK-4382:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2358
  
Not completely sure about the message stashing. Why not drop the messages? 
That would be safer (less to clean up, no overflow).

Is this mainly to not loose the messages from the "self gateway"?  Because 
for all remote messages, it should be not necessary, it should simply appear as 
if the actor/endpoint started a few milliseconds later.

It may in the common case not be much of an issue, as the period during 
which messages are stashed is small, but I think it is simply not necessary.


> Buffer rpc calls until RpcEndpoint is properly started
> --
>
> Key: FLINK-4382
> URL: https://issues.apache.org/jira/browse/FLINK-4382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> When creating a {{RpcEndpoint}} it starts a rpc server. The server should 
> wait to dispatch incoming rpc calls until the {{RpcEndpoint}} signals that 
> it's ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2358: [FLINK-4382] Buffer rpc calls until the RpcEndpoint has b...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2358
  
Not completely sure about the message stashing. Why not drop the messages? 
That would be safer (less to clean up, no overflow).

Is this mainly to not loose the messages from the "self gateway"?  Because 
for all remote messages, it should be not necessary, it should simply appear as 
if the actor/endpoint started a few milliseconds later.

It may in the common case not be much of an issue, as the period during 
which messages are stashed is small, but I think it is simply not necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4384) Add a "scheduleRunAsync()" feature to the RpcEndpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417626#comment-15417626
 ] 

ASF GitHub Bot commented on FLINK-4384:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2360

[FLINK-4384] [rpc] Add "scheduleRunAsync()" to the RpcEndpoint

NOTE: This builds on top of #2357 - only the second commit belongs actually 
to this pull request is relevant.

Add a `scheduleRunAsync()` method to the `RpcEndpoint`. It behaves like 
`runAsync()` but delays the call by a given number of milliseconds. The delay 
does not happen by a thread sleep, but by scheduling the message that triggers 
the Runnable it into the future of the message dispatcher.

This also adds tests for the `runAsync()`, `scheduleRunAsync()`, and 
`callAsync()` that validate that all these calls actually run in the same 
thread (the RPC endpoint's main thread).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink schedule_future

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2360


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service

commit 7b9eb2f3de23a7bee28346664ff39e0c28235a6e
Author: Stephan Ewen 
Date:   2016-08-11T17:10:48Z

[FLINK-4384] [rpc] Add "scheduleRunAsync()" to the RpcEndpoint




> Add a "scheduleRunAsync()" feature to the RpcEndpoint
> -
>
> Key: FLINK-4384
> URL: https://issues.apache.org/jira/browse/FLINK-4384
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
> Fix For: 1.2.0
>
>
> It is a common pattern to schedule a call to be executed in the future. 
> Examples are
>   - delays in retries
>   - heartbeats,
>   - checking for heartbeat timeouts
> I suggest to add a {{scheduleRunAsync()}} method to the {{RpcEndpoint}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2360: [FLINK-4384] [rpc] Add "scheduleRunAsync()" to the...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2360

[FLINK-4384] [rpc] Add "scheduleRunAsync()" to the RpcEndpoint

NOTE: This builds on top of #2357 - only the second commit belongs actually 
to this pull request is relevant.

Add a `scheduleRunAsync()` method to the `RpcEndpoint`. It behaves like 
`runAsync()` but delays the call by a given number of milliseconds. The delay 
does not happen by a thread sleep, but by scheduling the message that triggers 
the Runnable it into the future of the message dispatcher.

This also adds tests for the `runAsync()`, `scheduleRunAsync()`, and 
`callAsync()` that validate that all these calls actually run in the same 
thread (the RPC endpoint's main thread).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink schedule_future

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2360


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service

commit 7b9eb2f3de23a7bee28346664ff39e0c28235a6e
Author: Stephan Ewen 
Date:   2016-08-11T17:10:48Z

[FLINK-4384] [rpc] Add "scheduleRunAsync()" to the RpcEndpoint




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4374) GroupReduce Broken for null Date

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417617#comment-15417617
 ] 

Stephan Ewen commented on FLINK-4374:
-

True, but the test should not try a Tuple2 if tuples do not handle 
nulls. It's bound to produce something unpredictable.

> GroupReduce Broken for null Date
> 
>
> Key: FLINK-4374
> URL: https://issues.apache.org/jira/browse/FLINK-4374
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Reporter: Stefan Richter
>Assignee: Timo Walther
>
> The GroupReduceITCase has an error that allows a problem with {{null}} Dates 
> to go uncovered:
>  If I set the parallelism to 1 in {{testDateNullException()}} and all keys 
> actually end up on the same operator, then there is a problem in the 
> de/serialization.
> It seems that {{null}} values are somehow skipped by the serialization 
> process (e.g. maybe no {{null}} indicator is written), which leads to wrong 
> deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417611#comment-15417611
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2357
  
+1 from my side

I would suggest as a followup to find out the gateway type for reflection. 
The `ReflectionUtil` helps with that.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via Java pr...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2357
  
+1 from my side

I would suggest as a followup to find out the gateway type for reflection. 
The `ReflectionUtil` helps with that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417609#comment-15417609
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74463001
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

We should not, but it happened accidentally in the past (user exception 
reporting, user state handles, user accumulators) and we always figured it out 
late because it did not fail hard but only dropped the message.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74463001
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

We should not, but it happened accidentally in the past (user exception 
reporting, user state handles, user accumulators) and we always figured it out 
late because it did not fail hard but only dropped the message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2359: [FLINK-4198] Replace org.apache.flink.streaming.ap...

[kishorekgarg]

GitHub user kishorekgarg opened a pull request:

https://github.com/apache/flink/pull/2359

[FLINK-4198] Replace org.apache.flink.streaming.api.windowing.time.Ti…

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

…me with org.apache.flink.api.common.time.Time

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kishorekgarg/flink master_4198

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2359.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2359


commit a36c3290b5710dc4412dcc00ae82162631d79b62
Author: Kishore 
Date:   2016-08-11T16:42:41Z

[FLINK-4198] Replace org.apache.flink.streaming.api.windowing.time.Time 
with org.apache.flink.api.common.time.Time




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4198) Replace org.apache.flink.streaming.api.windowing.time.Time with org.apache.flink.api.common.time.Time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417562#comment-15417562
 ] 

ASF GitHub Bot commented on FLINK-4198:
---

GitHub user kishorekgarg opened a pull request:

https://github.com/apache/flink/pull/2359

[FLINK-4198] Replace org.apache.flink.streaming.api.windowing.time.Ti…

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

…me with org.apache.flink.api.common.time.Time

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kishorekgarg/flink master_4198

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2359.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2359


commit a36c3290b5710dc4412dcc00ae82162631d79b62
Author: Kishore 
Date:   2016-08-11T16:42:41Z

[FLINK-4198] Replace org.apache.flink.streaming.api.windowing.time.Time 
with org.apache.flink.api.common.time.Time




> Replace org.apache.flink.streaming.api.windowing.time.Time with 
> org.apache.flink.api.common.time.Time
> -
>
> Key: FLINK-4198
> URL: https://issues.apache.org/jira/browse/FLINK-4198
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
> Fix For: 2.0.0
>
>
> Remove {{org.apache.flink.streaming.api.windowing.time.Time}} and replace it 
> with {{org.apache.flink.api.common.time.Time}} which resides in 
> {{flink-core}}. The latter is basically the copy of the former which has been 
> moved to {{flink-core}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4383) Check parameters for serializability before sending a remote RpcInvocation message

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-4383:


Assignee: Till Rohrmann

> Check parameters for serializability before sending a remote RpcInvocation 
> message
> --
>
> Key: FLINK-4383
> URL: https://issues.apache.org/jira/browse/FLINK-4383
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> Before sending a remote {{RpcInvocation}} message we should check that the 
> rpc arguments are serializable. If not we should eagerly fail with an 
> appropriate exception message.
> If we don't do this, then Akka will silently fail serializing the message 
> without telling the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4382) Buffer rpc calls until RpcEndpoint is properly started

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417538#comment-15417538
 ] 

ASF GitHub Bot commented on FLINK-4382:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2358

[FLINK-4382] Buffer rpc calls until the RpcEndpoint has been started

This PR allows the AkkaRpcActor to stash messages until the corresponding 
RcpEndpoint
has been started. When receiving a Processing.START message, the 
AkkaRpcActor
unstashes all messages and starts processing rpcs. When receiving a 
Processing.STOP
message, it will stop processing messages and stash incoming messages again.

This PR is based on #2357.

R @StephanEwen 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink messageStashing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2358


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service

commit 32dbb077e49c8d6180afaaf844238fd6a3178395
Author: Till Rohrmann 
Date:   2016-08-11T15:27:18Z

Log unknown message type in AkkaRpcActor but do not fail actor

commit 2c8062fb5ad567932bb63a9c005587773e0f0ab9
Author: Till Rohrmann 
Date:   2016-08-11T16:13:25Z

[FLINK-4382] [rpc] Buffer rpc calls until the RpcEndpoint has been started

This PR allows the AkkaRpcActor to stash messages until the corresponding 
RcpEndpoint
has been started. When receiving a Processing.START message, the 
AkkaRpcActor
unstashes all messages and starts processing rpcs. When receiving a 
Processing.STOP
message, it will stop processing messages and stash incoming messages again.

commit 3477f1d819d4a30d1fe73d0689117acfc63c8534
Author: Till Rohrmann 
Date:   2016-08-11T16:35:03Z

Add test case for message stashing




> Buffer rpc calls until RpcEndpoint is properly started
> --
>
> Key: FLINK-4382
> URL: https://issues.apache.org/jira/browse/FLINK-4382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> When creating a {{RpcEndpoint}} it starts a rpc server. The server should 
> wait to dispatch incoming rpc calls until the {{RpcEndpoint}} signals that 
> it's ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2358: [FLINK-4382] Buffer rpc calls until the RpcEndpoin...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2358

[FLINK-4382] Buffer rpc calls until the RpcEndpoint has been started

This PR allows the AkkaRpcActor to stash messages until the corresponding 
RcpEndpoint
has been started. When receiving a Processing.START message, the 
AkkaRpcActor
unstashes all messages and starts processing rpcs. When receiving a 
Processing.STOP
message, it will stop processing messages and stash incoming messages again.

This PR is based on #2357.

R @StephanEwen 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink messageStashing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2358


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service

commit 32dbb077e49c8d6180afaaf844238fd6a3178395
Author: Till Rohrmann 
Date:   2016-08-11T15:27:18Z

Log unknown message type in AkkaRpcActor but do not fail actor

commit 2c8062fb5ad567932bb63a9c005587773e0f0ab9
Author: Till Rohrmann 
Date:   2016-08-11T16:13:25Z

[FLINK-4382] [rpc] Buffer rpc calls until the RpcEndpoint has been started

This PR allows the AkkaRpcActor to stash messages until the corresponding 
RcpEndpoint
has been started. When receiving a Processing.START message, the 
AkkaRpcActor
unstashes all messages and starts processing rpcs. When receiving a 
Processing.STOP
message, it will stop processing messages and stash incoming messages again.

commit 3477f1d819d4a30d1fe73d0689117acfc63c8534
Author: Till Rohrmann 
Date:   2016-08-11T16:35:03Z

Add test case for message stashing




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417522#comment-15417522
 ] 

Elias Levy commented on FLINK-4326:
---

That would be my expectation.  Writing out the PID would be nice, but it is not 
a requirement.  Process management becomes the job of the supervisor process.

Log handling is different.  Some supervisors can handle logs output to stdout, 
some don't.  So even when outputting logs to stdout, the process should still 
be capable of outputting to other locations, like a log directory or to syslog.

> Flink start-up scripts should optionally start services on the foreground
> -
>
> Key: FLINK-4326
> URL: https://issues.apache.org/jira/browse/FLINK-4326
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>
> This has previously been mentioned in the mailing list, but has not been 
> addressed.  Flink start-up scripts start the job and task managers in the 
> background.  This makes it difficult to integrate Flink with most processes 
> supervisory tools and init systems, including Docker.  One can get around 
> this via hacking the scripts or manually starting the right classes via Java, 
> but it is a brittle solution.
> In addition to starting the daemons in the foreground, the start up scripts 
> should use exec instead of running the commends, so as to avoid forks.  Many 
> supervisory tools assume the PID of the process to be monitored is that of 
> the process it first executes, and fork chains make it difficult for the 
> supervisor to figure out what process to monitor.  Specifically, 
> jobmanager.sh and taskmanager.sh should exec flink-daemon.sh, and 
> flink-daemon.sh should exec java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417467#comment-15417467
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
And remember that my ultimate goal is not to host that image but that we 
can convert it into an official one.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
And remember that my ultimate goal is not to host that image but that we 
can convert it into an official one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417465#comment-15417465
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
Of course, who do you think I am :P

You can check a ready to run build here in case you don't want to test that 
one. But it is true that we need some kind of automatized tests for the image.

https://hub.docker.com/r/iemejia/flink/


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2340
  
Of course, who do you think I am :P

You can check a ready to run build here in case you don't want to test that 
one. But it is true that we need some kind of automatized tests for the image.

https://hub.docker.com/r/iemejia/flink/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417458#comment-15417458
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74446628
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/RpcEndpoint.java ---
@@ -172,6 +173,13 @@ public void runAsync(Runnable runnable) {
return ((MainThreadExecutor) self).callAsync(callable, timeout);
}
 
+   /**
+* Returns the class of the self gateway type.
+*
+* @return Class of the self gateway type
+*/
+   public abstract Class getSelfGatewayType();
--- End diff --

I think one can find this out via reflection. The parameter is stored in 
the class signature.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74446628
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/RpcEndpoint.java ---
@@ -172,6 +173,13 @@ public void runAsync(Runnable runnable) {
return ((MainThreadExecutor) self).callAsync(callable, timeout);
}
 
+   /**
+* Returns the class of the self gateway type.
+*
+* @return Class of the self gateway type
+*/
+   public abstract Class getSelfGatewayType();
--- End diff --

I think one can find this out via reflection. The parameter is stored in 
the class signature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417450#comment-15417450
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2340
  
Thanks! Looks good. Did you run the changes with Docker?


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2340
  
Thanks! Looks good. Did you run the changes with Docker?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417447#comment-15417447
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74445659
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

Still would like to keep it because it is strongly encouraged to not use 
the load balanced servers instead of the archive server.


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74445659
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

Still would like to keep it because it is strongly encouraged to not use 
the load balanced servers instead of the archive server.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3155) Update Flink docker version to latest stable Flink version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417442#comment-15417442
 ] 

ASF GitHub Bot commented on FLINK-3155:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74445485
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

I see your point concerning fragility. You're probably right. Btw, you can 
always validate the download using the MD5/SHA256 files from the archive server 
:)


> Update Flink docker version to latest stable Flink version
> --
>
> Key: FLINK-3155
> URL: https://issues.apache.org/jira/browse/FLINK-3155
> Project: Flink
>  Issue Type: Task
>  Components: flink-contrib
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Maximilian Michels
>Priority: Minor
> Fix For: 1.0.0
>
>
> It would be nice to always set the Docker Flink binary URL to point to the 
> latest Flink version. Until then, this JIRA keeps track of the updates for 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2340#discussion_r74445485
  
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
 RUN apk add --no-cache bash snappy
 
 # Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
 ARG HADOOP_VERSION=27
 ARG SCALA_VERSION=2.11
 
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
 # Install build dependencies and flink
 RUN set -x && \
+  mkdir -p /opt && \
   apk --update add --virtual build-dependencies curl && \
-  curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
-  awk '/preferred/ {gsub(/"/,""); print 
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
 | \
-  tar xvz -C /usr/local/ && \
-  ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
-  sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g" 
/usr/local/flink/bin/flink-daemon.sh && \
+  curl -s 
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
 | \
--- End diff --

I see your point concerning fragility. You're probably right. Btw, you can 
always validate the download using the MD5/SHA256 files from the archive server 
:)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4384) Add a "scheduleRunAsync()" feature to the RpcEndpoint

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4384:
---

 Summary: Add a "scheduleRunAsync()" feature to the RpcEndpoint
 Key: FLINK-4384
 URL: https://issues.apache.org/jira/browse/FLINK-4384
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
 Environment: FLIP-6 feature branch
Reporter: Stephan Ewen
 Fix For: 1.2.0


It is a common pattern to schedule a call to be executed in the future. 
Examples are
  - delays in retries
  - heartbeats,
  - checking for heartbeat timeouts

I suggest to add a {{scheduleRunAsync()}} method to the {{RpcEndpoint}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4382) Buffer rpc calls until RpcEndpoint is properly started

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417432#comment-15417432
 ] 

Stephan Ewen commented on FLINK-4382:
-

Would it buffer the messages, or simply discard them while the endpoint is not 
ready?
I think discarding is fine. It would be equivalent to the behavior that the 
endpoint started later (was unavailable for some time), which all senders have 
to be able to deal with anyways.

> Buffer rpc calls until RpcEndpoint is properly started
> --
>
> Key: FLINK-4382
> URL: https://issues.apache.org/jira/browse/FLINK-4382
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> When creating a {{RpcEndpoint}} it starts a rpc server. The server should 
> wait to dispatch incoming rpc calls until the {{RpcEndpoint}} signals that 
> it's ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2275: FLINK-3929 Support for Kerberos Authentication with Keyta...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
>HDFS and Yarn are handled through the @BeforeClass and @AfterClass style 
and they do not use custom JRunner implementation. As you have suggested, I 
could keep just one or two tests for each of the modules to cut down the 
running time, if that's okay with you?

Thanks! Yes, please just one test per entity (HDFS, Yarn, Kafka/Zookeeper). 
Could you also convert the Kafka test to using `@BeforeClass` and 
`@AfterClass`? You don't necessarily have to duplicate code. How about changing 
the test base to include a method to instantiate the secure settings? I think 
you'll need to add an abstract method in `KafkaTestEnvironment`, e.g. 
loadSecureSettings(), then add another one in `KafkaTestBase`, e.g. 
getTestEnvironmentClass(), to load the appropriate test environment 
(secure/non-secure). You will have additional classes that implement these two 
methods. These classes will be very short as they just overload the method. 
This is cleaner although a bit more verbose.

>I am open to keep just 3 classes for each scenarios (HDFS, Yarn & Kafka) 
as you have suggested but in my opinion that will defeat the idea of reusing 
existing test program.

I understand but we're actually just increasing the testing time and not 
gaining much from running multiple security tests for the same component.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417431#comment-15417431
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
>HDFS and Yarn are handled through the @BeforeClass and @AfterClass style 
and they do not use custom JRunner implementation. As you have suggested, I 
could keep just one or two tests for each of the modules to cut down the 
running time, if that's okay with you?

Thanks! Yes, please just one test per entity (HDFS, Yarn, Kafka/Zookeeper). 
Could you also convert the Kafka test to using `@BeforeClass` and 
`@AfterClass`? You don't necessarily have to duplicate code. How about changing 
the test base to include a method to instantiate the secure settings? I think 
you'll need to add an abstract method in `KafkaTestEnvironment`, e.g. 
loadSecureSettings(), then add another one in `KafkaTestBase`, e.g. 
getTestEnvironmentClass(), to load the appropriate test environment 
(secure/non-secure). You will have additional classes that implement these two 
methods. These classes will be very short as they just overload the method. 
This is cleaner although a bit more verbose.

>I am open to keep just 3 classes for each scenarios (HDFS, Yarn & Kafka) 
as you have suggested but in my opinion that will defeat the idea of reusing 
existing test program.

I understand but we're actually just increasing the testing time and not 
gaining much from running multiple security tests for the same component.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4368) Eagerly initialize RrcProtocol members

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-4368.
---

> Eagerly initialize RrcProtocol members
> --
>
> Key: FLINK-4368
> URL: https://issues.apache.org/jira/browse/FLINK-4368
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
> {{start()}} call.
> I suggest to initialize them eagerly as they seem to be integral parts 
> without which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4366) Enforce parallelism=1 For AllWindowedStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417423#comment-15417423
 ] 

ASF GitHub Bot commented on FLINK-4366:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2354
  
Something like `NonParallelSingleOutputStreamOperator` but that's quite a 
mouthful. (`SingleOutputStreamOperator` is already too long for my taste ...). 
Or maybe `NonParallelStreamOperator`.

Maybe we could have it like you initially proposed but not named 
`allWindow` and in such a way that you cannot switch it back. For example: 
`SingleOutputStreamOperator.forceNonParallel()` that would set an internal flag 
that can never be unset from the outside again.


> Enforce parallelism=1 For AllWindowedStream
> ---
>
> Key: FLINK-4366
> URL: https://issues.apache.org/jira/browse/FLINK-4366
> Project: Flink
>  Issue Type: Improvement
>Reporter: Aljoscha Krettek
>Assignee: Jark Wu
>
> Right now, it is possible to use {{DataStream.windowAll/timeWindowAll}} and 
> then set a different parallelism afterwards. Flink will silently accept this 
> and spawn the number of parallel operators, only one instance of those will 
> do all the processing, though, since the elements are implicitly keyed by a 
> dummy key.
> We should throw an exception if users try to set a parallelism on an 
> all-windowed stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417422#comment-15417422
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443700
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+   

[GitHub] flink issue #2354: [FLINK-4366] Enforce parallelism=1 For AllWindowedStream

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2354
  
Something like `NonParallelSingleOutputStreamOperator` but that's quite a 
mouthful. (`SingleOutputStreamOperator` is already too long for my taste ...). 
Or maybe `NonParallelStreamOperator`.

Maybe we could have it like you initially proposed but not named 
`allWindow` and in such a way that you cannot switch it back. For example: 
`SingleOutputStreamOperator.forceNonParallel()` that would set an internal flag 
that can never be unset from the outside again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443700
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+   } else {
+   try {
+   Object result = 
rpcMethod.invoke(rpcEndpoint, rpcInvocation.getArgs());
+
+   if (result 

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417417#comment-15417417
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2357
  
Yes you're right @StephanEwen. We shouldn't stop the actor upon receiving 
an unknown message type. I'll change the behaviour.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417415#comment-15417415
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443304
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

That is correct. I'm wondering though, whether we'll ever send a message of 
a type which is not contained in the system class loader?


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via Java pr...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2357
  
Yes you're right @StephanEwen. We shouldn't stop the actor upon receiving 
an unknown message type. I'll change the behaviour.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443304
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

That is correct. I'm wondering though, whether we'll ever send a message of 
a type which is not contained in the system class loader?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417412#comment-15417412
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443020
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+   

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74443020
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+   } else {
+   try {
+   Object result = 
rpcMethod.invoke(rpcEndpoint, rpcInvocation.getArgs());
+
+   if (result 

[jira] [Created] (FLINK-4383) Check parameters for serializability before sending a remote RpcInvocation message

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-4383:


 Summary: Check parameters for serializability before sending a 
remote RpcInvocation message
 Key: FLINK-4383
 URL: https://issues.apache.org/jira/browse/FLINK-4383
 Project: Flink
  Issue Type: Sub-task
Reporter: Till Rohrmann


Before sending a remote {{RpcInvocation}} message we should check that the rpc 
arguments are serializable. If not we should eagerly fail with an appropriate 
exception message.

If we don't do this, then Akka will silently fail serializing the message 
without telling the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2337: [FLINK-3042] [FLINK-3060] [types] Define a way to let typ...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2337
  
@StephanEwen I have updated the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4382) Buffer rpc calls until RpcEndpoint is properly started

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-4382:


 Summary: Buffer rpc calls until RpcEndpoint is properly started
 Key: FLINK-4382
 URL: https://issues.apache.org/jira/browse/FLINK-4382
 Project: Flink
  Issue Type: Sub-task
Reporter: Till Rohrmann
Assignee: Till Rohrmann


When creating a {{RpcEndpoint}} it starts a rpc server. The server should wait 
to dispatch incoming rpc calls until the {{RpcEndpoint}} signals that it's 
ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3042) Define a way to let types create their own TypeInformation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417394#comment-15417394
 ] 

ASF GitHub Bot commented on FLINK-3042:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2337
  
@StephanEwen I have updated the PR.


> Define a way to let types create their own TypeInformation
> --
>
> Key: FLINK-3042
> URL: https://issues.apache.org/jira/browse/FLINK-3042
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Timo Walther
> Fix For: 1.0.0
>
>
> Currently, introducing new Types that should have specific TypeInformation 
> requires
>   - Either integration with the TypeExtractor
>   - Or manually constructing the TypeInformation (potentially at every place) 
> and using type hints everywhere.
> I propose to add a way to allow classes to create their own TypeInformation 
> (like a static method "createTypeInfo()").
> To support generic nested types (like Optional / Either), the type extractor 
> would provide a Map of what generic variables map to what types (deduced from 
> the input). The class can use that to create the correct nested 
> TypeInformation (possibly by calling the TypeExtractor again, passing the Map 
> of generic bindings).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417385#comment-15417385
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74439866
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

Similar as giving a better exception on non-serializable types, we can try 
and give a better exception on Classes that are not part of the system class 
loader.

The current Akka behavior (dropping the message) was always a bit to 
silent. Sending back an exception could help catching these things more easily.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74439866
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/messages/RpcInvocation.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka.messages;
+
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+
+/**
+ * Rpc invocation message containing the remote procedure name, its 
parameter types and the
+ * corresponding call arguments.
+ */
+public final class RpcInvocation implements Serializable {
+   private static final long serialVersionUID = -7058254033460536037L;
+
+   private final String methodName;
+   private final Class[] parameterTypes;
--- End diff --

Similar as giving a better exception on non-serializable types, we can try 
and give a better exception on Classes that are not part of the system class 
loader.

The current Akka behavior (dropping the message) was always a bit to 
silent. Sending back an exception could help catching these things more easily.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417379#comment-15417379
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74439096
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2357#discussion_r74439096
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcActor.java 
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import akka.actor.Status;
+import akka.actor.UntypedActor;
+import akka.pattern.Patterns;
+import org.apache.flink.runtime.rpc.RpcEndpoint;
+import org.apache.flink.runtime.rpc.RpcGateway;
+import org.apache.flink.runtime.rpc.akka.messages.CallAsync;
+import org.apache.flink.runtime.rpc.akka.messages.RpcInvocation;
+import org.apache.flink.runtime.rpc.akka.messages.RunAsync;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.concurrent.Future;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Akka rpc actor which receives {@link RpcInvocation}, {@link RunAsync} 
and {@link CallAsync}
+ * messages.
+ * 
+ * The {@link RpcInvocation} designates a rpc and is dispatched to the 
given {@link RpcEndpoint}
+ * instance.
+ * 
+ * The {@link RunAsync} and {@link CallAsync} messages contain executable 
code which is executed
+ * in the context of the actor thread.
+ *
+ * @param  Type of the {@link RpcGateway} associated with the {@link 
RpcEndpoint}
+ * @param  Type of the {@link RpcEndpoint}
+ */
+class AkkaRpcActor> extends 
UntypedActor {
+   private static final Logger LOG = 
LoggerFactory.getLogger(AkkaRpcActor.class);
+
+   private final T rpcEndpoint;
+
+   AkkaRpcActor(final T rpcEndpoint) {
+   this.rpcEndpoint = Preconditions.checkNotNull(rpcEndpoint, "rpc 
endpoint");
+   }
+
+   @Override
+   public void onReceive(final Object message)  {
+   if (message instanceof RunAsync) {
+   handleRunAsync((RunAsync) message);
+   } else if (message instanceof CallAsync) {
+   handleCallAsync((CallAsync) message);
+   } else if (message instanceof RpcInvocation) {
+   handleRpcInvocation((RpcInvocation) message);
+   } else {
+   throw new RuntimeException("Encountered unknown message 
type " + message.getClass() +
+   " with value " + message + '.');
+   }
+   }
+
+   /**
+* Handle rpc invocations by looking up the rpc method on the rpc 
endpoint and calling this
+* method with the provided method arguments. If the method has a 
return value, it is returned
+* to the sender of the call.
+*
+* @param rpcInvocation Rpc invocation message
+*/
+   private void handleRpcInvocation(RpcInvocation rpcInvocation) {
+   Method rpcMethod = null;
+
+   try {
+   rpcMethod = 
lookupRpcMethod(rpcInvocation.getMethodName(), 
rpcInvocation.getParameterTypes());
+   } catch (final NoSuchMethodException e) {
+   LOG.error("Could not find rpc method for rpc 
invocation: {}.", rpcInvocation, e);
+   }
+
+   if (rpcMethod != null) {
+   if (rpcMethod.getReturnType().equals(Void.TYPE)) {
+   // No return value to send back
+   try {
+   rpcMethod.invoke(rpcEndpoint, 
rpcInvocation.getArgs());
+   } catch (Throwable e) {
+   LOG.error("Error while executing remote 
procedure call {}.", rpcMethod, e);
+   }
+   } else {
+   try {
+   Object result = 
rpcMethod.invoke(rpcEndpoint, rpcInvocation.getArgs());
+
+   if (result 

[GitHub] flink issue #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via Java pr...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2357
  
Good stuff!

I am not sure we should throw an Exception in the `AkkaRpcActor` upon 
receiving an incorrect message. In the past we decided to only log those cases. 
Seems to easy to bring down the system with a poison message otherwise.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417376#comment-15417376
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2357
  
Good stuff!

I am not sure we should throw an Exception in the `AkkaRpcActor` upon 
receiving an incorrect message. In the past we decided to only log those cases. 
Seems to easy to bring down the system with a poison message otherwise.


> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4377) akka.remote.OversizedPayloadException: Discarding oversized payload

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417338#comment-15417338
 ] 

Stephan Ewen commented on FLINK-4377:
-

Yes, this can currently happen. The only way to immediately solve this is to 
increase the akka maximum frame size.

We are prototyping a new RPC abstraction on top of Akka, which will be a good 
point to solve this problem.

> akka.remote.OversizedPayloadException: Discarding oversized payload
> ---
>
> Key: FLINK-4377
> URL: https://issues.apache.org/jira/browse/FLINK-4377
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Affects Versions: 1.1.0
> Environment: Linux
>Reporter: Sajeev Ramakrishnan
>
> Dear Team,
>   I was trying to create a hash map with a size around 1 million. Then I 
> encountered the below issue. For smaller maps, I am not seeing any issues.
> akka.remote.OversizedPayloadException: Discarding oversized payload sent to 
> Actor[akka.tcp://flink@localhost:58247/user/$d#458673459]: max allowed size 
> 10485760 bytes, actual size of encoded class 
> org.apache.flink.runtime.messages.JobManagerMessages$JobResultSuccess was 
> 175254213 bytes.
> Regards,
> Sajeev



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4362) Auto generate message sender classes via Java Proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417309#comment-15417309
 ] 

ASF GitHub Bot commented on FLINK-4362:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2357

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic `AkkaRpcActor` which receives rpc calls as a
`RpcInvocation` message. The `RpcInvocation` message is generated by the
`AkkaInvocationHandler` which gets them from automatically generated Java 
Proxies.
The Java proxies are generated in the `AkkaRpcService` class upon 
connection or 
starting of a rpc server.

R @mxm, @StephanEwen 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink proxyRpc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2357


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service




> Auto generate message sender classes via Java Proxies
> -
>
> Key: FLINK-4362
> URL: https://issues.apache.org/jira/browse/FLINK-4362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The first version of the RPC service needs to manually create the sender 
> classes, which turn method calls into messages.
> This can be automated by using Java Proxies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417302#comment-15417302
 ] 

Stephan Ewen commented on FLINK-4370:
-

In theory, putting it in "tools" and referencing in the docs is nice.
In practice, many people do not read these things let alone import settings. I 
guess that if it does not work automatically, many contributors will not use it.

> Offer a default IntelliJ inspection profile with Flink
> --
>
> Key: FLINK-4370
> URL: https://issues.apache.org/jira/browse/FLINK-4370
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
> should be automatically picked up when the code is checked out and imported 
> into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4374) GroupReduce Broken for null Date

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417316#comment-15417316
 ] 

Timo Walther commented on FLINK-4374:
-

The null support in general is a bit inconsistent. The Date or String 
serializer support null values, but the comparators don't. 

> GroupReduce Broken for null Date
> 
>
> Key: FLINK-4374
> URL: https://issues.apache.org/jira/browse/FLINK-4374
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Reporter: Stefan Richter
>Assignee: Timo Walther
>
> The GroupReduceITCase has an error that allows a problem with {{null}} Dates 
> to go uncovered:
>  If I set the parallelism to 1 in {{testDateNullException()}} and all keys 
> actually end up on the same operator, then there is a problem in the 
> de/serialization.
> It seems that {{null}} values are somehow skipped by the serialization 
> process (e.g. maybe no {{null}} indicator is written), which leads to wrong 
> deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4368) Eagerly initialize RrcProtocol members

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-4368:
-
Assignee: Stephan Ewen

> Eagerly initialize RrcProtocol members
> --
>
> Key: FLINK-4368
> URL: https://issues.apache.org/jira/browse/FLINK-4368
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
> {{start()}} call.
> I suggest to initialize them eagerly as they seem to be integral parts 
> without which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4368) Eagerly initialize RrcProtocol members

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-4368.
--
Resolution: Done

Added via 08af7def60d54c22126b902e6fa57101d5fbb8fa

> Eagerly initialize RrcProtocol members
> --
>
> Key: FLINK-4368
> URL: https://issues.apache.org/jira/browse/FLINK-4368
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> The members of the RPC endpoint (RpcProtocol) are lazily created upon the 
> {{start()}} call.
> I suggest to initialize them eagerly as they seem to be integral parts 
> without which several functions cannot work properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2357: [FLINK-4362] [rpc] Auto generate rpc gateways via ...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2357

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic `AkkaRpcActor` which receives rpc calls as a
`RpcInvocation` message. The `RpcInvocation` message is generated by the
`AkkaInvocationHandler` which gets them from automatically generated Java 
Proxies.
The Java proxies are generated in the `AkkaRpcService` class upon 
connection or 
starting of a rpc server.

R @mxm, @StephanEwen 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink proxyRpc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2357


commit 13f6f392943e32b14bf0d08c7ded2d88496911ab
Author: Till Rohrmann 
Date:   2016-08-10T16:42:26Z

[FLINK-4362] [rpc] Auto generate rpc gateways via Java proxies

This PR introduces a generic AkkaRpcActor which receives rpc calls as a
RpcInvocation message. The RpcInvocation message is generated by the
AkkaInvocationHandler which gets them from automatically generated Java 
Proxies.

Add documentation for proxy based akka rpc service




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4374) GroupReduce Broken for null Date

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417298#comment-15417298
 ] 

Stephan Ewen commented on FLINK-4374:
-

Tuples do not support null fields, so the test is somewhat bogus anyways.

> GroupReduce Broken for null Date
> 
>
> Key: FLINK-4374
> URL: https://issues.apache.org/jira/browse/FLINK-4374
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Reporter: Stefan Richter
>Assignee: Timo Walther
>
> The GroupReduceITCase has an error that allows a problem with {{null}} Dates 
> to go uncovered:
>  If I set the parallelism to 1 in {{testDateNullException()}} and all keys 
> actually end up on the same operator, then there is a problem in the 
> de/serialization.
> It seems that {{null}} values are somehow skipped by the serialization 
> process (e.g. maybe no {{null}} indicator is written), which leads to wrong 
> deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4381) Refactor State to Prepare For Key-Group State Backends

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-4381:
---

 Summary: Refactor State to Prepare For Key-Group State Backends
 Key: FLINK-4381
 URL: https://issues.apache.org/jira/browse/FLINK-4381
 Project: Flink
  Issue Type: Sub-task
  Components: Streaming
Reporter: Aljoscha Krettek


In order to use the new {{KeyGroupAssigner}}/{{key group sharding}} the state 
backends need no be able to deal with key groups. For this, we first need to 
refactor the state abstractions. Specifically, this touches how key-grouped 
state should be stored and also how state is sent to the 
{{CheckpointCoordinator}} and how it is stored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4380) Introduce KeyGroupAssigner and Max-Parallelism Parameter

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-4380:
---

 Summary: Introduce KeyGroupAssigner and Max-Parallelism Parameter
 Key: FLINK-4380
 URL: https://issues.apache.org/jira/browse/FLINK-4380
 Project: Flink
  Issue Type: Sub-task
  Components: Streaming
Reporter: Aljoscha Krettek


For key-group sharding we need to introduce a {{KeyGroupAssigner}} that assigns 
key hashes to key-groups (or shards). Also, this issue is for tracking the 
addition of a {{max-parallelism}} parameter for tracking the number of key 
groups.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417270#comment-15417270
 ] 

Timo Walther commented on FLINK-4359:
-

Once the roadmap for that is defined (most likely with a corresponding FLIP). 
You are very welcome to help. I think we will discuss this in the next 1-2 
weeks.

> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.2.0
>
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417286#comment-15417286
 ] 

Greg Hogan commented on FLINK-4326:
---

Isn't the expectation that the supervisor would monitor and control process 
started in foreground mode?

I don't have an alternative to {{start-foreground}} to suggest.

> Flink start-up scripts should optionally start services on the foreground
> -
>
> Key: FLINK-4326
> URL: https://issues.apache.org/jira/browse/FLINK-4326
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>
> This has previously been mentioned in the mailing list, but has not been 
> addressed.  Flink start-up scripts start the job and task managers in the 
> background.  This makes it difficult to integrate Flink with most processes 
> supervisory tools and init systems, including Docker.  One can get around 
> this via hacking the scripts or manually starting the right classes via Java, 
> but it is a brittle solution.
> In addition to starting the daemons in the foreground, the start up scripts 
> should use exec instead of running the commends, so as to avoid forks.  Many 
> supervisory tools assume the PID of the process to be monitored is that of 
> the process it first executes, and fork chains make it difficult for the 
> supervisor to figure out what process to monitor.  Specifically, 
> jobmanager.sh and taskmanager.sh should exec flink-daemon.sh, and 
> flink-daemon.sh should exec java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4379) Add Rescalable Non-Partitioned State

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-4379:
--

 Summary: Add Rescalable Non-Partitioned State
 Key: FLINK-4379
 URL: https://issues.apache.org/jira/browse/FLINK-4379
 Project: Flink
  Issue Type: New Feature
  Components: State Backends, Checkpointing
Reporter: Ufuk Celebi


This issue is associated with [FLIP-8| 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-8%3A+Rescalable+Non-Partitioned+State].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2356: [FLINK-4378]Enable RollingSink to custom HDFS clie...

[wenlong88]

GitHub user wenlong88 opened a pull request:

https://github.com/apache/flink/pull/2356

[FLINK-4378]Enable RollingSink to custom HDFS client configuration

Adding a new interface to rolling sink


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wenlong88/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2356


commit 9b4dcff49aa8f1685ef3e38634f24b180ffcc3d5
Author: wenlong.lwl 
Date:   2016-08-11T13:26:01Z

enable rolling sink to custom hdfs client configuration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417255#comment-15417255
 ] 

Ismaël Mejía commented on FLINK-4326:
-

A separation of (daemon/console) scripts would be the nicest, no doubt. 
However, I am not sure if removing the PID code + output will be appropriate 
when we run daemons and foreground processes at the same time, how do we count 
the running instances if somebody runs a new process in foreground mode, or 
what would be the logic if we call stop-all, must we kill  all the processes 
even the foreground ones ? in these cases I think we need the PID/output refs, 
but well I am not really sure and maybe we can do such things without it.

Independent of this we must also not forget that we should preserve at least 
the same options (start|stop|stop-all) for both jobmanager.sh and taskmanager. 
because they do their magic (build the runtime options) and at the end they 
call the the  (daemon/console) script. I suppose we will need the new 
start-foreground option in these scripts too, or are there any other ideas of 
how to do it best ?


> Flink start-up scripts should optionally start services on the foreground
> -
>
> Key: FLINK-4326
> URL: https://issues.apache.org/jira/browse/FLINK-4326
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>
> This has previously been mentioned in the mailing list, but has not been 
> addressed.  Flink start-up scripts start the job and task managers in the 
> background.  This makes it difficult to integrate Flink with most processes 
> supervisory tools and init systems, including Docker.  One can get around 
> this via hacking the scripts or manually starting the right classes via Java, 
> but it is a brittle solution.
> In addition to starting the daemons in the foreground, the start up scripts 
> should use exec instead of running the commends, so as to avoid forks.  Many 
> supervisory tools assume the PID of the process to be monitored is that of 
> the process it first executes, and fork chains make it difficult for the 
> supervisor to figure out what process to monitor.  Specifically, 
> jobmanager.sh and taskmanager.sh should exec flink-daemon.sh, and 
> flink-daemon.sh should exec java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4378) Enable RollingSink to custom HDFS client configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417249#comment-15417249
 ] 

ASF GitHub Bot commented on FLINK-4378:
---

GitHub user wenlong88 opened a pull request:

https://github.com/apache/flink/pull/2356

[FLINK-4378]Enable RollingSink to custom HDFS client configuration

Adding a new interface to rolling sink


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wenlong88/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2356


commit 9b4dcff49aa8f1685ef3e38634f24b180ffcc3d5
Author: wenlong.lwl 
Date:   2016-08-11T13:26:01Z

enable rolling sink to custom hdfs client configuration




> Enable RollingSink to custom HDFS client configuration
> --
>
> Key: FLINK-4378
> URL: https://issues.apache.org/jira/browse/FLINK-4378
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Reporter: Wenlong Lyu
>Assignee: Wenlong Lyu
>
> Optimizing the configuration of hdfs client in different situation, such as 
> {{io.file.buffer.size}}  can make rolling sink perform better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417241#comment-15417241
 ] 

Greg Hogan commented on FLINK-4370:
---

I'd rather add this to its proper place ({{.idea}}) and have current developers 
inconvenienced once with the rebase on top of an untracked file than to clutter 
{{tools}} and the new contributor guidelines. A notice could be posted to the 
flink-devel mailing list explaining the one-time need to remove the existing 
files before rebasing to master.

> Offer a default IntelliJ inspection profile with Flink
> --
>
> Key: FLINK-4370
> URL: https://issues.apache.org/jira/browse/FLINK-4370
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
> should be automatically picked up when the code is checked out and imported 
> into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4378) Enable RollingSink to custom HDFS client configuration

[Wenlong Lyu (JIRA)]

Wenlong Lyu created FLINK-4378:
--

 Summary: Enable RollingSink to custom HDFS client configuration
 Key: FLINK-4378
 URL: https://issues.apache.org/jira/browse/FLINK-4378
 Project: Flink
  Issue Type: Improvement
  Components: filesystem-connector
Reporter: Wenlong Lyu
Assignee: Wenlong Lyu


Optimizing the configuration of hdfs client in different situation, such as 
{{io.file.buffer.size}}  can make rolling sink perform better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417229#comment-15417229
 ] 

ASF GitHub Bot commented on FLINK-4253:
---

Github user uce commented on the issue:

https://github.com/apache/flink/pull/2342
  
I think we need to change more than just that single variable. The other 
variables should match, e.g.
`recovery.jobmanager.port => high-availability.jobmanager.port` and also 
`recovery.zookeeper.* => recovery.zookeeper.*` etc.


> Rename "recovery.mode" config key to "high-availability"
> 
>
> Key: FLINK-4253
> URL: https://issues.apache.org/jira/browse/FLINK-4253
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ufuk Celebi
>Assignee: ramkrishna.s.vasudevan
>
> Currently, HA is configured via the following configuration keys:
> {code}
> recovery.mode: STANDALONE // No high availability (HA)
> recovery.mode: ZOOKEEPER // HA
> {code}
> This could be more straight forward by simply renaming the key to 
> {{high-availability}}. Furthermore, the term {{STANDALONE}} is overloaded. We 
> already have standalone cluster mode.
> {code}
> high-availability: NONE // No HA
> high-availability: ZOOKEEPER // HA via ZooKeeper
> {code}
> The {{recovery.mode}} configuration keys would have to be deprecated before 
> completely removing them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2342: FLINK-4253 - Rename "recovery.mode" config key to "high-a...

[uce]

Github user uce commented on the issue:

https://github.com/apache/flink/pull/2342
  
I think we need to change more than just that single variable. The other 
variables should match, e.g.
`recovery.jobmanager.port => high-availability.jobmanager.port` and also 
`recovery.zookeeper.* => recovery.zookeeper.*` etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417208#comment-15417208
 ] 

Jark Wu commented on FLINK-4370:


Yes. We can put this files into tools folder (or other) as well as  the code 
style file.  And document it in {{ide_setup.md}}

> Offer a default IntelliJ inspection profile with Flink
> --
>
> Key: FLINK-4370
> URL: https://issues.apache.org/jira/browse/FLINK-4370
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
> should be automatically picked up when the code is checked out and imported 
> into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4335) Add jar id, and job parameters information to job status rest call

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417195#comment-15417195
 ] 

Robert Metzger commented on FLINK-4335:
---

I see. The jar ID is an internal id of the web job submission client. It 
doesn't exist for the CliFrontend (./bin/flink), so its not a concept generally 
known to Flink.

The jar Id is directly derived from the jar file name (when uploading a jar in 
Flink's web interface, it'll be renamed).
As a temporary workaround, you could use this code-snippet to get the name of 
the jar and then derive the Jar id from it (you can put the id then into the 
global job parameters as well)
{code}
public class Consumer {
public static void main(String[] args) throws Exception {
String jarFileName = new 
File(Consumer.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath()).getName();
int idx = jarFileName.indexOf("_");
String jarID = jarFileName.substring(0, idx);
System.out.println("Jar ID = " + jarID);
{code}

I know that this solution is not very stable .. maybe we can come up with 
something better in the future.


> Add jar id, and job parameters information to job status rest call
> --
>
> Key: FLINK-4335
> URL: https://issues.apache.org/jira/browse/FLINK-4335
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Zhenzhong Xu
>Priority: Minor
>
> From declarative, reconcilation based job management perspective, there is a 
> need to identify the job jar id, and all job parameters for a running job to 
> determine if the current job is up to date. 
> I think these information needs to be available through the job manager rest 
> call (/jobs/$id).
> * Jar ID
> * Job entry class
> * parallelism
> * all user defined parameters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4359) Add INTERVAL type

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417187#comment-15417187
 ] 

Jark Wu commented on FLINK-4359:


Okay, I see. 
I'm interested in this, and I'm glad if I can help anything. 

> Add INTERVAL type
> -
>
> Key: FLINK-4359
> URL: https://issues.apache.org/jira/browse/FLINK-4359
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.2.0
>
>
> In order to start with StreamSQL windows we need a way to define intervals in 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4370) Offer a default IntelliJ inspection profile with Flink

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417185#comment-15417185
 ] 

Greg Hogan commented on FLINK-4370:
---

We've had pull requests rejected due to formatting changes, and even the 
smallest changes such as reordering imports muddy the review. I'd like to see 
this included if it improves consistency. I'd prefer to include configuration 
files in the repository rather than on the website where new contributors may 
not know to look.

Adding an IntelliJ code style in FLINK-3870 is still open.

> Offer a default IntelliJ inspection profile with Flink
> --
>
> Key: FLINK-4370
> URL: https://issues.apache.org/jira/browse/FLINK-4370
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> We can commit an inspection profile under {{.idea/inspectionProfiles}} which 
> should be automatically picked up when the code is checked out and imported 
> into IntelliJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4377) akka.remote.OversizedPayloadException: Discarding oversized payload

[Sajeev Ramakrishnan (JIRA)]

Sajeev Ramakrishnan created FLINK-4377:
--

 Summary: akka.remote.OversizedPayloadException: Discarding 
oversized payload
 Key: FLINK-4377
 URL: https://issues.apache.org/jira/browse/FLINK-4377
 Project: Flink
  Issue Type: Bug
  Components: DataSet API
Affects Versions: 1.1.0
 Environment: Linux
Reporter: Sajeev Ramakrishnan


Dear Team,
  I was trying to create a hash map with a size around 1 million. Then I 
encountered the below issue. For smaller maps, I am not seeing any issues.

akka.remote.OversizedPayloadException: Discarding oversized payload sent to 
Actor[akka.tcp://flink@localhost:58247/user/$d#458673459]: max allowed size 
10485760 bytes, actual size of encoded class 
org.apache.flink.runtime.messages.JobManagerMessages$JobResultSuccess was 
175254213 bytes.


Regards,
Sajeev



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417169#comment-15417169
 ] 

ASF GitHub Bot commented on FLINK-4282:
---

GitHub user Renkai opened a pull request:

https://github.com/apache/flink/pull/2355

[FLINK-4282]Add Offset Parameter to WindowAssigners

Although there is already a merge request for this issue,I think this 
implementation is more sensible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Renkai/flink FLINK-4282

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2355.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2355


commit d31cc8125b30212b6ac21996a48d703eb11354e9
Author: renkai 
Date:   2016-08-11T10:48:50Z

[FLINK-4282]Add Offset Parameter to WindowAssigners




> Add Offset Parameter to WindowAssigners
> ---
>
> Key: FLINK-4282
> URL: https://issues.apache.org/jira/browse/FLINK-4282
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>
> Currently, windows are always aligned to EPOCH, which basically means days 
> are aligned with GMT. This is somewhat problematic for people living in 
> different timezones.
> And offset parameter would allow to adapt the window assigner to the timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...

[Renkai]

GitHub user Renkai opened a pull request:

https://github.com/apache/flink/pull/2355

[FLINK-4282]Add Offset Parameter to WindowAssigners

Although there is already a merge request for this issue,I think this 
implementation is more sensible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Renkai/flink FLINK-4282

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2355.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2355


commit d31cc8125b30212b6ac21996a48d703eb11354e9
Author: renkai 
Date:   2016-08-11T10:48:50Z

[FLINK-4282]Add Offset Parameter to WindowAssigners




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-4374) GroupReduce Broken for null Date

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417162#comment-15417162
 ] 

Stefan Richter edited comment on FLINK-4374 at 8/11/16 12:52 PM:
-

Yes, it is the date field of the Tuple2 in this test.


was (Author: srichter):
Yes, it is the data field of the Tuple2 in this test.

> GroupReduce Broken for null Date
> 
>
> Key: FLINK-4374
> URL: https://issues.apache.org/jira/browse/FLINK-4374
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Reporter: Stefan Richter
>Assignee: Timo Walther
>
> The GroupReduceITCase has an error that allows a problem with {{null}} Dates 
> to go uncovered:
>  If I set the parallelism to 1 in {{testDateNullException()}} and all keys 
> actually end up on the same operator, then there is a problem in the 
> de/serialization.
> It seems that {{null}} values are somehow skipped by the serialization 
> process (e.g. maybe no {{null}} indicator is written), which leads to wrong 
> deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4374) GroupReduce Broken for null Date

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417162#comment-15417162
 ] 

Stefan Richter commented on FLINK-4374:
---

Yes, it is the data field of the Tuple2 in this test.

> GroupReduce Broken for null Date
> 
>
> Key: FLINK-4374
> URL: https://issues.apache.org/jira/browse/FLINK-4374
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API
>Reporter: Stefan Richter
>Assignee: Timo Walther
>
> The GroupReduceITCase has an error that allows a problem with {{null}} Dates 
> to go uncovered:
>  If I set the parallelism to 1 in {{testDateNullException()}} and all keys 
> actually end up on the same operator, then there is a problem in the 
> de/serialization.
> It seems that {{null}} values are somehow skipped by the serialization 
> process (e.g. maybe no {{null}} indicator is written), which leads to wrong 
> deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)