[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510804#comment-16510804
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6071
  
Thanks!


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
> Fix For: 1.6.0
>
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510741#comment-16510741
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6071


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
> Fix For: 1.6.0
>
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510737#comment-16510737
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/6071
  
Thanks for your work and addressing my comments @pnowojski. Travis gave 
green light. Merging this PR now.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509714#comment-16509714
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194769995
  
--- Diff: 
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/ChunkedByteBuf.java
 ---
@@ -88,6 +102,16 @@ public ByteBuf readChunk(ChannelHandlerContext ctx) 
throws Exception {
}
}
 
+   @Override
+   public long length() {
+   return length;
--- End diff --

changed to `-1`


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509715#comment-16509715
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194770093
  
--- Diff: 
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/ChunkedByteBuf.java
 ---
@@ -88,6 +102,16 @@ public ByteBuf readChunk(ChannelHandlerContext ctx) 
throws Exception {
}
}
 
+   @Override
+   public long length() {
+   return length;
+   }
+
+   @Override
+   public long progress() {
+   return length - buf.readableBytes();
--- End diff --

changed to `return buf.readerIndex()` since progress is not well defined.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509698#comment-16509698
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194763826
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/ReadOnlySlicedNetworkBuffer.java
 ---
@@ -75,12 +75,12 @@ private ReadOnlySlicedNetworkBuffer(ByteBuf buffer, int 
index, int length, int m
 
@Override
public ByteBuf unwrap() {
-   return super.unwrap().unwrap();
+   return super.unwrap();
--- End diff --

as discussed offline, this is because of change in the implementation of 
`SlicedByteBuf`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509576#comment-16509576
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194718801
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
--- End diff --

The same question would remain: whether such artificial setup, setups a 
network stack in the same way as Flink.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509548#comment-16509548
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194713008
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/router/RoutedRequest.java
 ---
@@ -97,11 +98,17 @@ public ReferenceCounted retain(int arg0) {
 
@Override
public ReferenceCounted touch() {
+   if (requestAsReferenceCounted.isPresent()) {
+   
ReferenceCountUtil.touch(requestAsReferenceCounted.get());
+   }
return this;
}
 
@Override
public ReferenceCounted touch(Object o) {
+   if (requestAsReferenceCounted.isPresent()) {
+   
ReferenceCountUtil.touch(requestAsReferenceCounted.get());
--- End diff --

I think we could do that. I'm not entirely sure how this whole `touch` is 
suppose to work, but passing down the hint shouldn't harm anything and be more 
correct.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509544#comment-16509544
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194711763
  
--- Diff: 
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/ChunkedByteBuf.java
 ---
@@ -88,6 +102,16 @@ public ByteBuf readChunk(ChannelHandlerContext ctx) 
throws Exception {
}
}
 
+   @Override
+   public long length() {
+   return length;
--- End diff --

It seems like content of `buf` is not changing. However we can not guard 
against it programmatically, only via java doc. On the other hand, I'm not sure 
what would the side effects of "unknown" length be.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509400#comment-16509400
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/6071
  
Changes look good to me @pnowojski. I had some minor comments. After 
addressing them, we should rebase and remove the unnecessary commits. Ping me 
once you have done it. I will then merge this PR.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509395#comment-16509395
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194671209
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
--- End diff --

Is it strictly necessary that this test must be an integration test case? 
As far as I understand the test case, wouldn't it be enough to setup a 
`NettyServer` with `epoll` activated and sending some data to it via a 
`NettyClient` with `epoll` activated? Since we don't assert anything else than 
whether the Flink program executes or not, it should be basically the same.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using 

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509383#comment-16509383
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r19412
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/AbstractByteBufTest.java
 ---
@@ -64,12 +69,13 @@
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeFalse;
 import static org.junit.Assume.assumeTrue;
 
 /**
  * An abstract test class for channel buffers.
  *
- * Copied from netty 4.0.50 with some changes to fit our netty version 
4.0.27.
+ * Copy from netty 4.1.24.Final.
  */
 public abstract class AbstractByteBufTest {
--- End diff --

Let's extend this class from `TestLogger`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509384#comment-16509384
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194663296
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/router/RoutedRequest.java
 ---
@@ -97,11 +98,17 @@ public ReferenceCounted retain(int arg0) {
 
@Override
public ReferenceCounted touch() {
+   if (requestAsReferenceCounted.isPresent()) {
+   
ReferenceCountUtil.touch(requestAsReferenceCounted.get());
+   }
return this;
}
 
@Override
public ReferenceCounted touch(Object o) {
+   if (requestAsReferenceCounted.isPresent()) {
+   
ReferenceCountUtil.touch(requestAsReferenceCounted.get());
--- End diff --

Shall we pass `o` to the `ReferenceCountUtil.touch` call?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509368#comment-16509368
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194660815
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/ReadOnlySlicedNetworkBuffer.java
 ---
@@ -75,12 +75,12 @@ private ReadOnlySlicedNetworkBuffer(ByteBuf buffer, int 
index, int length, int m
 
@Override
public ByteBuf unwrap() {
-   return super.unwrap().unwrap();
+   return super.unwrap();
--- End diff --

But isn't it a bit counter-intuitive that you instantiate a 
`ReadOnlySlicedNetworkBuffer` with a `NetworkBuffer` and when you call 
`ReadOnlySlicedNetworkBuffer.unwrap` you don't get a `NetworkBuffer` back but a 
`SlicedByteBuf`? Thus, you need to know the internals of this class to know 
that you have to call twice `unwrap` to obtain the `NetworkBuffer`. This shows 
for example in the `NetworkBufferTest` where we always call 
`slice.unwrap().unwrap()`.

What exactly did change with the structure of nested classes that this 
change is necessary?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509367#comment-16509367
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194652733
  
--- Diff: 
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/ChunkedByteBuf.java
 ---
@@ -88,6 +102,16 @@ public ByteBuf readChunk(ChannelHandlerContext ctx) 
throws Exception {
}
}
 
+   @Override
+   public long length() {
+   return length;
+   }
+
+   @Override
+   public long progress() {
+   return length - buf.readableBytes();
--- End diff --

If the length is not know, then I think it would be better to return 
something like `buf.readerIndex - initialReaderIndex`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509366#comment-16509366
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r194652495
  
--- Diff: 
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/ChunkedByteBuf.java
 ---
@@ -88,6 +102,16 @@ public ByteBuf readChunk(ChannelHandlerContext ctx) 
throws Exception {
}
}
 
+   @Override
+   public long length() {
+   return length;
--- End diff --

Is it always guaranteed that `buf.readableBytes` won't change? If not, then 
I think we should return `-1` here, because the length is unknown.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507959#comment-16507959
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6071
  
@zentol as @NicoK suggested I have missed netty 4.1 upgrade without credit 
base flow control. It turns out that this upgrade is unstable without this fix: 
https://github.com/apache/flink/pull/5571, so we should merge it before this 
one.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503037#comment-16503037
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r193345578
  
--- Diff: pom.xml ---
@@ -300,15 +300,7 @@ under the License.

org.apache.flink
flink-shaded-netty
-   
-   
4.0.27.Final-${flink.shaded.version}
+   
4.1.24.Final-${flink.shaded.version}
--- End diff --

I've opened #6128 to bump the remaining version so this PR can stay focused 
on netty.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494938#comment-16494938
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191685431
  
--- Diff: pom.xml ---
@@ -300,15 +300,7 @@ under the License.

org.apache.flink
flink-shaded-netty
-   
-   
4.0.27.Final-${flink.shaded.version}
+   
4.1.24.Final-${flink.shaded.version}
--- End diff --

I hope that bumping to from `2.0` to `4.0` will not break other things, but 
yes, that was my intention when `4.0` will be on mvn central.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494940#comment-16494940
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191685731
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBuffer.java
 ---
@@ -215,16 +242,33 @@ protected void _setMedium(int index, int value) {
setByte(index + 2, (byte) value);
}
 
+   @Override
+   protected void _setMediumLE(int index, int value){
+   setByte(index, (byte) value);
--- End diff --

added comment


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494937#comment-16494937
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191700141
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBuffer.java
 ---
@@ -424,6 +480,19 @@ public int setBytes(int index, ScatteringByteChannel 
in, int length) throws IOEx
}
}
 
+   @Override
+   public int setBytes(int index, FileChannel in, long position, int 
length) throws IOException {
+   // adapted from UnpooledDirectByteBuf:
+   checkIndex(index, length);
+
+   ByteBuffer tmpBuf = memorySegment.wrap(index, length);
+   try {
+   return in.read(tmpBuf);
--- End diff --

ops, good catch. Even better catch is that this was not covered by any 
test, because I assumed `AbstractByteBufTest` comes from netty, while in 
reality it was copied to our code.

Fixed this lack of `position` bug and upgraded the `AbstractByteBufTest` as 
well - newer version was correctly failing with this bug.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494941#comment-16494941
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191700895
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/ReadOnlySlicedNetworkBuffer.java
 ---
@@ -75,12 +75,12 @@ private ReadOnlySlicedNetworkBuffer(ByteBuf buffer, int 
index, int length, int m
 
@Override
public ByteBuf unwrap() {
-   return super.unwrap().unwrap();
+   return super.unwrap();
--- End diff --

Necessary for the upgrade, otherwise our tests (including `ITCases`) are 
failing. In upgraded version there is a different structure of nested classes 
in case of using slices of readonly buffers. Possibly this is a bug fix for a 
bug, that didn't show up in older netty.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494939#comment-16494939
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191684561
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
--- End diff --

Hmmm, depends what you would like to test and depends on black/white box 
approach. With white box where you assume/know that `Netty` is loading `native` 
libraries only during setup and if you assume that if they are loaded, that 
they will work correctly, then no, execution is unnecessary.  But I don't want 
to relay on both of those assumptions.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor 

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494881#comment-16494881
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191683459
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
+   }
+   finally {
+   cluster.get().after();
+   }
+   }
+
+   private Optional trySetUp() throws Exception {
+   try {
+   MiniClusterResource cluster = new MiniClusterResource(
+   new MiniClusterResourceConfiguration(
+   getConfiguration(),
+   TASK_MANAGERES,
+   1),
+   true);
+   cluster.before();
+   return Optional.of(cluster);
+   }
+   catch (UnsatisfiedLinkError ex) {
+   // If we failed to init netty because we are not on 
Linux platform, abort the test.
+   if (findThrowableWithMessage(ex, "Only supported on 
Linux").isPresent()) {
+   return Optional.empty();
--- End diff --

I didn't know about `AssumptionViolatedException`. Thanks for pointing this 
out.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> 

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493389#comment-16493389
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191386855
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBuffer.java
 ---
@@ -424,6 +480,19 @@ public int setBytes(int index, ScatteringByteChannel 
in, int length) throws IOEx
}
}
 
+   @Override
+   public int setBytes(int index, FileChannel in, long position, int 
length) throws IOException {
+   // adapted from UnpooledDirectByteBuf:
+   checkIndex(index, length);
+
+   ByteBuffer tmpBuf = memorySegment.wrap(index, length);
+   try {
+   return in.read(tmpBuf);
--- End diff --

are you ignoring the `position` argument intentionally?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493381#comment-16493381
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191379552
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
--- End diff --

typo: can only run


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493384#comment-16493384
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191380808
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
--- End diff --

The job execution isn't necessary is it?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493388#comment-16493388
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191385215
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/ReadOnlySlicedNetworkBuffer.java
 ---
@@ -75,12 +75,12 @@ private ReadOnlySlicedNetworkBuffer(ByteBuf buffer, int 
index, int length, int m
 
@Override
public ByteBuf unwrap() {
-   return super.unwrap().unwrap();
+   return super.unwrap();
--- End diff --

Is this change necessary for the upgrade or cleanup?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493382#comment-16493382
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191379687
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
--- End diff --

`NUM_TASK_MANAGERS`, also a type


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493390#comment-16493390
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191386142
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBuffer.java
 ---
@@ -215,16 +242,33 @@ protected void _setMedium(int index, int value) {
setByte(index + 2, (byte) value);
}
 
+   @Override
+   protected void _setMediumLE(int index, int value){
+   setByte(index, (byte) value);
--- End diff --

was this also taken from `UnpooledDirectByteBuf`?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493386#comment-16493386
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191380426
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
+   }
+   finally {
+   cluster.get().after();
+   }
+   }
+
+   private Optional trySetUp() throws Exception {
+   try {
+   MiniClusterResource cluster = new MiniClusterResource(
+   new MiniClusterResourceConfiguration(
+   getConfiguration(),
+   TASK_MANAGERES,
+   1),
+   true);
+   cluster.before();
+   return Optional.of(cluster);
+   }
+   catch (UnsatisfiedLinkError ex) {
+   // If we failed to init netty because we are not on 
Linux platform, abort the test.
+   if (findThrowableWithMessage(ex, "Only supported on 
Linux").isPresent()) {
+   return Optional.empty();
+   }
+   throw ex;
+   }
+   }
+
+   private static Configuration getConfiguration() {
--- End diff --

I would in-line this method.


> Bump Netty to 4.1
> -
>
> Key: 

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493383#comment-16493383
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191379939
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/runtime/NettyEpollITCase.java 
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.runtime;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterResource;
+import 
org.apache.flink.test.util.MiniClusterResource.MiniClusterResourceConfiguration;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+
+import static 
org.apache.flink.runtime.io.network.netty.NettyConfig.TRANSPORT_TYPE;
+import static 
org.apache.flink.util.ExceptionUtils.findThrowableWithMessage;
+
+/**
+ * Test network stack with taskmanager.network.netty.transport set to 
epoll. This test car only run
+ * on linux. On other platforms it's basically a NO-OP. See
+ * https://github.com/apache/flink-shaded/issues/30
+ */
+public class NettyEpollITCase extends TestLogger {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(NettyEpollITCase.class);
+
+   private static final int TASK_MANAGERES = 2;
+
+   @Test
+   public void testNettyEpoll() throws Exception {
+   Optional cluster = trySetUp();
+   if (!cluster.isPresent()) {
+   return;
+   }
+
+   try {
+   StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+   env.setParallelism(TASK_MANAGERES);
+   env.getConfig().disableSysoutLogging();
+
+   DataStream input = env.fromElements(1, 2, 3, 
4, 1, 2, 3, 42);
+   input.keyBy(new KeySelector() {
+   @Override
+   public Integer getKey(Integer value) 
throws Exception {
+   return value;
+   }
+   })
+   .sum(0)
+   .print();
+
+   env.execute();
+   }
+   finally {
+   cluster.get().after();
+   }
+   }
+
+   private Optional trySetUp() throws Exception {
+   try {
+   MiniClusterResource cluster = new MiniClusterResource(
+   new MiniClusterResourceConfiguration(
+   getConfiguration(),
+   TASK_MANAGERES,
+   1),
+   true);
+   cluster.before();
+   return Optional.of(cluster);
+   }
+   catch (UnsatisfiedLinkError ex) {
+   // If we failed to init netty because we are not on 
Linux platform, abort the test.
+   if (findThrowableWithMessage(ex, "Only supported on 
Linux").isPresent()) {
+   return Optional.empty();
--- End diff --

couldn't you fail here with an ´AssumptionViolatedException`? Then we 
wouldn't have to deal with optionals.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: 

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493385#comment-16493385
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191381531
  
--- Diff: pom.xml ---
@@ -300,15 +300,7 @@ under the License.

org.apache.flink
flink-shaded-netty
-   
-   
4.0.27.Final-${flink.shaded.version}
+   
4.1.24.Final-${flink.shaded.version}
--- End diff --

you (will) have to bump `flink.shaded.version` to 4.0. Doing this will not 
negatively affect other dependencies.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493387#comment-16493387
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191384185
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/router/RoutedRequest.java
 ---
@@ -94,4 +94,14 @@ public ReferenceCounted retain(int arg0) {
}
return this;
}
+
+   @Override
+   public ReferenceCounted touch() {
+   return this;
+   }
+
+   @Override
+   public ReferenceCounted touch(Object o) {
+   return this;
--- End diff --

We may want to `touch` the contained request as well like in 
[this](https://github.com/netty/netty/blob/4.1/codec-redis/src/main/java/io/netty/handler/codec/redis/ArrayRedisMessage.java)
 class, depending on `requestAsReferenceCounted`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493251#comment-16493251
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6071
  
@zentol I have also fixed added a test to cover 
https://github.com/apache/flink-shaded/issues/30 in the second commit


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492422#comment-16492422
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r191150025
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -247,7 +249,15 @@ public void shutdown(Time timeout) {
 
@Override
protected void channelRead0(ChannelHandlerContext ctx, Object 
msg) {
-   if (msg instanceof FullHttpResponse) {
+   // TODO: should this check for status OK (200) and 
treat all other as errors?
--- End diff --

yes I think we can just drop it.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490681#comment-16490681
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6071
  
Dead lock visible in failed travis is unrelated to this change 
(https://issues.apache.org/jira/browse/FLINK-9439)


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490675#comment-16490675
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r190884312
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -247,7 +249,15 @@ public void shutdown(Time timeout) {
 
@Override
protected void channelRead0(ChannelHandlerContext ctx, Object 
msg) {
-   if (msg instanceof FullHttpResponse) {
+   // TODO: should this check for status OK (200) and 
treat all other as errors?
--- End diff --

Thx for filling this out. In that case should I just drop this `// TODO:`?


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489646#comment-16489646
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r190705206
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -247,7 +249,15 @@ public void shutdown(Time timeout) {
 
@Override
protected void channelRead0(ChannelHandlerContext ctx, Object 
msg) {
-   if (msg instanceof FullHttpResponse) {
+   // TODO: should this check for status OK (200) and 
treat all other as errors?
--- End diff --

message semantics are handled later when parsing the payload, which 
effectively does what you're suggesting. The else branch for debugging; our 
rest servers always return `FullHttpResponses`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489387#comment-16489387
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

zentol closed pull request #41: [FLINK-3952] Bump Netty to 4.1.24 and drop 
netty-router
URL: https://github.com/apache/flink-shaded/pull/41
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/flink-shaded-netty-4/pom.xml b/flink-shaded-netty-4/pom.xml
index 3ac5c25..0aa734b 100644
--- a/flink-shaded-netty-4/pom.xml
+++ b/flink-shaded-netty-4/pom.xml
@@ -34,7 +34,7 @@ under the License.
 ${netty.version}-4.0
 
 
-4.0.56.Final
+4.1.24.Final
 
 
 
@@ -43,12 +43,6 @@ under the License.
 netty-all
 ${netty.version}
 
-
-
-tv.cntt
-netty-router
-1.10
-
 
 
 
@@ -101,4 +95,4 @@ under the License.
 
 
 
-
\ No newline at end of file
+


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488833#comment-16488833
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r190554721
  
--- Diff: pom.xml ---
@@ -308,7 +308,7 @@ under the License.
errors.
 
[1] https://github.com/netty/netty/issues/3704 
-->
--- End diff --

Ops, dropped.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488819#comment-16488819
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

GitHub user pnowojski opened a pull request:

https://github.com/apache/flink/pull/6071

[FLINK-3952][runtine] Upgrade to Netty 4.1

This PR adjusts our code to work with Netty 4.1. It also includes possible 
bug fix to file uploading cleanup in FileUploadHandler and HttpRequestHandler. 
For mor information look here:

https://github.com/netty/netty/issues/7611

First commit is only for having green travis and will be dropped once new 
`flink-shadded-netty` will be released.

## Verifying this change

This change is covered by variety of pre existing tests. Furthermore I have 
manually verified that issue mentioned by @uce in the commit message here: 
https://github.com/apache/flink/commit/d92e422ec7089376583a8f57043274d236c340a4
doesn't happen: 
- I have reproduced this issue on a test cluster with Flink 1.0-XXX
- I have verified that the same job passes without any problems after 
upgrading to Netty 4.1

I have also run our network benchmark suite and verified that there are no 
performance changes after this change.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (**yes** / no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
  - The serializers: (yes / **no** / don't know)
  - The runtime per-record code paths (performance sensitive): (**yes** / 
no / don't know)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
  - The S3 file system connector: (yes / **no** / don't know)

## Documentation

  - Does this pull request introduce a new feature? (yes / **no**)
  - If yes, how is the feature documented? (**not applicable** / docs / 
JavaDocs / not documented)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pnowojski/flink f3952

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6071.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6071


commit 7b3be7c9ebac7392136ed85bca4664559710552e
Author: Piotr Nowojski 
Date:   2018-05-14T11:46:08Z

[hotfix][tests] Report failure with error level instead of debug

commit afaf1d5181c7133a040bf3881723e240145a4b0a
Author: Piotr Nowojski 
Date:   2018-05-16T19:26:36Z

[FLINK-9386] Embed netty router

This commit replaces netty-router dependency with our own version of it, 
which is
simplified and adds guarantees about order of matching router patterns.

This is a prerequisite for FLINK-3952. netty-router 1.10 is incompatible 
with
Netty 4.1, while netty-router 2.2.0 brakes a compatibility in a way that we
were unable to use it.

commit 26bc92db0863bf53e60164ab5f6b92ac3b424506
Author: Piotr Nowojski 
Date:   2018-05-14T10:30:31Z

Embed flink-shaded-netty-4

commit 94a4cc2237b5dac0c004ec192eb4d7f1b782e5f2
Author: Piotr Nowojski 
Date:   2018-05-16T19:27:22Z

[FLINK-3952][runtine] Upgrade to Netty 4.1

This commit includes possible bug fix to file uploading cleanup in 
FileUploadHandler and
HttpRequestHandler. For mor information look here:

https://github.com/netty/netty/issues/7611




> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488731#comment-16488731
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

pnowojski opened a new pull request #41: [FLINK-3952] Bump Netty to 4.1.24 and 
drop netty-router
URL: https://github.com/apache/flink-shaded/pull/41
 
 
   This is a part of 
[FLINK-3952](https://issues.apache.org/jira/browse/FLINK-3952)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Piotr Nowojski (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474479#comment-16474479
 ] 

Piotr Nowojski commented on FLINK-3952:
---

Yes, I'm in the middle of trying to upgrade netty-router.

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474145#comment-16474145
 ] 

Chesnay Schepler commented on FLINK-3952:
-

netty-router 2.2.0 has a netty 4.1

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Alexey Diomin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474137#comment-16474137
 ] 

Alexey Diomin commented on FLINK-3952:
--

[~pnowojski] main problem with migration was netty-router, he still use old 
version of netty.

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018878#comment-16018878
 ] 

Stephan Ewen commented on FLINK-3952:
-

True, only Hadoop's netty is shaded.

We should be able to shade Flink's Netty as well. Upgrading it to 4.1 seems 
harder.

It is always an issue if libraries bread their backwards compatibility within a 
major version :-(

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Alexey Diomin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018605#comment-16018605
 ] 

Alexey Diomin commented on FLINK-3952:
--

No, you mistake.
try 
{code}
jar tvf flink-dist_2.10-1.2.1.jar  | grep netty
{code}

you found then shaded only old version of netty 3.х, 
but addition we distributed without shading 3.х and current used version of 
netty 4.0.x 
{code}
org/apache/flink/hadoop/shaded/org/jboss/netty/
org/jboss/netty/
io/netty/
{code}

as result if you try use Apache Beam with Flink you have error like 
https://gist.github.com/xhumanoid/291d7bfc50f830857971c15c34083351

reason it's mix netty 4.1 from beam and netty 4.0 from flink
current hotfix for me it's exclude netty from my result jar for app with beam, 
but it's potential problem because beam use grpc which require netty 4.1

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977390#comment-15977390
 ] 

Stephan Ewen commented on FLINK-3952:
-

Just to avoid confusion: You can use a newer Netty in the user code (for 
example if you need HTTP/2) without bumping Flink's Netty dependency. Flink 
shades its netty dependency away...

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[rektide de la fey (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306897#comment-15306897
 ] 

rektide de la fey commented on FLINK-3952:
--

I tried using 4.1.0.Final (which was just released). flink-runtime gave me one 
error, I haven't investigate further:

{{[ERROR] 
/usr/local/src/flink-git/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyBufferPool.java:[38,8]
 org.apache.flink.runtime.io.network.netty.NettyBufferPool is not abstract and 
does not override abstract method calculateNewCapacity(int,int) in 
io.netty.buffer.ByteBufAllocator}}

https://github.com/apache/flink/blob/6d83c9d959fc43e378259126dfd0190ff8ce6030/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyBufferPool.java#L38
 links to that line.

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Hadoop Compatibility
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299968#comment-15299968
 ] 

Greg Hogan commented on FLINK-3952:
---

Hi [~rektide], are there changes to Flink to make use of the new codecs?

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Hadoop Compatibility
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296460#comment-15296460
 ] 

Stephan Ewen commented on FLINK-3952:
-

Good idea in general. Last time we bumped Netty, however, we had a bad 
surprise, because the memory allocation pattern of the pools changed, and 
people were getting OOM exceptions.

We need to carefully validate the change.

> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Hadoop Compatibility
>Reporter: rektide de la fey
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)