[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5570


---

[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...

[pnowojski]

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/5570#discussion_r170911325
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
 ---
@@ -188,58 +186,83 @@ public void write(ChannelHandlerContext ctx, Object 
msg, ChannelPromise promise)
ctx.write(msg, promise);
}
}
-
-   // Create the frame length decoder here as it depends on the 
encoder
-   //
-   // 
+--+--++++
-   // | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM 
MESSAGE |
-   // 
+--+--++++
-   static LengthFieldBasedFrameDecoder createFrameLengthDecoder() {
-   return new 
LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, -4, 4);
-   }
}
 
-   @ChannelHandler.Sharable
-   static class NettyMessageDecoder extends 
MessageToMessageDecoder {
+   /**
+* Message decoder based on netty's {@link 
LengthFieldBasedFrameDecoder} but avoiding the
+* additional memory copy inside {@link 
#extractFrame(ChannelHandlerContext, ByteBuf, int, int)}
+* since we completely decode the {@link ByteBuf} inside {@link 
#decode(ChannelHandlerContext,
+* ByteBuf)} and will not re-use it afterwards.
+*
+* The frame-length encoder will be based on this transmission 
scheme created by {@link NettyMessage#allocateBuffer(ByteBufAllocator, byte, 
int)}:
+* 
+* +--+--++++
+* | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM MESSAGE |
+* +--+--++++
+* 
+*/
+   static class NettyMessageDecoder extends LengthFieldBasedFrameDecoder {
+   private final boolean restoreOldNettyBehaviour;
--- End diff --

drop the unused field?


---

[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...

[pnowojski]

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/5570#discussion_r170911555
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageSerializationTest.java
 ---
@@ -44,10 +44,11 @@
  */
 public class NettyMessageSerializationTest {
 
+   public static final boolean RESTORE_OLD_NETTY_BEHAVIOUR = false;
--- End diff --

dead code ;)


---

[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...

[NicoK]

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/5570

[FLINK-8768][network] Let NettyMessageDecoder inherit from 
LengthFieldBasedFrameDecoder

## What is the purpose of the change

Instead of being two steps in the channel pipeline, `NettyMessageDecoder` 
could derive from `LengthFieldBasedFrameDecoder` to reduce overhead and give us 
more control over the protocol.

As a first step, we will use this to override the `#extractFrame()` method 
to restore the old Netty 4.0.27 behaviour for non-credit based code paths which 
had a bug with Netty >= 4.0.28 (see FLINK-8759).

## Brief change log

- make `NettyMessageDecoder` inherit from `LengthFieldBasedFrameDecoder` 
(beware that this changes the decoder from a `MessageToMessageDecoder` to a 
`ByteToMessageDecoder` with different cleanup invariants!)

## Verifying this change

This change is already covered by existing tests, such as 
`NettyMessageSerializationTest` or other network tests using the 
encoding/decoding pipeline.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): **no**
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: **no**
  - The serializers: **no**
  - The runtime per-record code paths (performance sensitive): **no** (only 
per buffer)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
  - The S3 file system connector: **no**

## Documentation

  - Does this pull request introduce a new feature? **no**
  - If yes, how is the feature documented? (JavaDocs)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-8768

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5570.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5570


commit 272899e01c2836a2a5b47958db7d9d9f6cbf471d
Author: Nico Kruber 
Date:   2018-02-23T12:56:29Z

[FLINK-8768][network] Let NettyMessageDecoder inherit from 
LengthFieldBasedFrameDecoder

This replaces one additional step from the pipeline and does not only remove
overhead there but also allows use to override the #extractFrame() method to
restore the old Netty 4.0.27 behaviour for non-credit based code paths which
had a bug with Netty >= 4.0.28 there (see FLINK-8759).




---