ASF GitHub Bot commented on FLINK-8759:

GitHub user NicoK opened a pull request:


    [FLINK-8759][network] preparations for the update of netty to version 4.0.56

    ## What is the purpose of the change
    Based on the changes from #5570, this PR prepares a netty version bump by 
circumventing a bug we had with the change from Netty 4.0.27 to 4.0.28 with the 
old non-credit based flow control paths:
    Versions >= 4.0.28 contain an improvement by Netty, which slices a Netty 
buffer instead of doing a memory copy 
(https://github.com/netty/netty/issues/3704) in the 
`LengthFieldBasedFrameDecoder`. In some situations, this interacts badly with 
our Netty pipeline leading to `OutOfMemory` errors.
    With credit-based flow control this problem should not exist anymore which 
is why we can use the original netty code there.
    ## Brief change log
    - override `LengthFieldBasedFrameDecoder#extractFrame()` and implement two 
different code paths depending on whether credit based flow control is on or not
    ## Verifying this change
    This change is a trivial rework / code cleanup without any test coverage.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): **no**
      - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: **no**
      - The serializers: **no**
      - The runtime per-record code paths (performance sensitive): **no** (only 
per buffer)
      - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
      - The S3 file system connector: **no**
    ## Documentation
      - Does this pull request introduce a new feature? **no**
      - If yes, how is the feature documented? **JavaDocs**

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-8759

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5571
commit 272899e01c2836a2a5b47958db7d9d9f6cbf471d
Author: Nico Kruber <nico@...>
Date:   2018-02-23T12:56:29Z

    [FLINK-8768][network] Let NettyMessageDecoder inherit from 
    This replaces one additional step from the pipeline and does not only remove
    overhead there but also allows use to override the #extractFrame() method to
    restore the old Netty 4.0.27 behaviour for non-credit based code paths which
    had a bug with Netty >= 4.0.28 there (see FLINK-8759).

commit 11f673b415bce310cc2195fb6778051c095083fa
Author: Nico Kruber <nico@...>
Date:   2018-02-23T13:06:00Z

    [FLINK-8759][network] preparations for the update of netty to version 4.0.56


> Bump Netty to 4.0.56
> --------------------
>                 Key: FLINK-8759
>                 URL: https://issues.apache.org/jira/browse/FLINK-8759
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>            Priority: Major
>             Fix For: 1.5.0
> For a bug in Netty's shutdown sequence and overall improvements in Netty, I'd 
> like to bump the version (and stay within the 4.0 series for now). The 
> problem we faced in the past should not be relevant for credit-based flow 
> control anymore and can be worked around (for the fallback code path) by 
> restoring {{LengthFieldBasedFrameDecoder}}'s old behaviour of copying 
> contents to new buffers instead of slicing the existing one (please refer to 
> FLINK-7428 for the inverse direction).

This message was sent by Atlassian JIRA

Reply via email to