GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5288
[FLINK-8395][network] add a read-only sliced ByteBuf implementation based
on NetworkBuffer
## What is the purpose of the change
To insert the same buffer (or sub-regions from it) multiple times into
Netty's queues, we also need a (read-only) sliced variant of our
`NetworkBuffer` that implements our `Buffer` interface. This PR adds that
implementation and is based on #4613 (included here) and required by #4615.
Please note that this has been reviewed in dataArtisans/flink#5
## Brief change log
- add `ReadOnlySlicedNetworkBuffer`
- adapt `NetworkBuffer` to be able to generate these sliced buffers
- add `ReadOnlySlicedBufferTest` that performs tests for the `Buffer`
interface's wrapper methods
## Verifying this change
This change added tests and can be verified as follows:
- added `ReadOnlySlicedBufferTest` (see above)
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no** (per
buffer only)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **JavaDocs**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8395
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5288
----
commit b636592f237afb745e444bed5d0520c5736015b4
Author: Nico Kruber <nico@...>
Date: 2017-08-31T10:46:01Z
[hotfix][tests] replace DiscardingRecycler with FreeingBufferRecycler
commit 0761351db2c975ea02718f725e667b52dd9778a4
Author: Nico Kruber <nico@...>
Date: 2017-12-12T14:05:14Z
[hotfix][tests] do not use a mocked BufferRecycler for unpooled memory
segments
The mock will actually keep references to the segments instead of freeing
them.
commit 63832c24d22e672239389550e377661d1715a27c
Author: Nico Kruber <nico@...>
Date: 2018-01-03T14:57:47Z
[hotfix][tests] make SpillableSubpartitionTest use
TestBufferFactory.createBuffer
(this simplifies the test setups)
commit 070a314648b319839c2049d5c557b7608f5ce8ae
Author: Nico Kruber <nico@...>
Date: 2018-01-08T10:48:25Z
[hotfix][tests] replace InputChannelTestUtils#createMockBuffer() with
TestBufferFactory#createBuffer()
This eliminates one more unnecessary buffer mock.
commit 09d73527f90f48838631edb72d806a5bfd7d8b69
Author: Nico Kruber <nico@...>
Date: 2018-01-09T16:08:50Z
[FLINK-7520][network] let our Buffer class extend from netty's buffer class
For this, use a common (flink) Buffer interface and an implementation
(NetworkBuffer) that implements netty's buffer methods as well. In the
future,
with this, we are able to avoid unnecessary buffer copies when handing
buffers
over to netty while keeping our MemorySegment logic and configuration.
For the netty-specific part, the NetworkBuffer also requires a ByteBuf
allocator
which is otherwise not needed in our use cases, so if the buffer is handed
over
to netty, it requires a byte buffer allocator to be set.
commit 4fbef3c0c64750f6c38bafaa850d670392b0ecc8
Author: Nico Kruber <nico@...>
Date: 2018-01-08T16:02:03Z
[hotfix][io] remove duplicate code between SynchronousBufferFileReader and
BufferReadRequest
commit 7a1b4031202861fbdb3d8ae86f3810c36761b1e3
Author: Nico Kruber <nico@...>
Date: 2018-01-11T11:03:45Z
[hotfix][network] clarify BufferResponse#size() uses (by removing it)
This field was only used by the code paths on the receiver and was
inconsistent
with what was added on the sending side. We should use the contained
buffer's
readableBytes() instead, depending on the actual use case.
commit 2dad095f38e7ccd31cb5e12145848909b79d02d0
Author: Nico Kruber <nico@...>
Date: 2018-01-09T21:56:38Z
[hotfix][network] rename Buffer#retain() and #recycle in preparation for
FLINK-8396 and FLINK-8395
Since these two methods also exist in Netty's ByteBuf, we would otherwise
get
into overloading conflicts.
Also add Buffer#readableBytes() and Buffer#setAllocator().
commit e2fa9ccfed01bfdd756918116ccff20ed1833537
Author: Nico Kruber <nico@...>
Date: 2018-01-09T21:57:18Z
[FLINK-8395][network] add a read-only sliced ByteBuf implementation based
on NetworkBuffer
----
---
