[
https://issues.apache.org/jira/browse/FLINK-12536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873088#comment-16873088
]
Congxian Qiu(klion26) commented on FLINK-12536:
-----------------------------------------------
As discussed with [~pnowojski] offline, will use a buffer pool(default 2, can
be configurable) and reuse the read thread in IOManager to read the
BufferOrEvent asynchronous.
Create an issue FLINK-12994 to track the improvement for buffer processing
performance in SpilledBufferOrEventSequence#getNext.
The benchmark result is as following:
Benchmark Mode Cnt Score Error
Units
ConsumerBench.testBuffer thrpt 30 99.528 ± 1.923
ops/ms
ConsumerBench.testBufferWithWrap thrpt 30 827.931 ± 15.148
ops/ms
ConsumerBench.testCancelCheckpointMarkerEvent thrpt 30 40144.914 ± 1244.112
ops/ms
ConsumerBench.testCheckpointBarrierEvent thrpt 30 29784.258 ± 467.440
ops/ms
ConsumerBench.testEndOfPartitionEvent thrpt 30 48562.810 ± 1599.328
ops/ms
FileIOBench.test1SyncIO thrpt 30 359.367 ± 3.203
ops/ms
FileIOBench.testAsyncIO thrpt 30 66.573 ± 3.384
ops/ms
[1] `testBuffer` test `SpilledBufferOrEventSequence#getNext` using
`MemorySegmentFactory.allocateUnpooledSegment(pageSize)`, `testBufferWithWrap`
test `SpilledBufferOrEventSequence#getNext` using
`MemorySegmentFactory.wrap()`, `test***Event` test all the event processing,
`test1SyncIO` tests the `FileChannel` and `testAsyncIO` tests the
`AsynchronousFileChannel`
[2]
[https://github.com/klion26/FileIOBench/blob/8618fc4d6e745d8dd762b87505102e6cec78dc9b/src/main/java/com/klion26/data/ConsumerBench.java#L130]
[3]
https://github.com/klion26/FileIOBench/blob/8618fc4d6e745d8dd762b87505102e6cec78dc9b/src/main/java/com/klion26/data/ConsumerBench.java#L169
> Make BufferOrEventSequence#getNext() non-blocking
> -------------------------------------------------
>
> Key: FLINK-12536
> URL: https://issues.apache.org/jira/browse/FLINK-12536
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Piotr Nowojski
> Assignee: Congxian Qiu(klion26)
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently it is non-blocking in case of credit-based flow control (default),
> however for \{{SpilledBufferOrEventSequence}} it is blocking on reading from
> file. We might want to consider reimplementing it to be non blocking with
> {{CompletableFuture<?> isAvailable()}} method.
>
> Otherwise we will block mailbox processing for the duration of reading from
> file - for example we will block processing time timers and potentially in
> the future network flushes.
>
> This is not a high priority change, since it affects non-default
> configuration option AND at the moment only processing time timers are
> planned to be moved to the mailbox for 1.9.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
