pnowojski commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r405596704
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##########
@@ -83,24 +86,45 @@
/** The total number of bytes (both data and event buffers). */
private long totalNumberOfBytes;
+ /** The collection of buffers which are spanned over by checkpoint
barrier and needs to be persisted for snapshot. */
+ private final List<Buffer> inflightBufferSnapshot = new ArrayList<>();
+
//
------------------------------------------------------------------------
PipelinedSubpartition(int index, ResultPartition parent) {
super(index, parent);
}
@Override
- public boolean add(BufferConsumer bufferConsumer) {
- return add(bufferConsumer, false);
+ public boolean add(BufferConsumer bufferConsumer) throws IOException {
+ return add(bufferConsumer, false, false);
}
@Override
public void finish() throws IOException {
-
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true);
+
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true,
false);
LOG.debug("{}: Finished {}.", parent.getOwningTaskName(), this);
}
- private boolean add(BufferConsumer bufferConsumer, boolean finish) {
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException {
+ if (readView != null) {
+ switch (readView.priorityEvent(event,
eventBufferConsumer)) {
Review comment:
One more thing. @AHeise shouldn't we merge `BufferAvailabilityListener` and
`PriorityEventListener` interfaces? They are doing basically the same thing.
One is notifying about "hot" data, another about normal data and they should be
used together always (especially after dropping `PriorityEventListener`
inheritance from `ResultSubpartition` and `PipelinedSubpartitionView`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
