zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400638771
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##########
@@ -83,24 +86,45 @@
/** The total number of bytes (both data and event buffers). */
private long totalNumberOfBytes;
+ /** The collection of buffers which are spanned over by checkpoint
barrier and needs to be persisted for snapshot. */
+ private final List<Buffer> inflightBufferSnapshot = new ArrayList<>();
+
//
------------------------------------------------------------------------
PipelinedSubpartition(int index, ResultPartition parent) {
super(index, parent);
}
@Override
- public boolean add(BufferConsumer bufferConsumer) {
- return add(bufferConsumer, false);
+ public boolean add(BufferConsumer bufferConsumer) throws IOException {
+ return add(bufferConsumer, false, false);
}
@Override
public void finish() throws IOException {
-
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true);
+
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true,
false);
LOG.debug("{}: Finished {}.", parent.getOwningTaskName(), this);
}
- private boolean add(BufferConsumer bufferConsumer, boolean finish) {
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException {
+ if (readView != null) {
+ switch (readView.priorityEvent(event,
eventBufferConsumer)) {
+ case ENQUEUE_AS_FIRST_BUFFER:
+ add(eventBufferConsumer.copy(), false,
true);
+ break;
+ case ENQUEUE_AS_LAST_BUFFER:
+ add(eventBufferConsumer.copy(), false,
false);
+ break;
+ }
+ } else {
+ buffers.add(eventBufferConsumer.copy());
Review comment:
If the current view is null, maybe it is also worth inserting the barrier
into head, because the view might be established soon after inserting. Or are
there any considerations for making the difference based on the view status?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
