pnowojski commented on a change in pull request #16655:
URL: https://github.com/apache/flink/pull/16655#discussion_r683269734
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV3Serializer.java
##########
@@ -114,8 +115,14 @@ protected void serializeOperatorState(OperatorState
operatorState, DataOutputStr
operatorState.getSubtaskStates();
dos.writeInt(subtaskStateMap.size());
for (Map.Entry<Integer, OperatorSubtaskState> entry :
subtaskStateMap.entrySet()) {
- dos.writeInt(entry.getKey());
- serializeSubtaskState(entry.getValue(), dos);
+ if (entry.getValue().isFinished()) {
+ // We store a negative index for the finished subtask. In
consideration
+ // of the index 0, the negative index would start from -1.
+ dos.writeInt(-(entry.getKey() + 1));
Review comment:
Is there no other way around this problem? Creating V4 serialiser would
be cleaner, but I think I would be also fine with this hack. Unfortunately I
don't have much experience with modifying this code.
Regardless I have one concern. Let's assume subtask `1` ends and it's
waiting for final checkpoint, checkpoint `42` happens, it's completed, subtask
`1` finishes, checkpoints `43`, `44`... are triggered and completed.
1. We need this flag only for that checkpoint `42`. Previous checkpoints
would be still running, checkpoints `43`, and the following ones (assuming we
didn't loose `notifyCheckpointComplete()` call) wouldn't contain finished
state. Right?
2. And the problem is that if we have even single finished subtasks we have
to disallow job graph modifications in front of that subtask, right?
3. In that case, is it enough to write this flag per subtask? What if in
checkpoint `44` there was no subtask that was waiting for the final checkpoint?
Subtask `1` is long finished. Will we write down the metadata for that subtask
`1`, and will we write down this negative number/flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
