[
https://issues.apache.org/jira/browse/FLINK-19547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213701#comment-17213701
]
Yuan Mei edited comment on FLINK-19547 at 10/14/20, 7:40 AM:
-------------------------------------------------------------
*Some other prototypes for reference:*
* encode partial length at the head of a buffer, and wrap methods in
BufferBuilder and BufferConsumer. [https://github.com/apache/flink/pull/13501 -
Connect to preview|https://github.com/apache/flink/pull/13501]
* update BufferConsumer → BufferConsumerWithOffset.
[https://github.com/apache/flink/pull/13567 - Connect to
preview|https://github.com/apache/flink/pull/13567]
* Encode partial length at the head of a buffer, and decode from receiver
side’s deserializer.[https://github.com/apache/flink/pull/13586 - Connect to
preview|https://github.com/apache/flink/pull/13586]
*Some other options abandoned:*
* interrupt the RecordWriter's Serializer, abandoned because broadcasting
shares the same Serializer, and we can not clean up the Serializer because of
one broken channel
* use mailbox to send record barriers. This in theory should work, but the
mailbox is bounded with StreamTask but needed to be used in NettyServer (reader
side), having module conflicts. Besides, there might be problems that upstream
tasks have been finished, in which case record barriers would never be sent.
The reason is due to the lifecycle of tasks and result partitions are decoupled.
was (Author: ym):
Some other prototypes for reference:
# encode partial length at the head of a buffer, and wrap methods in
BufferBuilder and BufferConsumer. [https://github.com/apache/flink/pull/13501 -
Connect to preview|https://github.com/apache/flink/pull/13501]
# update BufferConsumer → BufferConsumerWithOffset.
[https://github.com/apache/flink/pull/13567 - Connect to
preview|https://github.com/apache/flink/pull/13567]
# Encode partial length at the head of a buffer, and decode from receiver
side’s deserializer.[https://github.com/apache/flink/pull/13586 - Connect to
preview|https://github.com/apache/flink/pull/13586]
Some other options abandoned:
# interrupt the RecordWriter's Serializer, abandoned because broadcasting
shares the same Serializer, and we can not clean up the Serializer because of
one broken channel
# use mailbox to send record barriers. This in theory should work, but the
mailbox is bounded with StreamTask but needed to be used in NettyServer (reader
side), having module conflicts. Besides, there might be problems that upstream
tasks have been finished, in which case record barriers would never be sent.
The reason is due to the lifecycle of tasks and result partitions are decoupled.
> Partial record clean up after receiver task fails and restart
> -------------------------------------------------------------
>
> Key: FLINK-19547
> URL: https://issues.apache.org/jira/browse/FLINK-19547
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.12.0
> Reporter: Yuan Mei
> Priority: Major
> Labels: pull-request-available
>
> Partial records happen if a record can not fit into one buffer, then the
> remaining part of the same record is put into the next buffer. Hence partial
> records only exist at the beginning of a buffer.
> Partial record clean-up is needed in the mode of approximate local recovery.
> If a record is spanning over multiple buffers, and the first (several)
> buffers have got lost due to the failure of the receiver task, the remaining
> data belonging to the same record in transition should be cleaned up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
