AHeise commented on a change in pull request #10435: [FLINK-13955][runtime]
migrate ContinuousFileReaderOperator to the mailbox execution model
URL: https://github.com/apache/flink/pull/10435#discussion_r370062615
##########
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/ContinuousFileReaderOperator.java
##########
@@ -54,39 +62,157 @@
* {@link ContinuousFileMonitoringFunction}. Contrary to the {@link
ContinuousFileMonitoringFunction}
* which has a parallelism of 1, this operator can have DOP > 1.
*
- * <p>As soon as a split descriptor is received, it is put in a queue, and
have another
- * thread read the actual data of the split. This architecture allows the
separation of the
- * reading thread from the one emitting the checkpoint barriers, thus removing
any potential
- * back-pressure.
+ * <p>This implementation uses {@link MailboxExecutor} to execute each action
and states:<ol>
+ * <li>start in {@link ReaderState#IDLE IDLE}</li>
+ * <li>upon receiving a split add it to the queue, switch to {@link
ReaderState#OPENING OPENING} and enqueue a
+ * {@link org.apache.flink.streaming.runtime.tasks.mailbox.Mail mail} with
self as {@link Runnable}</li>
+ * <li>open file, switch to {@link ReaderState#READING READING}, read one
record, re-enqueue self</li>
+ * <li>if no more records or splits available, switch back to {@link
ReaderState#IDLE IDLE}</li>
+ * </ol>
+ * On close:
+ * <ol>
+ * <li>if {@link ReaderState#IDLE IDLE} then close immediately</li>
+ * <li>otherwise switch to {@link ReaderState#CLOSING CLOSING}, call
{@link MailboxExecutor#yield() yield} in a loop
+ * until state is {@link ReaderState#CLOSED CLOSED}</li>
+ * <li>{@link MailboxExecutor#yield() yield()} causes remaining records
(and splits) to be processed in the same way as above</li>
+ * </ol></p>
+ * <p>Using {@link MailboxExecutor} allows to avoid explicit synchronization.
At most one mail should be enqueued at any
+ * given time.</p>
+ * <p>Using FSM approach allows to explicitly define states and enforce {@link
ReaderState#TRANSITIONS transitions} between them.</p>
*/
@Internal
public class ContinuousFileReaderOperator<OUT> extends
AbstractStreamOperator<OUT>
- implements OneInputStreamOperator<TimestampedFileInputSplit, OUT>,
OutputTypeConfigurable<OUT> {
+ implements OneInputStreamOperator<TimestampedFileInputSplit, OUT>,
OutputTypeConfigurable<OUT>, RunnableWithException {
private static final long serialVersionUID = 1L;
private static final Logger LOG =
LoggerFactory.getLogger(ContinuousFileReaderOperator.class);
- private FileInputFormat<OUT> format;
- private TypeSerializer<OUT> serializer;
+ private enum ReaderState {
+ IDLE {
+ @Override
+ public boolean
prepareToProcessRecord(ContinuousFileReaderOperator<?> op) {
+ LOG.warn("not processing any records in IDLE
state");
Review comment:
Should that be an exception?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
