Hello everyone,
I'm Caideyipi, and is developing the Streaming Engine module. Here I wish to
share a new feature I introduced to the "Extraction" phase of Streaming Engine.
Formerly, the Streaming Engine is long restricted to unbounded stream. For
instance, if users want to extract the historical data only, they can only
judge the end themselves by querying the receiver side, and drop the pipe
manually. The inconvenience is apparent in not only the normal scenarios, but
the data subscription and the tsFile export service, which may need a terminal
event to halt the entire procedure.
Hence, a bound is introduced in the extractor phase. Now users can use
"source.mode" in "with source" clause ("source" can be omitted) to decide
whether to send the terminal event and drop the pipe automatically when the
transmission is completed. When "source.mode" == "query" (default is
"subscribe"), when the historical data is all processed/transmitted, a terminal
event is sent to the subsequent steps. When all the pipe tasks has completed,
the ConfigNode will collect the information in pipe heartbeats, and drop the
pipe if all the pipeTasks have completed.
Fervently hope you like the feature, and glad to see any ideas or suggestions.
Thanks for your reading and support.
Sincerely,
Caideyipi
