This is an automated email from the ASF dual-hosted git repository.
rong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iotdb.git
The following commit(s) were added to refs/heads/master by this push:
new 55bea920618 Pipe: Swapped the start order of realtimeExtractor and
historicalExtractor to avoid losing data (#10927)
55bea920618 is described below
commit 55bea9206184681460a6548bec947391a36f502c
Author: Caideyipi <[email protected]>
AuthorDate: Wed Aug 23 18:14:50 2023 +0800
Pipe: Swapped the start order of realtimeExtractor and historicalExtractor
to avoid losing data (#10927)
---
.../apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java
b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java
index 35749c31a83..a895ef39fd9 100644
---
a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java
+++
b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java
@@ -191,8 +191,13 @@ public class IoTDBDataRegionExtractor implements
PipeExtractor {
private void startHistoricalExtractorAndRealtimeExtractor(
AtomicReference<Exception> exceptionHolder) {
try {
- historicalExtractor.start();
+ // Start realtimeExtractor first to avoid losing data. This may cause
some
+ // retransmission, yet it is OK according to the idempotency of IoTDB.
+ // Note: The order of historical collection is flushing data -> adding
all tsFile events.
+ // There can still be writing when tsFile events are added. If we start
+ // realtimeExtractor after the process, then this part of data will be
lost.
realtimeExtractor.start();
+ historicalExtractor.start();
} catch (Exception e) {
exceptionHolder.set(e);
LOGGER.warn(
