Hisoka-X commented on code in PR #6024:
URL: https://github.com/apache/seatunnel/pull/6024#discussion_r1433604128
##########
seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java:
##########
@@ -124,10 +125,26 @@ public void write(SeaTunnelRow element) throws
IOException {
return;
}
dorisStreamLoad.writeRecord(serialize);
+ if (!dorisConfig.getEnable2PC()
+ && dorisStreamLoad.getRecordCount() >=
dorisConfig.getBatchSize()) {
+ flush();
+ startLoad(labelGenerator.generateLabel(lastCheckpointId));
Review Comment:
It will only worked when disabled 2pc, therefore, it has no impact on the
consistency of the data. The flush label will use current time when on disabled
2pc. Please refer
https://github.com/apache/seatunnel/blob/c324d663b4661e315361b2a498cab278458021f5/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/LabelGenerator.java#L33
PS: The reason for add this PR is that we had disabled checkpoint in batch
mode, so too much data may be written at the end of the task. Has no impact on
our current consistency semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
