snleee commented on code in PR #12346:
URL: https://github.com/apache/pinot/pull/12346#discussion_r1475197414
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:
##########
@@ -52,6 +54,7 @@ public class UpsertCompactionTaskGenerator extends
BaseTaskGenerator {
private static final String DEFAULT_BUFFER_PERIOD = "7d";
private static final double DEFAULT_INVALID_RECORDS_THRESHOLD_PERCENT = 0.0;
private static final long DEFAULT_INVALID_RECORDS_THRESHOLD_COUNT = 0;
+ private static final String DEFAULT_VALID_DOC_ID_TYPE =
"validDocIdsSnapshot";
Review Comment:
@tibrewalpratik17 We found that using in-memory based validDocIds is a bit
dangerous as it will not give us the consistency (e.g. fetching validDocIds
bitmap while the server is restarting & updating validDocIds)
1. by default, we would want to enforce the customer to use `segment
compaction + snapshot enabled in the upsert config`
2. We would still want to allow the previous approach (im-memory bitmap
based). In this case, we will ask the customers to pick the appropriate
validDocIdType.
How do you think?
cc: @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
