tibrewalpratik17 commented on code in PR #12346:
URL: https://github.com/apache/pinot/pull/12346#discussion_r1475202125
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:
##########
@@ -52,6 +54,7 @@ public class UpsertCompactionTaskGenerator extends
BaseTaskGenerator {
private static final String DEFAULT_BUFFER_PERIOD = "7d";
private static final double DEFAULT_INVALID_RECORDS_THRESHOLD_PERCENT = 0.0;
private static final long DEFAULT_INVALID_RECORDS_THRESHOLD_COUNT = 0;
+ private static final String DEFAULT_VALID_DOC_ID_TYPE =
"validDocIdsSnapshot";
Review Comment:
In that case, can we tag this PR as backward-incompatible as existing
compaction tasks will break without snapshot?
I agree of this inconsistent scenario:
```
in-memory based validDocIds is a bit dangerous as it will not give us the
consistency (e.g. fetching validDocIds bitmap while the server is restarting &
updating validDocIds)
```
Also the implementation presently doesn't cover the scenario for
`validDocIds` as validDocIdType in the generator / executor task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
