tibrewalpratik17 opened a new pull request, #13424: URL: https://github.com/apache/pinot/pull/13424
Right now, there is no way to schedule UpsertCompaction task for a segment which has just 1 invalid record. We prevented the `invalidRecordsThresholdCount` to be set as 0 from tableConfig although by default we put it as 0 in the generator flow. Also during taskGenerator, we do a strict increment check of the invalid threshold count: https://github.com/apache/pinot/blob/99f6934166633308547d37cacc634e03c723ab17/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java#L242-L244 So now either we need to set a very low decimal value of `invalidRecordsThresholdPercent` which might go to a very low decimal places based on invalid : total docs ratio of a segment. Or we can allow the value to be set as 0 and allow users to do compaction for segments with just 1 invalid record. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
