robertzych commented on code in PR #12275:
URL: https://github.com/apache/pinot/pull/12275#discussion_r1456134847
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:
##########
@@ -172,62 +162,22 @@ public static SegmentSelectionResult
processValidDocIdMetadata(Map<String, Strin
String.valueOf(DEFAULT_INVALID_RECORDS_THRESHOLD_COUNT)));
List<SegmentZKMetadata> segmentsForCompaction = new ArrayList<>();
List<String> segmentsForDeletion = new ArrayList<>();
- for (Map.Entry<String, String> streamResponse : responseSet) {
- JsonNode allValidDocIdMetadata;
- try {
- allValidDocIdMetadata =
JsonUtils.stringToJsonNode(streamResponse.getValue());
- } catch (IOException e) {
- LOGGER.error("Unable to parse validDocIdMetadata response for: {}",
streamResponse.getKey());
- continue;
- }
- Iterator<JsonNode> iterator = allValidDocIdMetadata.elements();
- while (iterator.hasNext()) {
- JsonNode validDocIdMetadata = iterator.next();
- long totalInvalidDocs =
validDocIdMetadata.get("totalInvalidDocs").asLong();
- String segmentName = validDocIdMetadata.get("segmentName").asText();
- SegmentZKMetadata segment = completedSegmentsMap.get(segmentName);
- long totalDocs = validDocIdMetadata.get("totalDocs").asLong();
- double invalidRecordPercent = ((double) totalInvalidDocs / totalDocs)
* 100;
- if (totalInvalidDocs == totalDocs) {
- segmentsForDeletion.add(segment.getSegmentName());
- } else if (invalidRecordPercent > invalidRecordsThresholdPercent
- && totalInvalidDocs > invalidRecordsThresholdCount) {
- segmentsForCompaction.add(segment);
- }
+ for (ValidDocIdMetadataInfo validDocIdMetadata :
validDocIdMetadataInfoList) {
+ long totalInvalidDocs = validDocIdMetadata.getTotalInvalidDocs();
+ String segmentName = validDocIdMetadata.getSegmentName();
+ SegmentZKMetadata segment = completedSegmentsMap.get(segmentName);
Review Comment:
Is it possible for `validDocIdMetadataInfoList` to contain consuming
segments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
