ege-st commented on code in PR #12330:
URL: https://github.com/apache/pinot/pull/12330#discussion_r1468921167
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -500,15 +500,24 @@ public Response downloadValidDocIds(
String.format("Table %s segment %s is not a immutable segment",
tableNameWithType, segmentName),
Response.Status.BAD_REQUEST);
}
- MutableRoaringBitmap validDocIds =
- indexSegment.getValidDocIds() != null ?
indexSegment.getValidDocIds().getMutableRoaringBitmap() : null;
- if (validDocIds == null) {
+
+ // Adopt the same logic as the query execution to get the valid doc ids.
'FilterPlanNode.run()'
+ // If the queryableDocId is available (upsert delete is enabled), we
read the valid doc ids from it.
+ // Otherwise, we read the valid doc ids.
+ MutableRoaringBitmap validDocIdSnapshot = null;
Review Comment:
```suggestion
final MutableRoaringBitmap validDocIdSnapshot;
```
I believe this must either equal
`indexSegment.getQueryableDocIds().getMutableRoaringBitmap()` or
`indexSegment.getValidDocIds().getMutableRoaringBitmap()` and if it not
assigned one of those then an exception is thrown: so this can be changed to be
immutable.
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -580,17 +589,26 @@ private List<Map<String, Object>>
processValidDocIdMetadata(String tableNameWith
LOGGER.warn(msg);
continue;
}
- MutableRoaringBitmap validDocIds =
- indexSegment.getValidDocIds() != null ?
indexSegment.getValidDocIds().getMutableRoaringBitmap() : null;
- if (validDocIds == null) {
+
+ // Adopt the same logic as the query execution to get the valid doc
ids. 'FilterPlanNode.run()'
+ // If the queryableDocId is available (upsert delete is enabled), we
read the valid doc ids from it.
+ // Otherwise, we read the valid doc ids.
+ MutableRoaringBitmap validDocIdSnapshot = null;
+ if (indexSegment.getQueryableDocIds() != null) {
+ validDocIdSnapshot =
indexSegment.getQueryableDocIds().getMutableRoaringBitmap();
+ } else if (indexSegment.getValidDocIds() != null) {
+ validDocIdSnapshot =
indexSegment.getValidDocIds().getMutableRoaringBitmap();
+ }
+
+ if (validDocIdSnapshot == null) {
Review Comment:
```suggestion
} else {
```
Same as above, this logic is that `validDocIdSnapshot` must be assigned one
of the two values and if both are null than an exception is thrown. I think
this structure makes the logic more explicit.
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -500,15 +500,24 @@ public Response downloadValidDocIds(
String.format("Table %s segment %s is not a immutable segment",
tableNameWithType, segmentName),
Response.Status.BAD_REQUEST);
}
- MutableRoaringBitmap validDocIds =
- indexSegment.getValidDocIds() != null ?
indexSegment.getValidDocIds().getMutableRoaringBitmap() : null;
- if (validDocIds == null) {
+
+ // Adopt the same logic as the query execution to get the valid doc ids.
'FilterPlanNode.run()'
+ // If the queryableDocId is available (upsert delete is enabled), we
read the valid doc ids from it.
+ // Otherwise, we read the valid doc ids.
+ MutableRoaringBitmap validDocIdSnapshot = null;
+ if (indexSegment.getQueryableDocIds() != null) {
+ validDocIdSnapshot =
indexSegment.getQueryableDocIds().getMutableRoaringBitmap();
+ } else if (indexSegment.getValidDocIds() != null) {
+ validDocIdSnapshot =
indexSegment.getValidDocIds().getMutableRoaringBitmap();
+ }
+
+ if (validDocIdSnapshot == null) {
Review Comment:
```suggestion
} else {
```
I think this is equivalent and helps communicate that
index.getQueryableDocIds or index.getValidDocIds must be not null.
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:
##########
@@ -97,7 +99,14 @@ public List<PinotTaskConfig> generateTasks(List<TableConfig>
tableConfigs) {
continue;
}
- // TODO: add a check to see if the task is already running for the table
+ // Only schedule 1 task of this type, per table
+ Map<String, TaskState> incompleteTasks =
+ TaskGeneratorUtils.getIncompleteTasks(taskType, tableNameWithType,
_clusterInfoAccessor);
+ if (!incompleteTasks.isEmpty()) {
+ LOGGER.warn("Found incomplete tasks: {} for same table: {}. Skipping
task generation.",
Review Comment:
I haven't debugging Minion much, but I think it'd be helpful to include the
task type in the warning message.
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -580,17 +589,26 @@ private List<Map<String, Object>>
processValidDocIdMetadata(String tableNameWith
LOGGER.warn(msg);
continue;
}
- MutableRoaringBitmap validDocIds =
- indexSegment.getValidDocIds() != null ?
indexSegment.getValidDocIds().getMutableRoaringBitmap() : null;
- if (validDocIds == null) {
+
+ // Adopt the same logic as the query execution to get the valid doc
ids. 'FilterPlanNode.run()'
+ // If the queryableDocId is available (upsert delete is enabled), we
read the valid doc ids from it.
+ // Otherwise, we read the valid doc ids.
+ MutableRoaringBitmap validDocIdSnapshot = null;
Review Comment:
```suggestion
final MutableRoaringBitmap validDocIdSnapshot;
```
Same suggestion as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
