AmatyaAvadhanula commented on code in PR #12404:
URL: https://github.com/apache/druid/pull/12404#discussion_r860306548
##########
server/src/test/java/org/apache/druid/metadata/SQLMetadataStorageActionHandlerTest.java:
##########
@@ -450,4 +465,149 @@ public void testRemoveTasksOlderThan() throws Exception
Assert.assertEquals(1, handler.getLogs(entryId2).size());
Assert.assertEquals(1, handler.getLogs(entryId3).size());
}
+
+ @Test
+ public void testGetTaskStatusPlusList()
+ {
+ // SETUP
+ TaskInfo<Map<String, Object>, Map<String, Object>> activeUnaltered =
getRandomTaskInfo(true);
+ insertTaskInfo(activeUnaltered, false);
+
+ TaskInfo<Map<String, Object>, Map<String, Object>> completedUnaltered =
getRandomTaskInfo(false);
+ insertTaskInfo(completedUnaltered, false);
+
+ TaskInfo<Map<String, Object>, Map<String, Object>> activeAltered =
getRandomTaskInfo(true);
+ insertTaskInfo(activeAltered, true);
+
+ TaskInfo<Map<String, Object>, Map<String, Object>> completedAltered =
getRandomTaskInfo(false);
+ insertTaskInfo(completedAltered, true);
+
+ Map<TaskLookup.TaskLookupType, TaskLookup> taskLookups = new HashMap<>();
+ taskLookups.put(TaskLookup.TaskLookupType.ACTIVE,
ActiveTaskLookup.getInstance());
+ taskLookups.put(TaskLookup.TaskLookupType.COMPLETE,
CompleteTaskLookup.of(null, Duration.millis(86400000)));
+
+ List<TaskStatusPlus> taskStatusPlusList;
+
+ // BEFORE MIGRATION
+
+ // Payload based fetch. task type and groupid will be populated
+ taskStatusPlusList = handler.getTaskStatusPlusList(taskLookups, null,
true);
+ Assert.assertEquals(4, taskStatusPlusList.size());
+ verify(completedUnaltered, taskStatusPlusList, false, false, true);
+ verify(completedAltered, taskStatusPlusList, false, true, false);
+ verify(activeUnaltered, taskStatusPlusList, true, false, false);
+ verify(activeAltered, taskStatusPlusList, true, true, false);
+
+ // New columns based fetch before migration is complete. type and payload
are null when altered = false
+ taskStatusPlusList = handler.getTaskStatusPlusList(taskLookups, null,
false);
+ Assert.assertEquals(4, taskStatusPlusList.size());
+ verify(completedUnaltered, taskStatusPlusList, false, false, true);
+ verify(completedAltered, taskStatusPlusList, false, true, true);
+ verify(activeUnaltered, taskStatusPlusList, true, false, true);
+ verify(activeAltered, taskStatusPlusList, true, true, true);
+
+ // MIGRATION
+ derbyConnectorRule.getConnector().migrateTaskTable(entryTable);
+
+ // Payload based fetch. task type and groupid will still be populated
+ taskStatusPlusList = handler.getTaskStatusPlusList(taskLookups, null,
true);
+ Assert.assertEquals(4, taskStatusPlusList.size());
+ verify(completedUnaltered, taskStatusPlusList, false, false, false);
+ verify(completedAltered, taskStatusPlusList, false, true, false);
+ verify(activeUnaltered, taskStatusPlusList, true, false, false);
+ verify(activeAltered, taskStatusPlusList, true, true, false);
+
+ // New columns based fetch before migration is complete.
+ // type and payload are not null for completed task but are still null for
active ones since they aren't migrated
+ // An active task will be eventually updated on its own due to insertion
Review Comment:
What are the potential issues of trying to update the type and groupId of
active tasks during migration as well?
The tasks will be processed as small batches and the resources will be
returned after each read / update of each of these as you had suggested.
PROS:
1) Fetching and updating after (reverse) sorting by created_time ensures
that we utilize the index while also enabling storage of the timestamp as an
internal state in SQLMetadataStorageActionHandler. This timestamp can then be
used to query all tasks before it using the payload query, and the ones after
with the new query.
2) The above also means that we don't have to fetch the payload for all
tasks during the migration, and only those for which migration has yet to
happen.
3) Active tasks with empty type / groupId will not occur in the webconsole
4) We only need to handle type / groupId insertion during task insertion.
Any new / existing method to update the task in the db doesn't have to be
handled
CONS:
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
