abhishekrb19 commented on code in PR #19191:
URL: https://github.com/apache/druid/pull/19191#discussion_r3052026045
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java:
##########
@@ -265,6 +267,130 @@ protected
List<SeekableStreamIndexTask<KafkaTopicPartition, Long, KafkaRecordEnt
return taskList;
}
+ @Override
+ public void submitBackfillTask(
+ Map<KafkaTopicPartition, Long> startOffsets,
+ Map<KafkaTopicPartition, Long> endOffsets
+ )
+ {
+ if (startOffsets == null || startOffsets.isEmpty() || endOffsets == null
|| endOffsets.isEmpty()) {
+ log.info("No offsets to backfill, skipping backfill task submission");
+ return;
+ }
+
+ try {
+ String backfillSupervisorId =
spec.getSpec().getDataSchema().getDataSource() + "_backfill";
+
+ // Get the backfillTaskCount from config
+ int backfillTaskCount =
spec.getSpec().getIOConfig().getBackfillTaskCount();
Review Comment:
Is there a reason to include this in the supervisor's `ioConfig`? It feels
like a property that should be supplied by the operator when invoking the API,
rather than being defined at the supervisor's `ioConfig`. It could instead
default to `taskCount / 2` (or a similar value) as part of the API invocation.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java:
##########
@@ -592,9 +592,16 @@ public Response specGetHistory(
@Path("/{id}/reset")
@Produces(MediaType.APPLICATION_JSON)
@ResourceFilters(SupervisorResourceFilter.class)
- public Response reset(@PathParam("id") final String id)
+ public Response reset(
+ @PathParam("id") final String id,
+ @QueryParam("backfill") Boolean backfill
Review Comment:
IMO it would be cleaner to introduce a new endpoint
`/{id}/resetAndBackfill` especially if we’re adding more backfill-specific
properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
