Will-Lo commented on code in PR #3850:
URL: https://github.com/apache/gobblin/pull/3850#discussion_r1434588243
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskStateCollectorService.java:
##########
@@ -252,22 +255,32 @@ public boolean apply(String input) {
}
final Queue<TaskState> taskStateQueue = Queues.newConcurrentLinkedQueue();
+ AtomicLong numStateStoreMissing = new AtomicLong(0L);
+ GrowthMilestoneTracker growthTracker = new GrowthMilestoneTracker();
try (ParallelRunner stateSerDeRunner = new
ParallelRunner(numDeserializerThreads, null)) {
for (final String taskStateName : taskStateNames) {
log.debug("Found output task state file " + taskStateName);
// Deserialize the TaskState and delete the file
stateSerDeRunner.submitCallable(new Callable<Void>() {
@Override
public Void call() throws Exception {
- TaskState taskState = taskStateStore.getAll(taskStateTableName,
taskStateName).get(0);
- taskStateQueue.add(taskState);
+ List<TaskState> matchingTaskStates =
taskStateStore.getAll(taskStateTableName, taskStateName);
+ if (matchingTaskStates.isEmpty()) {
Review Comment:
This PR only reduces logs in the case where we are missing task states, but
if multiple tasks have the same reason of failure they are still not pared
down?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
