phet commented on code in PR #3850:
URL: https://github.com/apache/gobblin/pull/3850#discussion_r1435129018
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskStateCollectorService.java:
##########
@@ -252,22 +255,32 @@ public boolean apply(String input) {
}
final Queue<TaskState> taskStateQueue = Queues.newConcurrentLinkedQueue();
+ AtomicLong numStateStoreMissing = new AtomicLong(0L);
+ GrowthMilestoneTracker growthTracker = new GrowthMilestoneTracker();
try (ParallelRunner stateSerDeRunner = new
ParallelRunner(numDeserializerThreads, null)) {
for (final String taskStateName : taskStateNames) {
log.debug("Found output task state file " + taskStateName);
// Deserialize the TaskState and delete the file
stateSerDeRunner.submitCallable(new Callable<Void>() {
@Override
public Void call() throws Exception {
- TaskState taskState = taskStateStore.getAll(taskStateTableName,
taskStateName).get(0);
- taskStateQueue.add(taskState);
+ List<TaskState> matchingTaskStates =
taskStateStore.getAll(taskStateTableName, taskStateName);
+ if (matchingTaskStates.isEmpty()) {
Review Comment:
correct: this solely addresses cases where the state store does not retrieve
the task state, but otherwise exits normally. perhaps in another sort of
failure, a state store impl might throw. this consolidation still permits such
failure to pass through uninterrupted.
since the state store already gave us the list of task state names on line
244, I'd expect any other such failure to be ephemeral (else an abject logical
bug in the state store). either way, I've avoided over-engineering the
solution, precisely, as you point out, because we'd lose valuable debugging
info by conflating dissimilar errors.
if the a future failure scenario should arise from which we gain a concrete
grasp on what kind of errors these might be, I'd suggest at that time to extend
this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
