phet commented on code in PR #3829:
URL: https://github.com/apache/gobblin/pull/3829#discussion_r1399926218
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java:
##########
@@ -316,6 +316,7 @@ private void
finalizeDatasetStateBeforeCommit(JobState.DatasetState datasetState
if (taskState.getWorkingState() != WorkUnitState.WorkingState.SUCCESSFUL
&& this.jobContext.getJobCommitPolicy() ==
JobCommitPolicy.COMMIT_ON_FULL_SUCCESS) {
// The dataset state is set to FAILED if any task failed and
COMMIT_ON_FULL_SUCCESS is used
+ log.info("Failed task state for " +
taskState.getWorkunit().getOutputFilePath());
Review Comment:
three thoughts:
a. combine w/ logging on line 322?
b. could we get a failure reason to include in the log?
c. could we exhaustively list the failures for observability's sake? if
doing, I recommend to count the total and to include the counter's current
value in each log msg
if fully listing for the general case concerns you WRT perf, I'd suggest a
config to control whether or not to.
also open to other suggestions for ways to obtain the exhaustive listing of
what failed for debugging purposes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
