phet commented on a change in pull request #3416:
URL: https://github.com/apache/gobblin/pull/3416#discussion_r731540667
##########
File path:
gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java
##########
@@ -144,9 +145,11 @@ public void notificationCallback(Notification
notification) {
*/
public void addEventToReportingQueue(GobblinTrackingEvent event) {
Review comment:
logging makes sense AFA collecting more info, but of course this isn't a
`final` method. (I don't believe our present impl to be
`FileFailureEventReporter`, but calling it out to illustrate the consideration.)
##########
File path:
gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java
##########
@@ -235,7 +238,10 @@ public T withConfig(Config config) {
@Override
public void close() {
try {
+ LOGGER.info(String.format("Closing event reporter %s",
this.getClass().getCanonicalName()));
Review comment:
how do we decide whether to use the `static` `LOGGER` vs. the
`@Slf4j`-furnished `log`?
##########
File path:
gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java
##########
@@ -235,7 +238,10 @@ public T withConfig(Config config) {
@Override
public void close() {
try {
+ LOGGER.info(String.format("Closing event reporter %s",
this.getClass().getCanonicalName()));
Review comment:
and, in the end, do they identify the same underlying logger?
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/FlowSpec.java
##########
@@ -125,25 +125,39 @@
throw new RuntimeException("Unable to create a FlowSpec URI: " + e, e);
}
}
- public CompilationError getCompilationError(String src, String dst, String
errorMessage) {
- return new CompilationError(src, dst, errorMessage);
+ public void addCompilationError(String src, String dst, String errorMessage)
{
+ this.compilationErrors.add(new CompilationError(getConfig(), src, dst,
errorMessage));
}
- public class CompilationError {
+ public static class CompilationError {
public int errorPriority;
public String errorMessage;
- CompilationError(String src, String dst, String errorMessage) {
+ CompilationError(Config config, String src, String dst, String
errorMessage) {
errorPriority = 0;
- if (!src.equals(ConfigUtils.getString(getConfig(),
ServiceConfigKeys.FLOW_SOURCE_IDENTIFIER_KEY, ""))){
+ if (!src.equals(ConfigUtils.getString(config,
ServiceConfigKeys.FLOW_SOURCE_IDENTIFIER_KEY, ""))){
errorPriority++;
}
- if (!ConfigUtils.getStringList(getConfig(),
ServiceConfigKeys.FLOW_DESTINATION_IDENTIFIER_KEY)
+ if (!ConfigUtils.getStringList(config,
ServiceConfigKeys.FLOW_DESTINATION_IDENTIFIER_KEY)
.containsAll(Arrays.asList(StringUtils.split(dst, ",")))){
errorPriority++;
}
this.errorMessage = errorMessage;
}
+
+ @Override
+ public int hashCode() {
+ return errorPriority + errorMessage.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj instanceof CompilationError) {
+ return ((CompilationError) obj).errorPriority == this.errorPriority
+ && ((CompilationError) obj).errorMessage.equals(this.errorMessage);
+ }
+ return false;
+ }
Review comment:
`lombok.EqualsAndHashCode`?
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/MultiHopFlowCompiler.java
##########
@@ -191,15 +191,13 @@ public void awaitHealthy() throws InterruptedException {
DataNode sourceNode = this.flowGraph.getNode(source);
if (sourceNode == null) {
- flowSpec.getCompilationErrors()
- .add(flowSpec.getCompilationError(source, destination,
String.format("Flowgraph does not have a node with id %s", source)));
+ flowSpec.addCompilationError(source, destination,
String.format("Flowgraph does not have a node with id %s", source));
Review comment:
nice!
##########
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##########
@@ -103,18 +102,13 @@ private String getErrorMessage(FlowSpec flowSpec) {
StringBuilder message = new StringBuilder("Flow was not compiled
successfully.");
if (!flowSpec.getCompilationErrors().isEmpty()) {
message.append(" Compilation errors encountered (Sorted by relevance):
");
- Object[] errors = flowSpec.getCompilationErrors().toArray();
+ Object[] errors =
flowSpec.getCompilationErrors().stream().distinct().collect(Collectors.toList()).toArray();
Review comment:
`... .distinct().toArray(FlowSpec.CompilationError[]::new)`?
(even avoids creating the intermediate `List`)
##########
File path:
gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java
##########
@@ -144,9 +145,11 @@ public void notificationCallback(Notification
notification) {
*/
public void addEventToReportingQueue(GobblinTrackingEvent event) {
if (this.reportingQueue.size() > this.queueCapacity * 2 / 3) {
+ log.info("Trigger immediate run to report the event since queue is
almost full");
immediatelyScheduleReport();
}
try {
+ log.debug("Offering one event to the metrics queue");
Review comment:
certainly we don't want to add undue performance overhead (so I agree w/
the debug level)... but will we be able to discern which log line here relates
to the particular event we're curious about? (e.g. would it help to include an
identifier/correlator of the GTE being enqueued?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
