scwhittle commented on code in PR #37662:
URL: https://github.com/apache/beam/pull/37662#discussion_r2904224444
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java:
##########
@@ -121,6 +153,48 @@ enum Level {
void setWorkerLogLevelOverrides(WorkerLogLevelOverrides value);
+ /**
+ * This option controls the direct log levels for specifically named
loggers. If a message is
+ * configured to be sent to both directly to cloud logging and default
disk-based logging it will
+ * just be sent to disk-based logging. If an override only exists for a
logger for direct logging,
+ * the --defaultWorkerLogLevel will be used for the non-direct configuration
for the logger.
+ *
+ * <p>Later options with equivalent names override earlier options.
+ *
+ * <p>See {@link WorkerLogLevelOverrides} for more information on how to
configure logging on a
+ * per {@link Class}, {@link Package}, or name basis. If used from the
command line, the expected
+ * format is {"Name":"Level",...}, further details on {@link
WorkerLogLevelOverrides#from}.
+ */
+ @Description(
+ "This option controls the direct log levels for specifically named
loggers. "
+ + "The expected format is {\"Name\":\"Level\",...}. The Dataflow
worker supports a logging "
+ + "hierarchy based off of names that are '.' separated. For example,
by specifying the value "
+ + "{\"a.b.c.Foo\":\"DEBUG\"}, the logger for the class 'a.b.c.Foo'
will be configured to "
+ + "output logs at the DEBUG level. Similarly, by specifying the
value {\"a.b.c\":\"WARN\"}, "
+ + "all loggers underneath the 'a.b.c' package will be configured to
output logs at the WARN "
+ + "level. System.out and System.err levels are configured via
loggers of the corresponding "
+ + "name. Also, note that when multiple overrides are specified, the
exact name followed by "
+ + "the closest parent takes precedence. Note that if an override is
just provided for the direct log level "
+ + "for a logger, the default non-direct log level will be used for
non-direct logs.")
+ WorkerLogLevelOverrides getWorkerDirectLogLevelOverrides();
+
+ void setWorkerDirectLogLevelOverrides(WorkerLogLevelOverrides value);
Review Comment:
getSdkHarnessLogLevelOverrides and getWorkerLogLevelOverrides are synonyms
so I think they can be ignored here. I agree there is complication on the
other config but it boils down to for a given logger there is increasing
criticality where logs >= disk level are sent to disk and logs >= direct and <
disk are sent directly. The simplest usage just involves the two default
flags, for example could set disk to ERROR or OFF and direct to INFO.
```
FINEST ... INFO ... ERROR ... OFF
DROPPED | DIRECT | DISK
```
The customer explicitly wants control to send some loggers that are higher
throughput directly while sending less-throughput framework or higher
criticality logs to disk for more resilience to process crashes etc.
The timestamp is set when the log is created in java, so I think the
ordering/gaps concerns in the UI are not a concern except for the most very
recent logs where the disk has not yet uploaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
