robertwb commented on code in PR #30005:
URL: https://github.com/apache/beam/pull/30005#discussion_r1464121618
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -731,14 +731,46 @@ def _parse_window_spec(spec):
return beam.WindowInto(window_fn)
@staticmethod
- def log_for_testing():
+ def log_for_testing(
+ level: Optional[str] = 'INFO', prefix: Optional[str] = ''):
Review Comment:
It's not easy to get at the name of the transform here, runners often
prepend the name of the transform in their logs.
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -731,14 +731,46 @@ def _parse_window_spec(spec):
return beam.WindowInto(window_fn)
@staticmethod
- def log_for_testing():
+ def log_for_testing(
+ level: Optional[str] = 'INFO', prefix: Optional[str] = ''):
"""Logs each element of its input PCollection.
The output of this transform is a copy of its input for ease of use in
chain-style pipelines.
+
+ Args:
+ level: one of ERROR, INFO, or DEBUG, mapped to a corresponding
+ language-specific logging level
+ prefix: an optional identifier that will get prepended to the element
+ being logged
"""
+ # Keeping this simple to be language agnostic.
+ # The intent is not to develop a logging library (and users can always do)
+ # their own mappings to get fancier output.
+ log_levels = {
+ 'ERROR': logging.error,
+ 'INFO': logging.info,
+ 'DEBUG': logging.debug,
+ }
Review Comment:
Mostly here the intent was to be able to log above, at, or below what is
typically logged in service infrastructure (on the workers). FATAL is a bit
odd, 'cause we're not going to be killing the process on every message (one
could try to make every message fatal, but that's getting into different
territory), and different languages have WARN vs. WARNING. OFF is mostly useful
for a log filter, not a choice for what level to log at.
Mostly I'm trying to keep things simple and address the feedback that
BeamSchema_xxxxxxxxxxxxxxxxxxx was not very useful and we want to be able to do
this from Java as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
