robertwb commented on code in PR #30043:
URL: https://github.com/apache/beam/pull/30043#discussion_r1458182676
##########
sdks/python/apache_beam/yaml/yaml_transform.py:
##########
@@ -892,6 +892,20 @@ def ensure_config(spec):
return spec
+def process_logging(spec):
+ if 'logging' in spec:
+ log_level = str(spec['logging']).upper()
+ if log_level not in logging.getLevelNamesMapping():
+ raise ValueError(
+ 'Invalid logging level set: '
+ f"{identify_object(spec['logging'])}. "
+ 'Valid levels are '
+ f'{", ".join(logging.getLevelNamesMapping())}')
+ logging.getLogger().setLevel(log_level)
+ else:
+ _LOGGER.setLevel('INFO')
Review Comment:
This could be surprising, as setting the logging externally would have no
effect if logging is not in the spec.
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -690,8 +690,11 @@ def LogForTesting():
The output of this transform is a copy of its input for ease of use in
chain-style pipelines.
"""
+ logger = logging.getLogger(LogForTesting.__name__)
Review Comment:
This needs to be rebased on the other changes.
##########
sdks/python/apache_beam/yaml/README.md:
##########
@@ -527,3 +527,23 @@ in a yaml file as
```
python -m apache_beam.yaml.main --yaml_pipeline_file=/path/to/pipeline.yaml
[other pipeline options such as the runner]
```
+
+## Logging
+
+The logging level can be set to increase or decrease logging output during
+pipeline translation.
+
+```
+pipeline:
+ transforms:
+ - type: ReadFromCsv
+ config:
+ path: /path/to/input*.csv
+ - type: WriteToJson
+ config:
+ path: /path/to/output.json
+ input: ReadFromCsv
+logging: ERROR
Review Comment:
I'm not sure it makes sense to have this as a top-level attribute in the
yaml. (E.g. it doesn't impact pipeline execution, just the construction).
We could make it an arg for the template (likely defaulting to INFO as the
only reason one would look at the template logs is if something went wrong),
and normal invocation of the main.py should probably not set the logging high
(but one could set it via an arg or programmatically).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
