Ox0400 commented on issue #35013: URL: https://github.com/apache/beam/issues/35013#issuecomment-2914725437
> How about introducing a new logging option to control the overall logging level, e.g., > > ``` > from apache_beam.options.pipeline_options import PipelineOptions > > class LoggingOptions(PipelineOptions): > @classmethod > def _add_argparse_args(cls, parser): > parser.add_argument( > '--log_level', > default='WARNING', > help='Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)') > ``` This is a very good suggestion. It effectively controls the logs of all modules and provides a control interface. However, there is an issue: when using `google_cloud_pipeline_components` to create tasks with `DataflowPythonJobOp` and run workflows, the default log level (log_level=WARNING) might not be user-friendly. It is necessary to specifically remind users to set the log level in the documentation of `google_cloud_pipeline_components`. Alternatively, the default parameters of `DataflowPythonJobOp` need to be modified. Is it permissible to forcefully print logs such as the Job URL ? I mean that such logs can be printed under the default behavior. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L897 ``` from google_cloud_pipeline_components.v1.dataflow import DataflowPythonJobOp ... @dsl.pipeline(name='test dataflow') def test_dataflow(): dataflow_task = DataflowPythonJobOp( project=project, location=region, python_module_path=dataflow_clean_local_path, requirements_file_path=requirements_file_path, temp_location=temp_location, args=[ "--project", project, "--region", region, "--log_level", "INFO", # Default is WARNING, Require set INFO or DEBUG. ] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
