Feel free to open a github issue to track this idea. For Dataflow, the potential workaround is to filter the logs: https://cloud.google.com/dataflow/docs/guides/filter-logs
On Fri, May 24, 2024 at 5:01 AM Ferran Fernández Garrido < ffernandez....@gmail.com> wrote: > Hi! > > I am trying to adjust the log level for a Beam YAML pipeline that uses > Kafka. Since Kafka is a multi-language transform, > default_sdk_harness_log_level is not currently supported, which means > I can't limit the logs (Kafka can be a bit too verbose sometimes). > This could lead to increased logging costs if you use a cloud service > such as Dataflow to run the pipeline. > > Is there any initiative at the moment to support > default_sdk_harness_log_level in multi-language transforms? Maybe some > workaround? I suppose that in terms of Beam YAML, we could pass these > options to the schema providers somehow. However, I don't think that > solution is ideal, as it would not address the issue for other people > using multi-language transforms outside of Beam YAML. > > Thanks! > > Ferran >
