jbandoro commented on issue #39026: URL: https://github.com/apache/beam/issues/39026#issuecomment-5098113151
> Note that the orange line is the step of PeriodicImpulse output. Plus, the "overall" watermark is the minimum timestamp of all. It's more a visual effect. The downstream (BigQuery write, etc) watermark (can be shown in other lines in your screenshot) isn't affected, which would control when window gets fired. There fore this is more a visual effect and not affecting semantics of downstream main input branch. Thank you for the update and taking the time to reproduce. The reason why I raised this issue was because the dataflow job metric, `dataflow.googleapis.com/job/data_watermark_age`, now is tracking the periodic impulse because I believe this metric reports the max watermark age of data that is awaiting processing. Since our streaming job writes to many different bigquery sinks, having the single job metric was helpful for monitoring and alerting. But upgrading to v2.67.0+ breaks the monitoring/alerting since the job's watermark age is now representative of the periodic impulse. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
