fernando-wizeline commented on pull request #15663: URL: https://github.com/apache/beam/pull/15663#issuecomment-942465298
> @fernando-wizeline - I think there are 2 options. We could look into the metrics cluster logs (https://cloud.google.com/kubernetes/clusters/details/us-central1-a/metrics/nodes?project=apache-beam-testing) and tried to root cause the problem. I have tried this but I was not successful. Or we could revert this and other configuration changes and push the metrics again. > > Btw, is the metrics staging environment running as expected? Or is it also failing? Hi @aaltay! Sorry for the late late reply. Looks like after the revert of this PR, the jenkins job is back to green https://ci-beam.apache.org/job/beam_Prober_CommunityMetrics/ Not really sure why that specific line in the build.gradle file caused the freshness alert to activate. I'm taking a look at the configs for the beamgrafana container deployed to GCE (https://console.cloud.google.com/kubernetes/deployment/us-central1-a/metrics/default/beamgrafana/overview?project=apache-beam-testing&selectedType=c&selectedName=beamgrafana) and they're still there; JSON plugin and everything. That leads me to think that the new configs per se are not the root of the issue. What I still fail at determining is how the change in the build.gradle file can initiate the freshness alert. Whatever configuration the aforementioned change persisted is still there. The config itself was not reverted, just the line that applied the config. The newly added dashboard that uses the JSON plugin is still (a)live and kicking: http://metrics.beam.apache.org/d/dYwQFp7nk/github-actions-post-commit-tests?orgId=1 I'm gonna ask around the infra channel; see if they have any clue regarding this. Thanks for the help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
