Hmmm, the only thing that comes to my mind is that we started to use Spark AccumulatorsV2 API [1] but it was released just right in Beam 2.13.0 There is a related issue with this upgrade [2] but seems it’s not related.
[1] https://issues.apache.org/jira/browse/BEAM-4552 <https://issues.apache.org/jira/browse/BEAM-4552> [2] https://issues.apache.org/jira/browse/BEAM-10294 <https://issues.apache.org/jira/browse/BEAM-10294> > On 2 Jul 2020, at 22:07, Truebody, Kyle <[email protected]> wrote: > > Hi, > > We have recently upgraded to the latest version of Apache Beam 2.22.0. We > were previously using version 2.13.0 . > > We are using the SparkRunner. > > I noticed that after the upgrade that the Metrics query has stop producing > values. Through debugging I can see that the metrics and distribution are > still being incremented as expected. Has the metrics interface changed at > all? This is currently how we querying the metrics. > > > PipelineResult result = p.run(); > for(MetricResult r: > result.metrics().queryMetrics(MetricsFilter.builder().build()).getCounters()) > { > LOGGER.info <http://logger.info/>("n=" + r.getName() + " v=" + > r.getAttempted().toString()); > } > for(MetricResult r: > result.metrics().queryMetrics(MetricsFilter.builder().build()).getDistributions()) > { > LOGGER.info <http://logger.info/>("n=" + r.getName() + " v=" + > r.getAttempted().toString()); > } > result.waitUntilFinish(); > > > Thanks, > Kyle
