Hey Michael, That TestRule sounds like it might be pretty useful generally; would it be worth contributing back to Beam?
On Wed, May 10, 2017 at 4:12 AM, Michael Luckey <[email protected]> wrote: > Hi Pablo, > > we ended up to follow the path with setup of the global environment. > > Therefore we implemented a simple junit TestRule, which is used something > along the following: > > @Rule TestMetrics metrics = new TestMetrics(); > > .... > > @Test > public void invalids() { > final DoFnTester<InputT, OutputT> doFnTester = > DoFnTester.of(fixture); > doFnTester.processElement(input); > > assertThat(metrics,counterValue(fixture.ctr), is(1L)); > } > > Thats pretty close to what we did before and seems to be working very well. > > So again, thanks a lot for your help, very much appreciated, > > michel > > On Wed, May 10, 2017 at 11:01 AM, Michael Luckey <[email protected]> > wrote: > >> Hi Pablo, >> >> thx for the pointers. I'll have a look into it and let you know. >> >> Regards, >> >> michel >> >> On Wed, May 10, 2017 at 3:18 AM, Pablo Estrada <[email protected]> >> wrote: >> >>> Hi Michael, >>> I'm sorry. I see I did not read your first email properly. >>> >>> There are a couple places in the core SDK or runner code and tests that >>> used to use aggregators, and now use metrics. There are two reasonable >>> options for this: >>> 1. In [1], the test sets up the metrics global environment by setting >>> the current container (e.g. MetricsEnvironment.setCurrentContainer(new >>> MetricsContainer("anystep"));), and the LateDataFilter uses metrics >>> normally[2], by creating a counter that relies on the environment set up in >>> the test. >>> >>> 2. If you'd rather not rely on setting up a global environment, you can >>> use CounterCell, and pass it in to your test. In [3] it's not a test, but a >>> CounterCell is still created to keep internal statistics, and later its >>> value is checked [4]. As a note, you may note in [3] that CounterCells are >>> a bit quirky to create, as we did not intend for external users to be able >>> to create them. >>> >>> Let me know if these suggestions are helpful. >>> Best >>> -P. >>> >>> [1] - https://github.com/apache/beam/blob/master/runners/core-java >>> /src/test/java/org/apache/beam/runners/core/LateDataDropping >>> DoFnRunnerTest.java#L61 >>> [2] - https://github.com/apache/beam/blob/master/runners/core-java >>> /src/main/java/org/apache/beam/runners/core/LateDataDropping >>> DoFnRunner.java#L96 >>> [3] - https://github.com/apache/beam/blob/master/runners/spark/s >>> rc/main/java/org/apache/beam/runners/spark/stateful/SparkGro >>> upAlsoByWindowViaWindowSet.java#L210 >>> [4] - https://github.com/apache/beam/blob/master/runners/spark/s >>> rc/main/java/org/apache/beam/runners/spark/stateful/SparkGro >>> upAlsoByWindowViaWindowSet.java#L326 >>> >>> On Tue, May 9, 2017 at 4:33 PM Michael Luckey <[email protected]> >>> wrote: >>> >>>> Hi Pablo, >>>> >>>> thanks for your help! We certainly could change our testing code and >>>> involve execution of a pipeline during tests. >>>> >>>> But currently we are leveraging DoFnTester, i.e. scoping our tests to >>>> the DoFn only, which means, there is neither a pipeline nor a pipeline >>>> result involved, which i could call upon. >>>> >>>> It might be a bad idea trying to test counters on this basis, but as it >>>> was supported previously i thought we might have overlooked an API for >>>> accessing these metrics somehow within DoFnTesting. Not sure, wether it >>>> makes sense for the DoFnTester to somehow provide Metrics-Support to enable >>>> this kind of tests. I certainly do not like the idea to much starting to do >>>> some mocking of the metrics api within my test implementation. >>>> >>>> Regards, >>>> >>>> michel >>>> >>>> >>>> On Wed, May 10, 2017 at 1:10 AM, Pablo Estrada <[email protected]> >>>> wrote: >>>> >>>>> Hi Michael, >>>>> For the Metrics API, the way to programatically query the value of a >>>>> metric is by using the MetricResults.queryMetrics method. You get the >>>>> MetricResults object from the PipelineResult object, and query it like so: >>>>> >>>>> PipelineResult res = p.run(); >>>>> MetricQueryResults metricResult = res.metrics().queryMetrics(....); >>>>> >>>>> The queryMetrics method takes in a MetricsFilter instance. >>>>> >>>>> Not all runners support this operation. For the dataflow runner, PR >>>>> 2896[1] should add it. >>>>> >>>>> Let me know if you need more help with this. >>>>> Best >>>>> -P. >>>>> >>>>> [1] - https://github.com/apache/beam/pull/2896 >>>>> >>>>> On Tue, May 9, 2017 at 3:48 PM Michael Luckey <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> currently we are evaluating a migration from 0.6.0 to current. We >>>>>> encountered the following problem, which we currently not sure, how to >>>>>> best >>>>>> solve. >>>>>> >>>>>> Say we have a DoFn, which is using Aggregators, e.g. >>>>>> >>>>>> ctr = createAggregator("someCounter", Sum.ofLongs()); >>>>>> >>>>>> We were testing them with DoFn-Tester like >>>>>> >>>>>> final DoFnTester<Record, Record> doFnTester = >>>>>> DoFnTester.of(fixture); >>>>>> doFnTester.processElement(input); >>>>>> >>>>>> >>>>>> assertThat(doFnTester.getAggregatorValue(fixture.ctr).longValue(), >>>>>> is(1L)); >>>>>> >>>>>> As aggregators are removed now from the codebase, we are considering >>>>>> using Metrics instead. But we did not find an equivalent to the >>>>>> getAggregatorValue method on DoFnTester. >>>>>> >>>>>> Any suggestion how we could keep our counters tested within a unit >>>>>> test based on DoFnTester? Or do we have to find a completely different >>>>>> solution? Are we doing something completely wrong trying to test correct >>>>>> workings of our counters with this approach? >>>>>> >>>>>> Regards, >>>>>> >>>>>> michel >>>>>> >>>>>> >>>>>> >>>> >> >
