On Thu, Mar 30, 2017 at 3:42 PM, Ben Chambers <bchamb...@google.com.invalid> wrote:
> When you say instance do you mean a different named PTransform class? Or > just a separate instance of whatever "HandleAssertResult" PTransform is > introduced? If the latter, I believe that makes sense. The latter. One class that "counts to one", and one new CountToOneSuccess after each assertion. It would do the metrics thing (which may be silently dropped by the runner) and also fail if it never sees a Success. > It also addresses > the problems with committed vs. attempted metrics. Specifically: > > Since we want to know that each assertion has run, we'd like to do > something like "Number of successful assertions" = N. But, in the presence > of retries, the number of successful assertions may be over counted. But, > if we have a separate transform for each assertion, then we can say "for > each transform, the number of successful assertions should be >= 1", which > ensures that every assert has succeeded. This will work with any metric > implementation that may overcount but never (or sufficiently rarely) > undercount. > > On Thu, Mar 30, 2017 at 3:39 PM Kenneth Knowles <k...@google.com.invalid> > wrote: > > > This sounds pretty good to me. I really like emphasizing the ability of a > > runner to go ahead and use pipeline surgery to put in their own > > verification transform. > > > > On Thu, Mar 30, 2017 at 3:12 PM, Pablo Estrada > <pabl...@google.com.invalid > > > > > wrote: > > > > > * The current asserting DoFns in PAssert will go from being > DoFn<ActualT, > > > Void> to be DoFn<ActualT, SuccesOrFailure>, and pass the output > > > SuccessOrFailure objects downstream to the checking > > > PTransform<SuccessOrFailure, Void>. > > > * This default PTransform<SuccessOrFailure, Void> will use metrics to > > count > > > successes, and fail immediately on failures (after incrementing the > > failure > > > counter). > > > > > > > Could it possibly be useful to have a separate instance of a > > PTransform<SuccessOrFailure> for each assertion? It might be able to > proxy > > for counting, for runners that don't yet have even rudimentary metrics. > The > > transform could set up triggering to fire just once and fail if it has > not > > received one Success. The runner might still just not execute that > > transform, but we could set it up so that is extremely unlikely, and > anyhow > > it is better than no coverage. I could be missing something about the > > implementation issues here, though. > > > > Kenn > > >
