BjornPrime commented on code in PR #23830:
URL: https://github.com/apache/beam/pull/23830#discussion_r1009773376
##########
sdks/python/apache_beam/ml/inference/base_test.py:
##########
@@ -171,6 +171,38 @@ def test_unexpected_inference_args_passed(self):
FakeModelHandlerFailsOnInferenceArgs(),
inference_args=inference_args)
+ def test_increment_failed_batches_counter(self):
+ with self.assertRaises(ValueError, FakeModelHandlerFailsOnInferenceArgs):
+ with TestPipeline() as pipeline:
+ examples = [1, 5, 3, 10]
+ pcoll = pipeline | 'start' >> beam.Create(examples)
+ inference_args = {'key': True}
+ _ = pcoll | base.RunInference(FakeModelHandlerFailsOnInferenceArgs(),
+ inference_args=inference_args)
+ run_result = pipeline.run()
+ run_result.wait_until_finish()
+
+ metric_results = (
+
run_result.metrics().query(MetricsFilter().with_name('failed_batches_counter')))
+ num_failed_batches_counter = metric_results['counters'][0]
+ self.assertEqual(num_failed_batches_counter.committed, 1)
Review Comment:
If it breaks into several batches I expect that we would get a failure for
each of those batches. And it seems like you're saying that if the inference
fails, the test will retry the batch a certain number of times. If that's
consistent, we could force the batching to make sure we have a reliable count.
For the moment I've just changed it to assert that the failed_batches_counter
will be greater than zero, but I could make that more precise if you can
confirm my interpretation of what you're saying.
I'm not totally following what you're saying about transient failures
though. Do you mean, set up a FakeModelHandler to fail the first time and then
succeed, to see if we count the failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
