TheNeuralBit commented on PR #17384:
URL: https://github.com/apache/beam/pull/17384#issuecomment-1117595689
It looks like its a pre-existing issue related to passing a large input
(order 1000 elements) through test_batch_pardo in order to trigger a flush. I
can reproduce the flake in a pipeline that doesn't have a batched DoFn:
```
def test_large_input_pardo(self):
with self.create_pipeline() as p:
res = (
p
| beam.Create(np.array(range(5000),
dtype=np.int64)).with_output_types(np.int64)
| beam.Map(lambda e: e * 2)
| beam.Map(lambda e: e + 3))
assert_that(res, equal_to([(i * 2) + 3 for i in range(5000)]))
```
It's worth noting I haven't seen the flake in a cython environment. Perhaps
it's a bug in the "slow" stack which is often only tested with trivial inputs?
I don't think I'll be able to diagnose the existing issue today. A solution
for this PR might be to skip the flush-triggering test in non-cythonized
environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
