prodriguezdefino opened a new pull request, #41454:
URL: https://github.com/apache/beam/pull/41454
Problem:
A `ParDo` after a `GroupByKey` can read an empty windowed side input on
Prism. An aggregation stage sets its output watermark to its earliest pending
timestamp W before it fires. When it fires all its windows in one bundle, the
consumer stage gets elements in windows that end after W, while its upstream
watermark is still W.
`bundleReady` only compares W with the output watermark of the side input
parents, so the stage can be ready. The ordinary stage kind then takes all
pending elements, but `GetSideData` gives the bundle only the side input
windows that end at or before W.
Elements in later windows read an empty side input: wrong results in Python
(`AsIter` gives an empty iterable), a failed bundle in SDKs that reject an
empty singleton. The stateful stage kind takes data elements the same way.
Fix:
A stage with side inputs holds a data element until the bundle watermark
passes the end of the window of the element, which is the rule `GetSideData`
uses to select side input windows. Elements in the global window and stages
without side inputs are not held. The ordinary stage kind keeps held elements
pending and does not reschedule for them alone, because a watermark change
refreshes the stage. The stateful stage kind and `hasBuildableDataLocked` apply
the same check to data elements.
Tests:
Unit tests for both stage kinds at several watermarks, and for the stage
state of the race: each element in a bundle has side input data for its window.
A Go integration test, `ValidateWindowedSideInputsAfterGBK`, reads a windowed
side input after a `GroupByKey`.
```python
"""Reproduces a Prism race: a ParDo after a GroupByKey reads a windowed side
input
before the runner has the side input for the window.
Run: python prism_side_input_race.py [--runs=150] [--prism_location=<prism
binary>]
"""
import argparse
import sys
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import window
ELEMENTS = [('a', 1, 0), ('a', 1, 1), ('a', 4, 6), ('b', 1, 7), ('b', 13, 8)]
# (key, sum of the key in the window, sums of the window read from the side
input)
EXPECTED = sorted([('a', 2, [2]), ('a', 4, [5]), ('b', 1, [5]), ('b', 13,
[13])])
def run_once(pipeline_args):
options = PipelineOptions(
['--runner=PrismRunner', '--environment_type=LOOPBACK'] +
pipeline_args)
with beam.Pipeline(options=options) as p:
windowed = (
p
| beam.Create(ELEMENTS)
| 'Timestamp' >> beam.Map(
lambda e: window.TimestampedValue((e[0], e[1]), e[2]))
| beam.WindowInto(window.FixedWindows(2)))
window_sum = (
windowed
| 'Values' >> beam.Values()
| 'Sum' >> beam.CombineGlobally(sum).without_defaults())
result = (
windowed
| beam.GroupByKey()
| 'ReadSide' >> beam.Map(
lambda kv, sums: (kv[0], sum(kv[1]), list(sums)),
sums=beam.pvalue.AsIter(window_sum))
| 'Globally' >> beam.WindowInto(window.GlobalWindows()))
assert_that(result, equal_to(EXPECTED))
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--runs', type=int, default=20)
args, pipeline_args = parser.parse_known_args()
failures = 0
for i in range(args.runs):
try:
run_once(pipeline_args)
print('run %d: ok' % (i + 1), flush=True)
except Exception as e: # pylint: disable=broad-except
failures += 1
print('run %d: FAILED %s' % (i + 1, str(e)[-400:]), flush=True)
print('%d of %d runs failed' % (failures, args.runs))
sys.exit(1 if failures else 0)
if __name__ == '__main__':
main()
```
How to reproduce:
```
python prism_side_input_race.py --runs=150
python prism_side_input_race.py --runs=150 --prism_location=/path/to/prism
```
Expected failure:
```
Failed assert: [('a', 2, [2]), ('a', 4, [5]), ('b', 1, [5]), ('b', 13,
[13])] ==
[('a', 2, []), ('a', 4, []), ('b', 1, []), ('b', 13, [])]
```
The occurence of the failure is low:
| Prism binary | Runs | Failed |
| --- | --- | --- |
| Master at `1abc30e62a4`, no fix | 150 | 15 |
| The same master with the fix | 150 | 0 |
| Released Prism 2.77.0 | 40 | 2 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]