prodriguezdefino opened a new pull request, #41454:
URL: https://github.com/apache/beam/pull/41454

   Problem:
   A `ParDo` after a `GroupByKey` can read an empty windowed side input on 
Prism. An aggregation stage sets its output watermark to its earliest pending 
timestamp W before it fires. When it fires all its windows in one bundle, the 
consumer stage gets elements in windows that end after W, while its upstream 
watermark is still W. 
   
   `bundleReady` only compares W with the output watermark of the side input 
parents, so the stage can be ready. The ordinary stage kind then takes all 
pending elements, but `GetSideData` gives the bundle only the side input 
windows that end at or before W. 
   
   Elements in later windows read an empty side input: wrong results in Python 
(`AsIter` gives an empty iterable), a failed bundle in SDKs that reject an 
empty singleton. The stateful stage kind takes data elements the same way.
   
   Fix:
   A stage with side inputs holds a data element until the bundle watermark 
passes the end of the window of the element, which is the rule `GetSideData` 
uses to select side input windows. Elements in the global window and stages 
without side inputs are not held. The ordinary stage kind keeps held elements 
pending and does not reschedule for them alone, because a watermark change 
refreshes the stage. The stateful stage kind and `hasBuildableDataLocked` apply 
the same check to data elements.
   
   Tests:
   Unit tests for both stage kinds at several watermarks, and for the stage 
state of the race: each element in a bundle has side input data for its window. 
A Go integration test, `ValidateWindowedSideInputsAfterGBK`, reads a windowed 
side input after a `GroupByKey`.
   
   ```python
   """Reproduces a Prism race: a ParDo after a GroupByKey reads a windowed side 
input
   before the runner has the side input for the window.
   
   Run: python prism_side_input_race.py [--runs=150] [--prism_location=<prism 
binary>]
   """
   
   import argparse
   import sys
   
   import apache_beam as beam
   from apache_beam.options.pipeline_options import PipelineOptions
   from apache_beam.testing.util import assert_that
   from apache_beam.testing.util import equal_to
   from apache_beam.transforms import window
   
   ELEMENTS = [('a', 1, 0), ('a', 1, 1), ('a', 4, 6), ('b', 1, 7), ('b', 13, 8)]
   # (key, sum of the key in the window, sums of the window read from the side 
input)
   EXPECTED = sorted([('a', 2, [2]), ('a', 4, [5]), ('b', 1, [5]), ('b', 13, 
[13])])
   
   
   def run_once(pipeline_args):
     options = PipelineOptions(
         ['--runner=PrismRunner', '--environment_type=LOOPBACK'] + 
pipeline_args)
     with beam.Pipeline(options=options) as p:
       windowed = (
           p
           | beam.Create(ELEMENTS)
           | 'Timestamp' >> beam.Map(
               lambda e: window.TimestampedValue((e[0], e[1]), e[2]))
           | beam.WindowInto(window.FixedWindows(2)))
       window_sum = (
           windowed
           | 'Values' >> beam.Values()
           | 'Sum' >> beam.CombineGlobally(sum).without_defaults())
       result = (
           windowed
           | beam.GroupByKey()
           | 'ReadSide' >> beam.Map(
               lambda kv, sums: (kv[0], sum(kv[1]), list(sums)),
               sums=beam.pvalue.AsIter(window_sum))
           | 'Globally' >> beam.WindowInto(window.GlobalWindows()))
       assert_that(result, equal_to(EXPECTED))
   
   
   def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('--runs', type=int, default=20)
     args, pipeline_args = parser.parse_known_args()
     failures = 0
     for i in range(args.runs):
       try:
         run_once(pipeline_args)
         print('run %d: ok' % (i + 1), flush=True)
       except Exception as e:  # pylint: disable=broad-except
         failures += 1
         print('run %d: FAILED %s' % (i + 1, str(e)[-400:]), flush=True)
     print('%d of %d runs failed' % (failures, args.runs))
     sys.exit(1 if failures else 0)
   
   
   if __name__ == '__main__':
     main()
   ```
   
   How to reproduce: 
   ```
   python prism_side_input_race.py --runs=150
   python prism_side_input_race.py --runs=150 --prism_location=/path/to/prism
   ```
   
   Expected failure:
   ```
   Failed assert: [('a', 2, [2]), ('a', 4, [5]), ('b', 1, [5]), ('b', 13, 
[13])] ==
                  [('a', 2, []), ('a', 4, []), ('b', 1, []), ('b', 13, [])]
   ```
   
   The occurence of the failure is low: 
   | Prism binary | Runs | Failed |
   | --- | --- | --- |
   | Master at `1abc30e62a4`, no fix | 150 | 15 |
   | The same master with the fix | 150 | 0 |
   | Released Prism 2.77.0 | 40 | 2 |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to