TengHu opened a new pull request #12297:
URL: https://github.com/apache/flink/pull/12297

   > ## What is the purpose of the change
   > Flink triggers all panes belonging to one window at the same time. In 
other words, all panes are aligned and their triggers all fire simultaneously, 
causing the spiking workload effect. This pull request adds WindowStagger to 
generate staggering offset for each window assignment at runtime, so the 
workloads are distributed across time. Hence each window assignment is based on 
window size, window offset and staggering offset (generated in runtime).
   > This change only modifies TumblingProcessingTimeWindows, will send out 
other windows change in other PRs.
   > ## Brief change log
   > * _Add WindowStagger for generating staggering offsets_
   > * _Enable TumblingProcessingTimeWindows to generate staggering offsets if 
user enabled_
   > ## Verifying this change
   > This change is already covered by existing tests, such as 
   > This change added tests and can be verified as follows:
   > * _Added unit tests for WindowStagger_
   > * _Validated the change by running in our clusters with 3500 task managers 
in total on a stateful streaming program using sliding and tumbling windowing. 
Some dashboards are shown below_
   > _some system metrics_
   > ## Does this pull request potentially affect one of the following parts:
   > * Dependencies (does it add or upgrade a dependency): no
   > * The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: yes, TumblingProcessingTimeWindows(potentially all 
   > * The serializers: no
   > * The runtime per-record code paths (performance sensitive): don't know, 
probably no ?
   > * Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
   > * The S3 file system connector: no
   > ## Documentation
   > * Does this pull request introduce a new feature? yes
   > * If yes, how is the feature documented? JavaDocs

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

Reply via email to