StephanEwen opened a new pull request #8880: [FLINK-12986] [network] Fix instability of BoundedBlockingSubpartition under memory pressure. URL: https://github.com/apache/flink/pull/8880 ## What is the purpose of the change This PR changes the implementation of the `BoundedBlockingResultpartition` and adds alternative implementations to evaluate. The current implementation (based on memory mapped files) turned out to have stability issues under large scale and memory pressure. Big thanks to Yingjie Cao for testing/benchmarking this at scale. The theory for the cause of the stability issue is that processes may stall when trying to allocate memory, because first memory would have to be freed by evicting a page from the memory mapped file and writing it to disk. This lazy persistence is the problem. We implemented and tested different approaches: - `MMAP`: Current implementation - `FILE`: Directly writes to a file and reads from the file, so no lazy writing causing stalls - `FILE_MMAP`: Directly writes to a file and maps the file only for reading. This also has no lazy write (on page eviction) Tests concluded that the `FILE_MMAP` variant is stable and the implementation of choice. The PR still includes the other implementations (with tests) for future experiments. ## Brief change log - Factors out the implementation of the bounded data storage from the `BoundedBlockingResultpartition` - Add the implementations for `FILE`, `MMAP`, `FILE_MMAP` - Generify tests to cover all three implementations (through test bases / parameterized tests) - Change `ResultPartitionFactory` to instantiate the `FILE_MMAP` variant for blocking result partitions. ## Verifying this change - Any batch job (DataSet API) with blocking shuffles can be used to verify this. Set the `ExecutionMode` to `BATCH_FORCED`. - The batch job should have multiple TaskManagers to trigger the Netty shuffle code paths ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable**
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
