This is an automated email from the ASF dual-hosted git repository.
xtsong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 7611928d0f1 [FLINK-28585][runtime] Ensure all the concurrent
executions of SpeculativeExecutionVertex to share the same input splits
new 5494fe68d86 [hotfix] Introduce BufferIndexAndChannel and make
HsSpillingStrategy using it instead of BufferWithIdentity.
new 50dc51191f5 [hotfix] HsSpillingStrategy.Decision return Map group by
subpartition id instead of List
new 0610e16ae17 [hotfix] HsMemoryDataSpiller's spilling thread will
trigger fatal error when an exception is encountered.
new 0d466603a98 [FLINK-27904][runtime] Introduce HsMemoryDataManager to
manage in-memory data of hybrid shuffle mode
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../partition/hybrid/BufferIndexAndChannel.java | 39 ++
.../network/partition/hybrid/HsBufferContext.java | 128 ++++++
.../partition/hybrid/HsFullSpillingStrategy.java | 19 +-
.../partition/hybrid/HsMemoryDataManager.java | 286 +++++++++++++
.../hybrid/HsMemoryDataManagerOperation.java | 52 +++
.../partition/hybrid/HsMemoryDataSpiller.java | 18 +-
.../hybrid/HsSelectiveSpillingStrategy.java | 18 +-
.../partition/hybrid/HsSpillingInfoProvider.java | 2 +-
.../partition/hybrid/HsSpillingStrategy.java | 47 +-
.../partition/hybrid/HsSpillingStrategyUtils.java | 28 +-
.../hybrid/HsSubpartitionMemoryDataManager.java | 471 +++++++++++++++++++++
.../partition/hybrid/HsBufferContextTest.java | 131 ++++++
.../hybrid/HsFullSpillingStrategyTest.java | 48 ++-
.../partition/hybrid/HsMemoryDataManagerTest.java | 214 ++++++++++
.../partition/hybrid/HsMemoryDataSpillerTest.java | 21 -
.../hybrid/HsSelectiveSpillingStrategyTest.java | 38 +-
.../hybrid/HsSpillingStrategyUtilsTest.java | 24 +-
.../HsSubpartitionMemoryDataManagerTest.java | 427 +++++++++++++++++++
...yTestUtils.java => HybridShuffleTestUtils.java} | 38 +-
.../partition/hybrid/TestingFileDataIndex.java | 96 +++++
.../hybrid/TestingMemoryDataManagerOperation.java | 119 ++++++
.../hybrid/TestingSpillingInfoProvider.java | 14 +-
.../partition/hybrid/TestingSpillingStrategy.java | 119 ++++++
23 files changed, 2250 insertions(+), 147 deletions(-)
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/BufferIndexAndChannel.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsBufferContext.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManager.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerOperation.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionMemoryDataManager.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsBufferContextTest.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerTest.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionMemoryDataManagerTest.java
rename
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/{HsSpillingStrategyTestUtils.java
=> HybridShuffleTestUtils.java} (55%)
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/TestingFileDataIndex.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/TestingMemoryDataManagerOperation.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/TestingSpillingStrategy.java
