This is an automated email from the ASF dual-hosted git repository.
xtsong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 46757739cf5 [FLINK-30168][python] Fix DataStream.execute_and_collect
to support None data and ObjectArray
new 9175bc924ff [FLINK-30332][network] Introduce
InternalRegionWriteReadUtils to read and write region.
new 910dc1b0bab [FLINK-30332][network] Introduce
HsFileDataIndexSpilledRegionManager to manage spilled regions.
new e6dc74ac9c7 [FLINK-30332][network] Introduce HsFileDataIndexCache to
cache and manage index data.
new 8e50e24797f [FLINK-30332][network] HsFileDataIndex supports caching
index entry and introduce config option to configure cache size.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../generated/all_taskmanager_network_section.html | 12 +
.../netty_shuffle_environment_configuration.html | 12 +
.../NettyShuffleEnvironmentOptions.java | 18 +
.../io/network/NettyShuffleServiceFactory.java | 4 +-
.../network/partition/ResultPartitionFactory.java | 35 +-
.../network/partition/hybrid/HsFileDataIndex.java | 3 +
.../partition/hybrid/HsFileDataIndexCache.java | 245 +++++++++++++
.../partition/hybrid/HsFileDataIndexImpl.java | 93 +++--
.../HsFileDataIndexSpilledRegionManager.java | 58 +++
.../HsFileDataIndexSpilledRegionManagerImpl.java | 402 +++++++++++++++++++++
.../partition/hybrid/HsFileDataManager.java | 7 +-
.../partition/hybrid/HsResultPartition.java | 12 +-
.../hybrid/HybridShuffleConfiguration.java | 46 ++-
.../hybrid/InternalRegionWriteReadUtils.java | 102 ++++++
.../NettyShuffleEnvironmentConfiguration.java | 32 +-
.../io/network/NettyShuffleEnvironmentBuilder.java | 21 +-
.../network/partition/ResultPartitionBuilder.java | 21 +-
.../partition/ResultPartitionFactoryTest.java | 4 +-
.../partition/hybrid/HsFileDataIndexCacheTest.java | 142 ++++++++
.../partition/hybrid/HsFileDataIndexImplTest.java | 8 +-
...sFileDataIndexSpilledRegionManagerImplTest.java | 195 ++++++++++
.../partition/hybrid/HsFileDataManagerTest.java | 13 +-
.../partition/hybrid/HsMemoryDataManagerTest.java | 11 +-
.../hybrid/HsSubpartitionFileReaderImplTest.java | 22 +-
.../partition/hybrid/HsSubpartitionViewTest.java | 3 +-
.../partition/hybrid/HybridShuffleTestUtils.java | 37 ++
.../hybrid/InternalRegionWriteReadUtilsTest.java | 86 +++++
.../partition/hybrid/TestingFileDataIndex.java | 5 +
.../TestingFileDataIndexSpilledRegionManager.java | 111 ++++++
29 files changed, 1695 insertions(+), 65 deletions(-)
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsFileDataIndexCache.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsFileDataIndexSpilledRegionManager.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsFileDataIndexSpilledRegionManagerImpl.java
create mode 100644
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/InternalRegionWriteReadUtils.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsFileDataIndexCacheTest.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsFileDataIndexSpilledRegionManagerImplTest.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/InternalRegionWriteReadUtilsTest.java
create mode 100644
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/TestingFileDataIndexSpilledRegionManager.java
