This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 765467f94ee [hotfix][runtime] Avoid duplicating broadcast records
redundantly for hybrid shuffle
new 957804a9591 [hotfix] Create ConfigOptions for s3 access/secret keys
and endpoint
new 1226ab58a16 [hotfix] Move CompressionUtils to flink-core
new f71059732c6 [hotfix] Use newly defined ConfigOptions in
MinioTestContainer
new 5c4b1aa2bd9 [FLINK-35767][filesystems] Add PathsCopyingFileSystem
new 1da48e53fa7 [FLINK-35767][s3] Use s5cmd for fast files download from
the s3
new 73b0facb96a [FLINK-35768][state,rocksdb] Refactor
RocksDBStateDownloader
new dd853492a97 [FLINK-35768][state,rocksdb] Use fast files download in
RocksdBStateDownloader
new a056e0230e8 [FLINK-35768][tests] Add an ITCase for s5cmd
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../java/org/apache/flink/core/fs/FileSystem.java | 309 ++++-----------
.../java/org/apache/flink/core/fs/IFileSystem.java | 415 +++++++++++++++++++++
.../flink/core/fs/PathsCopyingFileSystem.java | 91 +++++
.../flink/core/fs/PluginFileSystemFactory.java | 18 +-
.../flink/core/fs/SafetyNetWrapperFileSystem.java | 15 +-
.../org/apache/flink}/util/CompressionUtils.java | 4 +-
.../main/java/org/apache/flink/util/FileUtils.java | 22 +-
.../flink/docs/rest/RestAPIDocGeneratorTest.java | 3 +-
.../fs/s3/common/AbstractS3FileSystemFactory.java | 43 ++-
.../flink/fs/s3/common/FlinkS3FileSystem.java | 290 +++++++++++++-
.../flink/fs/s3/common/MinioTestContainer.java | 26 +-
.../flink/fs/s3/common/MinioTestContainerTest.java | 4 +-
.../flink/fs/s3/common/S5CmdOnMinioITCase.java | 356 ++++++++++++++++++
...e.java => S5CmdOnHadoopS3FileSystemITCase.java} | 3 +-
.../flink/fs/s3presto/FlinkS3PrestoFileSystem.java | 2 +
.../flink/fs/s3presto/S3FileSystemFactory.java | 3 +
.../fs/s3presto/PrestoS3FileSystemITCase.java | 5 +-
.../flink/fs/s3presto/PrestoS3FileSystemTest.java | 6 +-
...e.java => S5CmdOnPrestoS3FileSystemITCase.java} | 3 +-
.../apache/flink/client/python/PythonEnvUtils.java | 2 +-
.../env/AbstractPythonEnvironmentManager.java | 2 +-
.../flink/python/util/CompressionUtilsTest.java | 1 +
.../flink/runtime/state/KeyGroupsStateHandle.java | 5 +
.../runtime/state/OperatorStreamStateHandle.java | 5 +
.../state/RetrievableStreamStateHandle.java | 5 +
.../flink/runtime/state/StreamStateHandle.java | 8 +
.../runtime/state/filesystem/FileStateHandle.java | 5 +
.../streaming/state/RocksDBStateDownloader.java | 158 ++++++--
28 files changed, 1495 insertions(+), 314 deletions(-)
create mode 100644
flink-core/src/main/java/org/apache/flink/core/fs/IFileSystem.java
create mode 100644
flink-core/src/main/java/org/apache/flink/core/fs/PathsCopyingFileSystem.java
rename {flink-python/src/main/java/org/apache/flink/python =>
flink-core/src/main/java/org/apache/flink}/util/CompressionUtils.java (99%)
create mode 100644
flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/S5CmdOnMinioITCase.java
copy
flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/{HAJobRunOnHadoopS3FileSystemITCase.java
=> S5CmdOnHadoopS3FileSystemITCase.java} (88%)
copy
flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/{HAJobRunOnPrestoS3FileSystemITCase.java
=> S5CmdOnPrestoS3FileSystemITCase.java} (88%)
