This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a change to branch CELEBORN-83-FOLLOWUP
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
omit 4fd6af99 address comments.
omit e86ee6bc [CELEBORN-83][FOLLOWUP] Fix various bugs when using HDFS as
storage.
add 34adcb9d [CELEBORN-83][FOLLOWUP] Fix various bugs when using HDFS as
storage.
add de3ef0d6 [CELEBORN-102][REFACTOR] TIMEOUT default value should be
changed with network timeout (#1047)
add 0d38bad7 [CELEBORN-20][REFACTOR] Extract CommitManager from
LifecycleManager (#1050)
add c178adcb Merge branch 'main' into CELEBORN-83-FOLLOWUP
add f3f10487 [CELEBORN-75] Initialize flink plugin module (#1027)
add e2196e93 [CELEBORN-56] [ISSUE-945] handle map partition mapper end
(#1003)
add ea1c6301 [CELEBORN-80] FileWriter supports MapPartition (#1025)
add c7258cfc [CELEBORN-103] add handleMapPartitionPushData to support
mappartition (#1048)
add d3d40f73 [CELEBORN-106] flink-plugin supports shufflewrite:OutputGate
(#1051)
add 426918c1 [CELEBORN-109][FEATURE] Add new DiskStatus to represent
critical error (#1054)
add 60f6f878 [CELEBORN-11] ShuffleClient supports MapPartition shuffle
write:pushdata (#1036)
add c924a4ff [CELEBORN-61][CELEBORN-62][FEATURE] Shuffle client support
slow start, congestion avoidance and congestion control (#1052)
add e68954db [CELEBORN-81] Add codecov. (#1026)
add dac2ba6b [CELEBORN-114][REFACTOR] Keep same log code in spark2/spark3
of quota exceed (#1058)
add 84d67d01 [Celeborn-81][FOLLOWUP]Correct sure fire plugin args. (#1059)
add edf85de8 [CELEBORN-123] PushDataHandler handleRpcRequestCore fix bug
about val isMaster (#1063)
add 07217fa3 Merge branch 'main' into CELEBORN-83-FOLLOWUP
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4fd6af99)
\
N -- N -- N refs/heads/CELEBORN-83-FOLLOWUP (07217fa3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/maven.yml | 5 +
.../common => client-flink/flink-1.14}/pom.xml | 37 +-
.../plugin/flink/FlinkResultPartitionInfo.java | 52 ++
.../plugin/flink/RemoteShuffleDescriptor.java | 59 ++
.../celeborn/plugin/flink/RemoteShuffleMaster.java | 148 +++++
.../plugin/flink/RemoteShuffleOutputGate.java | 226 +++++++
.../plugin/flink/RemoteShuffleResource.java | 50 ++
.../plugin/flink/RemoteShuffleServiceFactory.java | 41 +-
.../celeborn/plugin/flink/ShuffleResource.java | 9 +-
.../plugin/flink/ShuffleResourceDescriptor.java | 67 ++
.../celeborn/plugin/flink/utils/FlinkUtils.java | 55 ++
.../celeborn/plugin/flink/utils/ThreadUtils.java | 48 ++
.../plugin/flink/RemoteShuffleMasterTest.java | 185 ++++++
.../common => client-flink/flink-common}/pom.xml | 36 +-
.../apache/celeborn/plugin/flink/utils/Utils.java | 73 +++
.../flink-shaded}/pom.xml | 6 +-
.../celeborn/RssShuffleFallbackPolicyRunner.scala | 2 +-
client/pom.xml | 5 +
.../org/apache/celeborn/client/ShuffleClient.java | 31 +-
.../apache/celeborn/client/ShuffleClientImpl.java | 245 +++++++-
.../celeborn/client/read/DfsPartitionReader.java | 6 +-
.../celeborn/client/ChangePartitionManager.scala | 8 +-
.../org/apache/celeborn/client/CommitManager.scala | 699 +++++++++++++++++++++
.../apache/celeborn/client/LifecycleManager.scala | 642 ++++---------------
.../apache/celeborn/client/ShuffleTaskInfo.scala | 95 +++
.../apache/celeborn/client/DummyShuffleClient.java | 31 +
.../celeborn/client/ShuffleClientBaseSuiteJ.java | 108 ++++
.../celeborn/client/ShuffleClientImplSuiteJ.java | 147 +++++
.../celeborn/client/ShuffleTaskInfoSuite.scala | 50 ++
.../apache/celeborn/common/meta/DiskStatus.java | 3 +-
common/src/main/proto/TransportMessages.proto | 6 +-
.../common/meta/PartitionLocationInfo.scala | 88 ++-
.../common/protocol/message/ControlMessages.scala | 36 +-
.../{IntParam.scala => FunctionConverter.scala} | 15 +-
.../org/apache/celeborn/common/util/Utils.scala | 1 +
.../apache/celeborn/common/util/UtilsSuite.scala | 46 ++
dev/reformat | 2 +
pom.xml | 79 ++-
.../celeborn/tests/client/ShuffleClientSuite.scala | 43 +-
.../apache/celeborn/tests/spark/HugeDataTest.scala | 12 +-
.../service/deploy/worker/storage/FileWriter.java | 116 +---
.../worker/storage/MapPartitionFileWriter.java | 97 +++
.../worker/storage/ReducePartitionFileWriter.java | 113 ++++
.../service/deploy/worker/PushDataHandler.scala | 133 +++-
.../service/deploy/worker/WorkerSource.scala | 1 +
.../deploy/worker/storage/StorageManager.scala | 62 +-
.../deploy/worker/storage/FileWriterSuiteJ.java | 9 +-
47 files changed, 3209 insertions(+), 819 deletions(-)
copy {client-spark/common => client-flink/flink-1.14}/pom.xml (71%)
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/FlinkResultPartitionInfo.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleDescriptor.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleMaster.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleOutputGate.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResource.java
copy
common/src/main/java/org/apache/celeborn/common/network/util/AbstractFileRegion.java
=>
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleServiceFactory.java
(52%)
copy common/src/main/scala/org/apache/celeborn/common/metrics/sink/Sink.scala
=>
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/ShuffleResource.java
(80%)
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/ShuffleResourceDescriptor.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/utils/FlinkUtils.java
create mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/utils/ThreadUtils.java
create mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterTest.java
copy {client-spark/common => client-flink/flink-common}/pom.xml (63%)
create mode 100644
client-flink/flink-common/src/main/java/org/apache/celeborn/plugin/flink/utils/Utils.java
copy {client-spark/spark-2-shaded => client-flink/flink-shaded}/pom.xml (95%)
create mode 100644
client/src/main/scala/org/apache/celeborn/client/CommitManager.scala
create mode 100644
client/src/main/scala/org/apache/celeborn/client/ShuffleTaskInfo.scala
create mode 100644
client/src/test/java/org/apache/celeborn/client/ShuffleClientBaseSuiteJ.java
create mode 100644
client/src/test/java/org/apache/celeborn/client/ShuffleClientImplSuiteJ.java
create mode 100644
client/src/test/scala/org/apache/celeborn/client/ShuffleTaskInfoSuite.scala
copy common/src/main/scala/org/apache/celeborn/common/util/{IntParam.scala =>
FunctionConverter.scala} (72%)
create mode 100644
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/MapPartitionFileWriter.java
create mode 100644
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/ReducePartitionFileWriter.java
