This is an automated email from the ASF dual-hosted git repository.
binjieyang pushed a change to branch CELEBORN-1768
in repository https://gitbox.apache.org/repos/asf/celeborn.git
discard 73730f64a add apache license
discard 562752121 fix doc style
discard f080f7ce1 [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to
extract common methods
add 372ef79a0 [CELEBORN-1760] OOM causes disk buffer unable to be released
add 22ee8bfed [CELEBORN-1765] Fix NPE when removeFileInfo in StorageManager
add 91d8f955c [CELEBORN-1622][CIP-11] Adding documentation for Worker Tags
feature
add 80ebb1983 [CELEBORN-1761][CIP-14] Add cppProto to cppClient
add 11cbacb04 [CELEBORN-1767] Fix occasional errors in UT when creating
workers
add 069e5b6c1 [CELEBORN-1769] Fix packed partition location cause
GetReducerFileGroupResponse lose location
add f7b036d4c [CELEBORN-1770] FlushNotifier should setException for all
Throwables in Flusher
add c316fdbdf Revert "[CELEBORN-1376] Push data failed should always
release request body"
add ca8831e55 [CELEBORN-1736] Add tez integration tests
add 74c1ec0a7 [CELEBORN-1670] Avoid swallowing InterruptedException in
ShuffleClientImpl
add 4aabe3776 [CELEBORN-1778] Fix commitInfo NPE and add assert in
LifecycleManagerCommitFilesSuite
add d85fb7826 [CELEBORN-1711][TEST] RetryReviveTest - shutdownMiniCluster
after each test
add c40f69b94 [CELEBORN-1766] Add detail metrics about fetch chunk
add 33ba0e02f [CELEBORN-1775] Improve some logs
add 17df678c7 [CELEBORN-1780] Add support for NodePort Service per Master
replica
add 2efdf755c [CELEBORN-1711][TEST] Fix flaky test caused by master/worker
setup issue
add b24f86778 [CELEBORN-1510] Partial task unable to switch to the replica
add 0eb8af98d [CELEBORN-1774] Update default value of
celeborn.<module>.io.mode to whether epoll mode is available
add f3dac7e87 [CELEBORN-1712] Bump Netty version from 4.1.109.Final to
4.1.115.Final
add e75d84fc1 [CELEBORN-1772][CIP-14] Add memory module to cppClient
add 6cce51e59 [CELEBORN-1786] add serviceAccount helm chart
add cec88b2de [CELEBORN-1782] Worker in congestion control should be in
blacklist to avoid impact new shuffle
add 67971df68 [CELEBORN-1783] Fix Pending task in commitThreadPool wont be
canceled
add 6b884dee6 [CELEBORN-1777] Add `java.security.jgss/sun.security.krb5`
to DEFAULT_MODULE_OPTIONS
add a357df94f [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount
metric for fallback to vanilla Flink built-in shuffle implementation
add 80523214e [MINOR] Add documentation for `CELEBORN_NO_DAEMONIZE`
add eaa0726c5 [CELEBORN-1788] Add role and roleBinding helm charts
add 9e04ff4a9 [CELEBORN-1763] Fix DataPusher be blocked for a long time
add e496a3cfa [CELEBORN-1785][CIP-14] Add baseConf to cppClient
add 406ceb64c [CELEBORN-1794] Fix TestCongestionController occasional
failing
add 680b072b5 [CELEBORN-1753] Optimize the code for `exists` and `find`
method
add 2eb4c23eb [CELEBORN-1771] Bring forward
PUSH_DATA_CREATE_CONNECTION_FAIL_REPLICA
add 03656b5b1 [CELEBORN-1634][FOLLOWUP] Add rpc metrics into grafana
dashboard
add 27e34ecad [CELEBORN-1797] Support to adjust the logger level with
RESTful API during runtime
add 6028a049d [CELEBORN-1700][FOLLOWUP] Fix flaky test
RemoteShuffleMasterSuiteJ - testRegisterPartitionWithProducer
add 4b60dae0f [CELEBORN-1789][DOC] Document on Java Columnar Shuffle
add fde6365f6 [CELEBORN-1413] Support Spark 4.0
add 7f030d424 [CELEBORN-1799][CIP-14] Add celebornConf to cppClient
add 52fa151aa [CELEBORN-1701][FOLLOWUP] Support stage rerun for shuffle
data lost
add 1b3bd6eb3 [CELEBORN-1802] Fail the celeborn master/worker start if
CELEBORN_CONF_DIR is not directory
add eb59c1763 [CELEBORN-1806] Bump Spark from 3.5.3 to 3.5.4
add 4714e9142 [CELEBORN-1809][CIP-14] Add partitionLocation to cppClient
add d0d8edfe2 [CELEBORN-1737] Support build tez client package
add a57238024 [CELEBORN-1801] Remove out-of-dated flink 1.14 and 1.15
add 56019c714 [CELEBORN-1804] Shuffle environment metrics of
RemoteShuffleEnvironment should use Shuffle.Remote metric group
add 4ec02286e [CELEBORN-1811] Update default value for
`celeborn.master.slot.assign.extraSlots`
add d6496ae18 [CELEBORN-1801][FOLLOWUP] Extract RemoteShuffleEnvironment,
NettyShuffleEnvironmentWrapper, SimpleResultPartitionAdapter to flink common
module
add a318eb43a [CELEBORN-1815] Support UnpooledByteBufAllocator
add 16762c659 [CELEBORN-1774][FOLLOWUP] Change celeborn.<module>.io.mode
optional to explain default behavior in description
add b7e3eaa46 [CELEBORN-1477][FOLLOWUP] Minor fix for v1 RESTful apis
before release
add 5d2831bba [CELEBORN-1816] Bump scala-maven-plugin to avoid compilation
loop
add 0c3ceeb0a [CELEBORN-1791] All NettyMemoryMetrics should register to
source
add 8b096ea87 [CELEBORN-1814][CIP-14] Add transportMessage to cppClient
add 4ccb0c7fc [MINOR] Rename org.apache.celeborn.plugin.flink.readclient
to org.apache.celeborn.plugin.flink.client
add f886751e8 [CELEBORN-1812] Distinguish sorting-file from sort-tasks
waiting to be submitted
add 6853b23b4 [CELEBORN-1819][CIP-14] Refactor cppClient with nested
namespace
add ca60613f2 [CELEBORN-1817] add committed file size metrics
add 61c90e3a0 [CELEBORN-1818] Fix incorrect timeout exception when waiting
on no pending writes
add eb9e16480 [CELEBORN-1820] Failing to write and flush StreamChunk data
should be counted as FETCH_CHUNK_FAIL
add becc6b065 [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to
extract common methods
add 1671c86c2 fix doc style
add 52da4bd35 add apache license
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (73730f64a)
\
N -- N -- N refs/heads/CELEBORN-1768 (52da4bd35)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/deps.yml | 5 +-
.github/workflows/license.yml | 2 -
.github/workflows/maven.yml | 8 +-
.github/workflows/sbt.yml | 2 -
.github/workflows/style.yml | 2 -
LICENSE | 2 +
README.md | 29 +-
assets/grafana/celeborn-dashboard.json | 3060 +++++++++++++++-----
build/make-distribution.sh | 33 +-
build/release/release.sh | 6 -
charts/celeborn/ci/values.yaml | 15 +
charts/celeborn/templates/_helpers.tpl | 22 +
charts/celeborn/templates/master/service.yaml | 31 +
charts/celeborn/templates/master/statefulset.yaml | 1 +
.../{worker/priorityclass.yaml => role.yaml} | 11 +-
.../priorityclass.yaml => rolebinding.yaml} | 17 +-
.../priorityclass.yaml => serviceaccount.yaml} | 10 +-
charts/celeborn/templates/worker/statefulset.yaml | 1 +
charts/celeborn/tests/master/service_test.yaml | 26 +
.../priorityclass_test.yaml => role_test.yaml} | 49 +-
...iorityclass_test.yaml => rolebinding_test.yaml} | 41 +-
...onfigmap_test.yaml => serviceaccount_test.yaml} | 30 +-
charts/celeborn/values.yaml | 29 +-
.../flink/AbstractRemoteShuffleInputGate.java} | 118 +-
.../AbstractRemoteShuffleInputGateFactory.java | 2 +-
.../AbstractRemoteShuffleResultPartition.java} | 30 +-
...bstractRemoteShuffleResultPartitionFactory.java | 2 +-
.../flink/AbstractRemoteShuffleServiceFactory.java | 31 +-
.../plugin/flink/RemoteBufferStreamReader.java | 4 +-
...ironment.java => RemoteShuffleEnvironment.java} | 117 +-
.../flink/RemoteShuffleInputGateDelegation.java | 2 +-
.../celeborn/plugin/flink/RemoteShuffleMaster.java | 55 +-
.../plugin/flink/RemoteShuffleOutputGate.java | 2 +-
.../plugin/flink/ResultPartitionAdapter.java | 24 -
.../plugin/flink/ShuffleResourceTracker.java | 4 +-
.../CelebornBufferStream.java | 2 +-
.../FlinkShuffleClientImpl.java | 2 +-
.../fallback/ShuffleFallbackPolicyRunner.java | 8 +-
.../flink/metric/RemoteShuffleMetricFactory.java | 76 +
.../flink/metric/RequestedMemoryUsageMetric.java | 40 +-
.../netty/NettyShuffleEnvironmentWrapper.java | 0
.../flink/network/FlinkTransportClientFactory.java | 2 +-
.../plugin/flink/FlinkShuffleClientImplSuiteJ.java | 4 +-
.../flink/RemoteShuffleOutputGateSuiteJ.java | 2 +-
client-flink/flink-1.14-shaded/pom.xml | 137 -
.../src/main/resources/META-INF/LICENSE | 248 --
.../src/main/resources/META-INF/NOTICE | 45 -
client-flink/flink-1.14/pom.xml | 75 -
.../plugin/flink/RemoteShuffleEnvironment.java | 138 -
.../plugin/flink/RemoteShuffleInputGate.java | 270 --
.../flink/RemoteShuffleInputGateFactory.java | 57 -
.../plugin/flink/RemoteShuffleResultPartition.java | 218 --
.../flink/RemoteShuffleResultPartitionFactory.java | 85 -
.../plugin/flink/RemoteShuffleServiceFactory.java | 63 -
.../plugin/flink/SimpleResultPartitionAdapter.java | 28 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 320 --
.../RemoteShuffleResultPartitionFactorySuiteJ.java | 55 -
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 618 ----
.../flink/RemoteShuffleServiceFactorySuiteJ.java | 58 -
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 140 -
.../src/main/resources/META-INF/LICENSE | 248 --
.../src/main/resources/META-INF/NOTICE | 45 -
client-flink/flink-1.15/pom.xml | 75 -
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../flink/RemoteShuffleInputGateFactory.java | 57 -
.../plugin/flink/RemoteShuffleResultPartition.java | 220 --
.../flink/RemoteShuffleResultPartitionFactory.java | 86 -
.../plugin/flink/RemoteShuffleServiceFactory.java | 63 -
.../plugin/flink/SimpleResultPartitionAdapter.java | 28 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 320 --
.../RemoteShuffleResultPartitionFactorySuiteJ.java | 55 -
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 618 ----
.../flink/RemoteShuffleServiceFactorySuiteJ.java | 58 -
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 140 -
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../plugin/flink/RemoteShuffleInputGate.java | 145 +-
.../plugin/flink/RemoteShuffleResultPartition.java | 187 +-
.../plugin/flink/RemoteShuffleServiceFactory.java | 18 +-
.../plugin/flink/SimpleResultPartitionAdapter.java | 27 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 38 +-
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 2 +-
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 20 +-
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../plugin/flink/RemoteShuffleInputGate.java | 145 +-
.../plugin/flink/RemoteShuffleResultPartition.java | 181 +-
.../plugin/flink/RemoteShuffleServiceFactory.java | 18 +-
.../plugin/flink/SimpleResultPartitionAdapter.java | 27 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 38 +-
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 2 +-
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 20 +-
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../plugin/flink/RemoteShuffleInputGate.java | 145 +-
.../plugin/flink/RemoteShuffleResultPartition.java | 181 +-
.../plugin/flink/RemoteShuffleServiceFactory.java | 18 +-
.../plugin/flink/SimpleResultPartitionAdapter.java | 27 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 38 +-
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 2 +-
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 20 +-
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../plugin/flink/RemoteShuffleInputGate.java | 145 +-
.../plugin/flink/RemoteShuffleResultPartition.java | 181 +-
.../plugin/flink/RemoteShuffleServiceFactory.java | 18 +-
.../plugin/flink/SimpleResultPartitionAdapter.java | 27 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 38 +-
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 2 +-
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 20 +-
.../plugin/flink/RemoteShuffleEnvironment.java | 139 -
.../plugin/flink/RemoteShuffleInputGate.java | 145 +-
.../plugin/flink/RemoteShuffleResultPartition.java | 181 +-
.../plugin/flink/RemoteShuffleServiceFactory.java | 18 +-
.../plugin/flink/SimpleResultPartitionAdapter.java | 27 -
.../netty/NettyShuffleEnvironmentWrapper.java | 95 -
.../flink/tiered/CelebornChannelBufferReader.java | 4 +-
.../flink/tiered/CelebornTierConsumerAgent.java | 2 +-
.../flink/tiered/CelebornTierProducerAgent.java | 2 +-
.../plugin/flink/RemoteShuffleMasterSuiteJ.java | 38 +-
.../flink/RemoteShuffleResultPartitionSuiteJ.java | 2 +-
.../plugin/flink/ShuffleResourceTrackerSuiteJ.java | 20 +-
.../spark/shuffle/celeborn/SparkCommonUtils.java | 6 +-
client-spark/{spark-3 => spark-3-4}/pom.xml | 4 +-
.../spark/shuffle/celeborn/BasedShuffleWriter.java | 0
.../shuffle/celeborn/CelebornShuffleDataIO.java | 0
.../shuffle/celeborn/HashBasedShuffleWriter.java | 0
.../shuffle/celeborn/SortBasedShuffleWriter.java | 0
.../shuffle/celeborn/SparkShuffleManager.java | 0
.../apache/spark/shuffle/celeborn/SparkUtils.java | 0
.../scala/org/apache/spark/SparkVersionUtil.scala | 0
.../spark/celeborn/ExceptionMakerHelper.scala | 0
.../CelebornShuffleFallbackPolicyRunner.scala | 0
.../shuffle/celeborn/CelebornShuffleHandle.scala | 0
.../shuffle/celeborn/CelebornShuffleReader.scala | 18 +-
.../celeborn/CelebornShuffleWriterSuiteBase.java | 0
.../celeborn/HashBasedShuffleWriterSuiteJ.java | 0
.../spark/shuffle/celeborn/ShuffleManagerHook.java | 0
.../celeborn/SortBasedShuffleWriterSuiteJ.java | 0
.../celeborn/TestCelebornShuffleManager.java | 0
.../src/test/resources/log4j.properties | 0
.../spark-3-4}/src/test/resources/log4j2-test.xml | 0
.../celeborn/CelebornShuffleManagerSuite.scala | 0
client-spark/spark-3-columnar-common/pom.xml | 2 +-
client-spark/spark-3-columnar-shuffle/pom.xml | 2 +-
client-spark/spark-3-shaded/pom.xml | 2 +-
.../pom.xml | 4 +-
.../celeborn/ColumnarHashBasedShuffleWriter.java | 0
.../celeborn/CelebornColumnarShuffleReader.scala | 0
.../columnar/CelebornColumnAccessor.scala | 0
.../execution/columnar/CelebornColumnBuilder.scala | 2 +-
.../execution/columnar/CelebornColumnStats.scala | 0
.../execution/columnar/CelebornColumnType.scala | 6 +-
.../columnar/CelebornColumnarBatchBuilder.scala | 0
.../CelebornColumnarBatchCodeGenBuild.scala | 0
.../columnar/CelebornColumnarBatchSerializer.scala | 0
.../CelebornCompressibleColumnAccessor.scala | 0
.../CelebornCompressibleColumnBuilder.scala | 0
.../columnar/CelebornCompressionScheme.scala | 0
.../columnar/CelebornCompressionSchemes.scala | 0
.../columnar/CelebornNullableColumnAccessor.scala | 0
.../columnar/CelebornNullableColumnBuilder.scala | 0
.../ColumnarHashBasedShuffleWriterSuiteJ.java | 0
.../src/test/resources/log4j2-test.xml | 0
.../CelebornColumnarShuffleReaderSuite.scala | 0
.../spark-4-shaded}/pom.xml | 12 +-
.../src/main/resources/META-INF/LICENSE | 0
.../src/main/resources/META-INF/NOTICE | 0
.../META-INF/licenses/LICENSE-protobuf.txt | 0
.../mr-shaded => client-tez/tez-shaded}/pom.xml | 35 +-
.../src/main/resources/META-INF/LICENSE | 0
.../tez-shaded}/src/main/resources/META-INF/NOTICE | 4 +-
.../META-INF/licenses/LICENSE-protobuf.txt | 0
.../META-INF/licenses/LICENSE-zstd-jni.txt | 0
.../apache/tez/dag/app/CelebornDagAppMaster.java | 84 +-
.../shuffle/orderedgrouped/CelebornShuffle.java | 2 +-
.../library/input/CelebornUnorderedKVInput.java | 2 +-
.../output/CelebornOrderedPartitionedKVOutput.java | 2 +-
.../library/output/CelebornUnorderedKVOutput.java | 3 +-
.../CelebornUnorderedPartitionedKVOutput.java | 3 +-
.../apache/celeborn/client/ShuffleClientImpl.java | 45 +-
.../celeborn/client/read/CelebornInputStream.java | 11 +-
.../apache/celeborn/client/write/DataPusher.java | 13 +-
.../celeborn/client/ChangePartitionManager.scala | 19 +-
.../org/apache/celeborn/client/CommitManager.scala | 13 +-
.../apache/celeborn/client/LifecycleManager.scala | 18 +-
.../celeborn/client/ReleasePartitionManager.scala | 14 +-
.../celeborn/client/WorkerStatusTracker.scala | 4 +-
.../celeborn/client/commit/CommitHandler.scala | 31 +-
.../client/commit/MapPartitionCommitHandler.scala | 2 +-
.../commit/ReducePartitionCommitHandler.scala | 4 +-
.../celeborn/client/ShuffleClientSuiteJ.java | 47 +-
.../celeborn/common/network/TransportContext.java | 13 +
.../common/network/client/TransportClient.java | 54 +-
.../network/client/TransportClientFactory.java | 15 +-
.../common/network/protocol/MessageEncoder.java | 10 +-
.../common/network/protocol/MessageWithHeader.java | 23 +-
.../common/network/server/TransportServer.java | 6 +-
.../common/network/util/NettyMemoryMetrics.java | 126 +-
.../celeborn/common/network/util/NettyUtils.java | 80 +-
.../common/network/util/TransportConf.java | 3 +-
.../celeborn/common/util/ExceptionUtils.java | 8 +
.../org/apache/celeborn/common/CelebornConf.scala | 40 +-
.../common/meta/ShufflePartitionLocationInfo.scala | 2 +-
.../apache/celeborn/common/meta/WorkerInfo.scala | 4 +-
.../common/meta/WorkerPartitionLocationInfo.scala | 16 +-
.../common/metrics/source/AbstractSource.scala | 65 +-
.../celeborn/common/metrics/source/Role.scala | 1 +
.../common/protocol/message/ControlMessages.scala | 15 +-
.../org/apache/celeborn/common/rpc/RpcEnv.scala | 8 +-
.../celeborn/common/rpc/RpcMetricsTracker.scala | 18 +-
.../org/apache/celeborn/common/rpc/RpcSource.scala | 12 +-
.../celeborn/common/rpc/netty/Dispatcher.scala | 2 +-
.../celeborn/common/rpc/netty/NettyRpcEnv.scala | 6 +-
.../apache/celeborn/common/util/PbSerDeUtils.scala | 12 +-
.../org/apache/celeborn/common/util/Utils.scala | 3 +-
.../apache/celeborn/common/CelebornConfSuite.scala | 3 +-
.../meta/ShufflePartitionLocationInfoSuite.scala | 3 +-
.../celeborn/common/meta/WorkerInfoSuite.scala | 2 +
.../metrics/source/CelebornSourceSuite.scala | 18 +-
.../celeborn/common/rpc/netty/InboxSuite.scala | 3 +-
.../common/rpc/netty/NettyRpcEnvSuite.scala | 4 +
.../celeborn/common/util/PbSerDeUtilsTest.scala | 99 +-
conf/celeborn-env.sh.template | 6 +
cpp/CMakeLists.txt | 17 +
cpp/README.md | 7 +
cpp/celeborn/CMakeLists.txt | 4 +
cpp/celeborn/conf/BaseConf.cpp | 230 ++
cpp/celeborn/conf/BaseConf.h | 265 ++
cpp/celeborn/{utils => conf}/CMakeLists.txt | 12 +-
cpp/celeborn/conf/CelebornConf.cpp | 201 ++
cpp/celeborn/conf/CelebornConf.h | 88 +
cpp/celeborn/conf/tests/BaseConfTest.cpp | 202 ++
cpp/celeborn/{utils => conf}/tests/CMakeLists.txt | 11 +-
cpp/celeborn/conf/tests/CelebornConfTest.cpp | 103 +
cpp/celeborn/memory/ByteBuffer.cpp | 78 +
cpp/celeborn/memory/ByteBuffer.h | 182 ++
cpp/celeborn/{utils => memory}/CMakeLists.txt | 13 +-
cpp/celeborn/memory/tests/ByteBufferTest.cpp | 221 ++
.../{utils => memory}/tests/CMakeLists.txt | 13 +-
cpp/celeborn/{ => proto}/CMakeLists.txt | 4 +-
cpp/celeborn/proto/TransportMessagesCpp.proto | 1 +
cpp/celeborn/{utils => protocol}/CMakeLists.txt | 15 +-
cpp/celeborn/protocol/PartitionLocation.cpp | 93 +
cpp/celeborn/protocol/PartitionLocation.h | 97 +
cpp/celeborn/protocol/StatusCode.h | 95 +
cpp/celeborn/protocol/TransportMessage.cpp | 50 +
.../celeborn/protocol/TransportMessage.h | 48 +-
.../{utils => protocol}/tests/CMakeLists.txt | 19 +-
.../protocol/tests/PartitionLocationTest.cpp | 111 +
.../protocol/tests/TransportMessageTest.cpp | 58 +
cpp/celeborn/utils/CMakeLists.txt | 10 +-
cpp/celeborn/utils/CelebornException.cpp | 2 +
cpp/celeborn/utils/CelebornException.h | 2 +
.../celeborn/utils/CelebornUtils.h | 32 +-
cpp/celeborn/utils/Exceptions.cpp | 12 +-
cpp/celeborn/utils/Exceptions.h | 176 +-
cpp/celeborn/utils/StackTrace.cpp | 6 +-
cpp/celeborn/utils/StackTrace.h | 6 +-
cpp/celeborn/utils/tests/ExceptionTest.cpp | 102 +-
dev/dependencies.sh | 14 +-
dev/deps/dependencies-client-flink-1.14 | 81 -
dev/deps/dependencies-client-flink-1.15 | 81 -
dev/deps/dependencies-client-flink-1.16 | 68 +-
dev/deps/dependencies-client-flink-1.17 | 68 +-
dev/deps/dependencies-client-flink-1.18 | 68 +-
dev/deps/dependencies-client-flink-1.19 | 68 +-
dev/deps/dependencies-client-flink-1.20 | 68 +-
dev/deps/dependencies-client-mr | 68 +-
dev/deps/dependencies-client-spark-2.4 | 68 +-
dev/deps/dependencies-client-spark-3.0 | 68 +-
dev/deps/dependencies-client-spark-3.1 | 68 +-
dev/deps/dependencies-client-spark-3.2 | 68 +-
dev/deps/dependencies-client-spark-3.3 | 68 +-
dev/deps/dependencies-client-spark-3.4 | 68 +-
dev/deps/dependencies-client-spark-3.5 | 68 +-
dev/deps/dependencies-client-tez | 178 ++
dev/deps/dependencies-server | 68 +-
dev/reformat | 4 +-
docs/README.md | 6 +-
docs/configuration/index.md | 31 +-
docs/configuration/master.md | 2 +-
docs/configuration/network.md | 2 +-
docs/deploy.md | 4 +-
docs/developers/java-columnar-shuffle.md | 56 +
docs/developers/overview.md | 2 +-
docs/developers/sbt.md | 26 +-
docs/migration.md | 6 +
docs/monitoring.md | 2 +
docs/worker_tags.md | 171 ++
.../celeborn/service/deploy/master/Master.scala | 31 +-
.../deploy/master/http/api/v1/WorkerResource.scala | 8 +-
.../service/deploy/master/tags/TagsManager.scala | 4 +-
.../ApiMasterResourceAuthenticationSuite.scala | 36 +-
.../deploy/master/MasterClusterFeature.scala | 157 +
.../service/deploy/master/MasterSuite.scala | 137 +-
.../master/http/api/ApiMasterResourceSuite.scala | 36 +-
.../http/api/v1/ApiV1MasterResourceSuite.scala | 36 +-
.../deploy/master/tags/TagsManagerSuite.scala | 18 +-
mkdocs.yml | 2 +
.../v1/master/{DefaultApi.java => LoggerApi.java} | 63 +-
.../apache/celeborn/rest/v1/master/WorkerApi.java | 4 +-
.../v1/model/{ConfigData.java => LoggerInfo.java} | 50 +-
.../model/{ConfResponse.java => LoggerInfos.java} | 52 +-
.../rest/v1/model/WorkerResourceConsumption.java | 40 +-
.../v1/worker/{ShuffleApi.java => LoggerApi.java} | 63 +-
.../apache/celeborn/rest/v1/worker/WorkerApi.java | 4 +-
.../src/main/openapi3/master_rest_v1.yaml | 71 +-
.../src/main/openapi3/worker_rest_v1.yaml | 71 +-
pom.xml | 86 +-
project/CelebornBuild.scala | 283 +-
sbin/celeborn-daemon.sh | 5 +
sbin/restart-worker.sh | 1 +
sbin/start-master.sh | 1 +
sbin/start-worker.sh | 1 +
service/pom.xml | 10 +
.../celeborn/common/metrics/sink/JsonServlet.scala | 4 +-
.../server/common/http/api/v1/ApiUtils.scala | 2 +-
.../common/http/api/v1/ApiV1BaseResource.scala | 3 +
.../server/common/http/api/v1/LoggerResource.scala | 88 +
.../server/common/http/ApiBaseResourceSuite.scala | 10 +-
.../server/common/http/HttpTestHelper.scala | 1 -
.../http/api/v1/ApiV1BaseResourceSuite.scala | 82 +-
.../apache/celeborn/tests/flink/FlinkVersion.java | 6 -
.../celeborn/tests/flink/HeartbeatTest.scala | 2 +-
tests/spark-it/pom.xml | 42 +-
.../ChangePartitionManagerUpdateWorkersSuite.scala | 34 +-
.../client/LifecycleManagerCommitFilesSuite.scala | 95 +-
.../celeborn/tests/spark/RetryReviveTest.scala | 1 +
tests/{mr-it => tez-it}/pom.xml | 168 +-
.../tests/tez/BroadcastAndOneToOneExample.java | 282 ++
.../tests/tez/BroadcastAndOneToOneTest.scala | 24 +-
.../tests/tez/TezCartesianProductTest.scala | 63 +
.../tests/tez/TezHashJoinBroadcastTest.scala | 36 +
.../celeborn/tests/tez/TezHashJoinTest.scala | 35 +
.../tests/tez/TezIntegrationTestBase.scala | 295 ++
.../tests/tez/TezJoinIntegrationTestBase.scala | 72 +
.../tests/tez/TezOrderedWordCountTest.scala | 73 +
.../celeborn/tests/tez/TezSortMergeJoinTest.scala | 37 +
.../celeborn/tests/tez/TezWordCountTest.scala | 60 +
tests/tez-it/src/test/resources/log4j.properties | 19 +
tests/tez-it/src/test/resources/log4j2.xml | 40 +
tests/tez-it/src/test/resources/tez-site.xml | 21 +
.../congestcontrol/CongestionController.java | 14 +-
.../deploy/worker/memory/MemoryManager.java | 20 +-
.../deploy/worker/memory/ReadBufferDispatcher.java | 15 +-
.../deploy/worker/storage/PartitionDataWriter.java | 33 +-
.../worker/storage/PartitionFilesSorter.java | 12 +-
.../service/deploy/worker/Controller.scala | 70 +-
.../service/deploy/worker/FetchHandler.scala | 10 +-
.../service/deploy/worker/PushDataHandler.scala | 86 +-
.../celeborn/service/deploy/worker/Worker.scala | 27 +-
.../service/deploy/worker/WorkerSource.scala | 8 +
.../deploy/worker/http/api/v1/WorkerResource.scala | 2 +-
.../service/deploy/worker/storage/Flusher.scala | 23 +-
.../deploy/worker/storage/StorageManager.scala | 18 +-
.../congestcontrol/TestCongestionController.java | 10 +-
.../storage/PartitionDataWriterSuiteUtils.java | 3 +-
.../local/DiskMapPartitionDataWriterSuiteJ.java | 2 +-
.../local/DiskReducePartitionDataWriterSuiteJ.java | 4 +-
.../memory/MemoryPartitionFilesSorterSuiteJ.java | 8 +-
.../MemoryReducePartitionDataWriterSuiteJ.java | 10 +-
.../service/deploy/MiniClusterFeature.scala | 68 +-
.../deploy/memory/ReadBufferDispactherSuite.scala | 4 +-
.../http/api/v1/ApiV1WorkerResourceSuite.scala | 2 +-
.../ApiWorkerResourceAuthenticationSuite.scala | 18 +-
367 files changed, 9973 insertions(+), 10757 deletions(-)
copy charts/celeborn/templates/{worker/priorityclass.yaml => role.yaml} (80%)
copy charts/celeborn/templates/{worker/priorityclass.yaml => rolebinding.yaml}
(71%)
copy charts/celeborn/templates/{worker/priorityclass.yaml =>
serviceaccount.yaml} (80%)
copy charts/celeborn/tests/{master/priorityclass_test.yaml => role_test.yaml}
(56%)
copy charts/celeborn/tests/{master/priorityclass_test.yaml =>
rolebinding_test.yaml} (60%)
copy charts/celeborn/tests/{configmap_test.yaml => serviceaccount_test.yaml}
(68%)
rename
client-flink/{flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleInputGate.java
=>
common/src/main/java/org/apache/celeborn/plugin/flink/AbstractRemoteShuffleInputGate.java}
(55%)
copy
client-flink/{flink-1.19/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartition.java
=>
common/src/main/java/org/apache/celeborn/plugin/flink/AbstractRemoteShuffleResultPartition.java}
(86%)
rename
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/{AbstractRemoteShuffleEnvironment.java
=> RemoteShuffleEnvironment.java} (60%)
delete mode 100644
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/ResultPartitionAdapter.java
rename
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/{readclient
=> client}/CelebornBufferStream.java (99%)
rename
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/{readclient
=> client}/FlinkShuffleClientImpl.java (99%)
create mode 100644
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/metric/RemoteShuffleMetricFactory.java
copy
spi/src/main/java/org/apache/celeborn/spi/authentication/BasicPrincipal.java =>
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/metric/RequestedMemoryUsageMetric.java
(51%)
rename client-flink/{flink-1.14 =>
common}/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
(100%)
delete mode 100644 client-flink/flink-1.14-shaded/pom.xml
delete mode 100644
client-flink/flink-1.14-shaded/src/main/resources/META-INF/LICENSE
delete mode 100644
client-flink/flink-1.14-shaded/src/main/resources/META-INF/NOTICE
delete mode 100644 client-flink/flink-1.14/pom.xml
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleInputGate.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleInputGateFactory.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartition.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionFactory.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleServiceFactory.java
delete mode 100644
client-flink/flink-1.14/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
delete mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionFactorySuiteJ.java
delete mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionSuiteJ.java
delete mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleServiceFactorySuiteJ.java
delete mode 100644
client-flink/flink-1.14/src/test/java/org/apache/celeborn/plugin/flink/ShuffleResourceTrackerSuiteJ.java
delete mode 100644
client-flink/flink-1.15-shaded/src/main/resources/META-INF/LICENSE
delete mode 100644
client-flink/flink-1.15-shaded/src/main/resources/META-INF/NOTICE
delete mode 100644 client-flink/flink-1.15/pom.xml
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleInputGateFactory.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartition.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionFactory.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleServiceFactory.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.15/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
delete mode 100644
client-flink/flink-1.15/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
delete mode 100644
client-flink/flink-1.15/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionFactorySuiteJ.java
delete mode 100644
client-flink/flink-1.15/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleResultPartitionSuiteJ.java
delete mode 100644
client-flink/flink-1.15/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleServiceFactorySuiteJ.java
delete mode 100644
client-flink/flink-1.15/src/test/java/org/apache/celeborn/plugin/flink/ShuffleResourceTrackerSuiteJ.java
delete mode 100644
client-flink/flink-1.16/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.16/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.16/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
delete mode 100644
client-flink/flink-1.17/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.17/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.17/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
delete mode 100644
client-flink/flink-1.18/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.18/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.18/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
delete mode 100644
client-flink/flink-1.19/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.19/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.19/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
delete mode 100644
client-flink/flink-1.20/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleEnvironment.java
delete mode 100644
client-flink/flink-1.20/src/main/java/org/apache/celeborn/plugin/flink/SimpleResultPartitionAdapter.java
delete mode 100644
client-flink/flink-1.20/src/main/java/org/apache/celeborn/plugin/flink/netty/NettyShuffleEnvironmentWrapper.java
rename client-spark/{spark-3 => spark-3-4}/pom.xml (96%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/BasedShuffleWriter.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/CelebornShuffleDataIO.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriter.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriter.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/scala/org/apache/spark/SparkVersionUtil.scala (100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/scala/org/apache/spark/celeborn/ExceptionMakerHelper.scala
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleHandle.scala
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala
(96%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/java/org/apache/spark/shuffle/celeborn/CelebornShuffleWriterSuiteBase.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriterSuiteJ.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/java/org/apache/spark/shuffle/celeborn/ShuffleManagerHook.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriterSuiteJ.java
(100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/java/org/apache/spark/shuffle/celeborn/TestCelebornShuffleManager.java
(100%)
rename client-spark/{spark-3 => spark-3-4}/src/test/resources/log4j.properties
(100%)
rename {client-flink/flink-1.14 =>
client-spark/spark-3-4}/src/test/resources/log4j2-test.xml (100%)
rename client-spark/{spark-3 =>
spark-3-4}/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleManagerSuite.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/pom.xml (95%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/java/org/apache/spark/shuffle/celeborn/ColumnarHashBasedShuffleWriter.java
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornColumnarShuffleReader.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnAccessor.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnBuilder.scala
(99%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnStats.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnType.scala
(99%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnarBatchBuilder.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnarBatchCodeGenBuild.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornColumnarBatchSerializer.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornCompressibleColumnAccessor.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornCompressibleColumnBuilder.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornCompressionScheme.scala
(100%)
copy client-spark/{spark-3.5-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornCompressionSchemes.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornNullableColumnAccessor.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/main/scala/org/apache/spark/sql/execution/columnar/CelebornNullableColumnBuilder.scala
(100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/test/java/org/apache/spark/shuffle/celeborn/ColumnarHashBasedShuffleWriterSuiteJ.java
(100%)
rename client-spark/{spark-3 =>
spark-4-columnar-shuffle}/src/test/resources/log4j2-test.xml (100%)
copy client-spark/{spark-3-columnar-shuffle =>
spark-4-columnar-shuffle}/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornColumnarShuffleReaderSuite.scala
(100%)
rename {client-flink/flink-1.15-shaded => client-spark/spark-4-shaded}/pom.xml
(92%)
copy client-spark/{spark-3-shaded =>
spark-4-shaded}/src/main/resources/META-INF/LICENSE (100%)
copy client-spark/{spark-2-shaded =>
spark-4-shaded}/src/main/resources/META-INF/NOTICE (100%)
rename {client-flink/flink-1.14-shaded =>
client-spark/spark-4-shaded}/src/main/resources/META-INF/licenses/LICENSE-protobuf.txt
(100%)
copy {client-mr/mr-shaded => client-tez/tez-shaded}/pom.xml (85%)
copy {client-mr/mr-shaded =>
client-tez/tez-shaded}/src/main/resources/META-INF/LICENSE (100%)
copy {client-mr/mr-shaded =>
client-tez/tez-shaded}/src/main/resources/META-INF/NOTICE (97%)
rename {client-flink/flink-1.15-shaded =>
client-tez/tez-shaded}/src/main/resources/META-INF/licenses/LICENSE-protobuf.txt
(100%)
copy {client-mr/mr-shaded =>
client-tez/tez-shaded}/src/main/resources/META-INF/licenses/LICENSE-zstd-jni.txt
(100%)
create mode 100644 cpp/celeborn/conf/BaseConf.cpp
create mode 100644 cpp/celeborn/conf/BaseConf.h
copy cpp/celeborn/{utils => conf}/CMakeLists.txt (85%)
create mode 100644 cpp/celeborn/conf/CelebornConf.cpp
create mode 100644 cpp/celeborn/conf/CelebornConf.h
create mode 100644 cpp/celeborn/conf/tests/BaseConfTest.cpp
copy cpp/celeborn/{utils => conf}/tests/CMakeLists.txt (81%)
create mode 100644 cpp/celeborn/conf/tests/CelebornConfTest.cpp
create mode 100644 cpp/celeborn/memory/ByteBuffer.cpp
create mode 100644 cpp/celeborn/memory/ByteBuffer.h
copy cpp/celeborn/{utils => memory}/CMakeLists.txt (83%)
create mode 100644 cpp/celeborn/memory/tests/ByteBufferTest.cpp
copy cpp/celeborn/{utils => memory}/tests/CMakeLists.txt (79%)
copy cpp/celeborn/{ => proto}/CMakeLists.txt (81%)
create mode 120000 cpp/celeborn/proto/TransportMessagesCpp.proto
copy cpp/celeborn/{utils => protocol}/CMakeLists.txt (82%)
create mode 100644 cpp/celeborn/protocol/PartitionLocation.cpp
create mode 100644 cpp/celeborn/protocol/PartitionLocation.h
create mode 100644 cpp/celeborn/protocol/StatusCode.h
create mode 100644 cpp/celeborn/protocol/TransportMessage.cpp
copy client/src/main/java/org/apache/celeborn/client/write/PushTask.java =>
cpp/celeborn/protocol/TransportMessage.h (55%)
copy cpp/celeborn/{utils => protocol}/tests/CMakeLists.txt (73%)
create mode 100644 cpp/celeborn/protocol/tests/PartitionLocationTest.cpp
create mode 100644 cpp/celeborn/protocol/tests/TransportMessageTest.cpp
copy cli/src/main/scala/org/apache/celeborn/cli/common/CliLogging.scala =>
cpp/celeborn/utils/CelebornUtils.h (55%)
delete mode 100644 dev/deps/dependencies-client-flink-1.14
delete mode 100644 dev/deps/dependencies-client-flink-1.15
create mode 100644 dev/deps/dependencies-client-tez
create mode 100644 docs/developers/java-columnar-shuffle.md
create mode 100644 docs/worker_tags.md
create mode 100644
master/src/test/scala/org/apache/celeborn/service/deploy/master/MasterClusterFeature.scala
copy
openapi/openapi-client/src/main/java/org/apache/celeborn/rest/v1/master/{DefaultApi.java
=> LoggerApi.java} (75%)
copy
openapi/openapi-client/src/main/java/org/apache/celeborn/rest/v1/model/{ConfigData.java
=> LoggerInfo.java} (76%)
copy
openapi/openapi-client/src/main/java/org/apache/celeborn/rest/v1/model/{ConfResponse.java
=> LoggerInfos.java} (69%)
copy
openapi/openapi-client/src/main/java/org/apache/celeborn/rest/v1/worker/{ShuffleApi.java
=> LoggerApi.java} (75%)
create mode 100644
service/src/main/scala/org/apache/celeborn/server/common/http/api/v1/LoggerResource.scala
copy tests/{mr-it => tez-it}/pom.xml (53%)
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/BroadcastAndOneToOneExample.java
copy cli/src/main/scala/org/apache/celeborn/cli/common/CliLogging.scala =>
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/BroadcastAndOneToOneTest.scala
(60%)
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezCartesianProductTest.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezHashJoinBroadcastTest.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezHashJoinTest.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezIntegrationTestBase.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezJoinIntegrationTestBase.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezOrderedWordCountTest.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezSortMergeJoinTest.scala
create mode 100644
tests/tez-it/src/test/java/org/apache/celeborn/tests/tez/TezWordCountTest.scala
create mode 100644 tests/tez-it/src/test/resources/log4j.properties
create mode 100644 tests/tez-it/src/test/resources/log4j2.xml
create mode 100644 tests/tez-it/src/test/resources/tez-site.xml
