This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/deploy/kubernetes/operator/golang.org/x/net-0.23.0
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
discard 6221acd00 build(deps): bump golang.org/x/net in
/deploy/kubernetes/operator
add 3bc0ee50d [#1662] fix(test): Fix Netty related flaky tests (#1663)
add 1516a539e [#1664] improvement(docs): Update the descriptions and
default values of outdated conf (#1665)
add 6f6d35aab [#378][FOLLOWUP] fix(server): Fix huge_partition_num metric
(#1669)
add 636d0be21 [#1341] fix(mr): Fix MR Combiner
ArrayIndexOutOfBoundsException Bug. (#1666)
add 89d2f1c8d [#1585] feat(server): Support app-level block size
statistics to report metrics (#1593)
add 65ccce543 [MINOR] chore(rust): disable flaky test of local_store_test
(#1674)
add 6d193278f [#1657] build: Add license information after version 0.9.0
(#1671)
add a1e0d56c8 [#1543] improvement(spark): Speed up reading when both sort
and combine are used. (#1640)
add 917456c5e [#1675] fix(test): Fix tests which may be flaky on different
machines (#1676)
add 43bd09c65 [#1682] feat(server): Introduce localfile isWritable metric
(#1683)
add 30bf8dcc4 [#1608][part-5] feat(spark3): always use the available
assignment (#1652)
add ae7283161 [#1678] fix(server): disk size leak on removing resources by
AppPurgeEvent (#1679)
add 40bd14b93 [#1684] fix(server): use the diskSize obtained from periodic
check to determine whether is writable (#1685)
add 8e66ef484 [#1680] improvement(server): Remove partial HDFS files that
written by server self for expired apps (#1681)
add 7b631773e [#1686] feat(netty): Support pending tasks number metrics
for Netty EventLoopGroup (#1687)
add 313d4e02e [#1538] feat(spark): report blockIds to spark driver
optionally (#1677)
add 8e26a34e0 [MINOR] docs: modify the default value of
`rss.coordinator.select.partition.strategy` in docs (#1692)
add d4a5fb753 [MINOR] fix(typo): Correct the removeShuffle method name
(#1697)
add dc022d684 [#1594] feat(client):support generating larger block size
during shuffle map task by spill partial partitions data (#1670)
add 2311682fd [#1701] improvement(server): Bump gRPC from 1.61.1 to 1.63.0
(#1704)
add a3461fb1e [#1701] improvement(Netty): Bump Netty from 4.1.106.Final to
4.1.109.Final (#1702)
add bc46d4a12 [#1608][part-6] improvement(spark): verify the sent blocks
count (#1690)
add 4f4f7e39e [#1675][FOLLOWUP] fix(test): Fix flaky tests which may cause
port conflicts (#1696)
add de4b26112 [#1608][part-7] improvement(doc): add doc and optimize
reassign config options (#1693)
add 93f43473a [#1709] feat(coordinator): Introduce pluggable
`ClientConfApplyStrategy` for `fetchClientConf` rpc (#1710)
add 2085e425d [#1699] improvement: Upgrade from
commons-collections:commons-collections:3.2.2 to
org.apache.commons:commons-collections:4.4 (#1700)
add 3cc052bc2 [#1673] fix(K8S): Fix the deployment of stable version K8S
cluster (#1694)
add c2aecf533 build(deps): bump golang.org/x/net in
/deploy/kubernetes/operator
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6221acd00)
\
N -- N -- N
refs/heads/dependabot/go_modules/deploy/kubernetes/operator/golang.org/x/net-0.23.0
(c2aecf533)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/build.yml | 2 +-
LICENSE-binary | 14 +
.../org/apache/hadoop/mapred/SortWriteBuffer.java | 45 +-
.../hadoop/mapred/SortWriteBufferManagerTest.java | 7 +-
.../apache/hadoop/mapred/SortWriteBufferTest.java | 53 +
.../org/apache/spark/shuffle/RssShuffleHandle.java | 7 +-
.../org/apache/spark/shuffle/RssSparkConfig.java | 26 +
.../apache/spark/shuffle/RssSparkShuffleUtils.java | 15 +-
.../apache/spark/shuffle/ShuffleHandleInfo.java | 250 ---
.../spark/shuffle/ShuffleHandleInfoManager.java | 36 +-
.../shuffle/handle/MutableShuffleHandleInfo.java | 268 +++
.../spark/shuffle/handle/ShuffleHandleInfo.java | 50 +
.../shuffle/handle/ShuffleHandleInfoBase.java | 29 +-
.../shuffle/handle/SimpleShuffleHandleInfo.java | 75 +
.../spark/shuffle/writer/BufferManagerOptions.java | 9 +
.../shuffle/writer/TaskAttemptAssignment.java | 52 +
.../spark/shuffle/writer/WriteBufferManager.java | 93 +-
.../org/apache/uniffle/shuffle/BlockIdManager.java | 75 +
.../BlockIdSelfManagedShuffleWriteClient.java | 106 +
.../uniffle/shuffle/RssShuffleClientFactory.java | 72 +
.../shuffle/manager/RssShuffleManagerBase.java | 36 +-
.../manager/RssShuffleManagerInterface.java | 15 +-
.../shuffle/manager/ShuffleManagerGrpcService.java | 184 +-
.../manager/ShuffleManagerServerFactory.java | 2 +-
.../MutableShuffleHandleInfoTest.java} | 74 +-
.../shuffle/reader/RssShuffleDataIteratorTest.java | 2 +
.../shuffle/writer/WriteBufferManagerTest.java | 52 +-
.../shuffle/manager/DummyRssShuffleManager.java | 21 +-
.../manager/ShuffleManagerServerFactoryTest.java | 25 +-
.../apache/spark/shuffle/RssShuffleManager.java | 123 +-
.../spark/shuffle/reader/RssShuffleReader.java | 60 +-
.../spark/shuffle/writer/RssShuffleWriter.java | 30 +-
.../spark/shuffle/writer/RssShuffleWriterTest.java | 8 +-
.../apache/spark/shuffle/RssShuffleManager.java | 318 ++-
.../spark/shuffle/reader/RssShuffleReader.java | 53 +-
.../spark/shuffle/writer/RssShuffleWriter.java | 229 +-
.../spark/shuffle/RssShuffleManagerTest.java | 1 +
.../spark/shuffle/writer/RssShuffleWriterTest.java | 282 ++-
.../tez/common/ShuffleAssignmentsInfoWritable.java | 4 +-
.../PartitionDataReplicaRequirementTracking.java | 22 +
.../client/factory/ShuffleClientFactory.java | 62 +-
.../uniffle/client/impl/ShuffleReadClientImpl.java | 9 +-
.../client/impl/ShuffleWriteClientImpl.java | 2 +-
.../client/impl/ShuffleReadClientImplTest.java | 2 +
common/pom.xml | 4 +
.../uniffle/common/ReceivingFailureServer.java | 87 +
.../apache/uniffle/common/config/ConfigUtils.java | 11 +
.../apache/uniffle/common/config/RssBaseConf.java | 3 +-
.../uniffle/common/config/RssClientConf.java | 7 +-
.../uniffle/common/metrics/MetricsManager.java | 4 +
.../uniffle/common/metrics/NettyMetrics.java | 18 +
.../org/apache/uniffle/common/util/Constants.java | 1 +
.../org/apache/uniffle/common/util/RssUtils.java | 8 +
coordinator/pom.xml | 4 +
.../apache/uniffle/coordinator/AccessManager.java | 2 +-
.../uniffle/coordinator/ApplicationManager.java | 2 +-
.../uniffle/coordinator/CoordinatorConf.java | 7 +
.../coordinator/CoordinatorGrpcService.java | 22 +-
.../uniffle/coordinator/CoordinatorServer.java | 17 +-
...ava => AbstractRssClientConfApplyStrategy.java} | 14 +-
....java => BypassRssClientConfApplyStrategy.java} | 16 +-
.../conf/RssClientConfApplyManager.java | 62 +
.../coordinator/conf/RssClientConfFetchInfo.java | 48 +-
.../coordinator/web/resource/ServerResource.java | 2 +-
.../coordinator/SimpleClusterManagerTest.java | 2 +-
.../conf/RssClientConfApplyManagerTest.java | 113 +
deploy/kubernetes/operator/Makefile | 2 +-
.../api/uniffle/v1alpha1/groupversion_info.go | 4 +-
.../uniffle/v1alpha1/remoteshuffleservice_types.go | 2 +-
.../api/uniffle/v1alpha1/zz_generated.deepcopy.go | 6 +-
.../uniffle.apache.org_remoteshuffleservices.yaml | 2277 +++++++++++---------
.../operator/examples/configuration.yaml | 47 +-
deploy/kubernetes/operator/go.mod | 57 +-
deploy/kubernetes/operator/go.sum | 342 ++-
.../kubernetes/operator/pkg/constants/constants.go | 2 +-
.../operator/pkg/controller/controller/rss_test.go | 4 +-
.../pkg/controller/sync/coordinator/coordinator.go | 3 +-
.../sync/coordinator/coordinator_test.go | 3 +-
.../controller/sync/shuffleserver/shuffleserver.go | 2 +-
.../sync/shuffleserver/shuffleserver_test.go | 2 +-
.../operator/pkg/controller/util/kubernetes/hpa.go | 10 +-
.../pkg/controller/util/kubernetes/hpa_test.go | 14 +-
.../operator/pkg/controller/util/util.go | 2 +-
.../pkg/generated/clientset/versioned/clientset.go | 38 +-
.../generated/clientset/versioned/fake/register.go | 14 +-
.../clientset/versioned/scheme/register.go | 14 +-
.../v1alpha1/fake/fake_remoteshuffleservice.go | 2 +-
.../typed/uniffle/v1alpha1/uniffle_client.go | 20 +-
.../operator/pkg/webhook/inspector/inspector.go | 6 +-
docs/client_guide/client_guide.md | 56 +-
docs/client_guide/spark_client_guide.md | 25 +-
docs/coordinator_guide.md | 2 +-
docs/operator/install.md | 4 +-
docs/server_guide.md | 114 +-
.../uniffle/test/DiskErrorToleranceTest.java | 6 +-
.../test/HybridStorageFaultToleranceBase.java | 22 +-
.../test/HybridStorageHadoopFallbackTest.java | 6 -
.../test/HybridStorageLocalFileFallbackTest.java | 6 -
.../java/org/apache/uniffle/test/QuorumTest.java | 1 +
.../apache/uniffle/test/RpcClientRetryTest.java | 1 +
.../java/org/apache/uniffle/test/ServletTest.java | 14 +-
.../ShuffleServerConcurrentWriteOfHadoopTest.java | 2 +
.../test/ShuffleServerFaultToleranceTest.java | 6 +-
.../apache/uniffle/test/ShuffleServerGrpcTest.java | 6 -
.../uniffle/test/ShuffleServerWithHadoopTest.java | 18 +-
.../ShuffleServerWithKerberizedHadoopTest.java | 23 +-
.../ShuffleServerWithLocalOfExceptionTest.java | 1 -
.../ShuffleServerWithLocalOfLocalOrderTest.java | 6 -
.../uniffle/test/ShuffleServerWithLocalTest.java | 6 -
.../test/ShuffleServerWithMemLocalHadoopTest.java | 15 +-
.../uniffle/test/ShuffleServerWithMemoryTest.java | 6 -
.../uniffle/test/ShuffleWithRssClientTest.java | 2 +
.../org/apache/uniffle/test/DynamicConfTest.java | 1 +
.../test/RepartitionWithLocalFileRssTest.java | 31 +-
.../uniffle/test/RepartitionWithMemoryRssTest.java | 2 +-
.../apache/uniffle/test/RssShuffleManagerTest.java | 10 +-
.../uniffle/test/SparkClientWithLocalTest.java | 6 +-
.../uniffle/test/SparkIntegrationTestBase.java | 40 +-
.../ContinuousSelectPartitionStrategyTest.java | 22 +-
.../test/GetShuffleReportForMultiPartTest.java | 11 +-
...va => PartitionBlockDataReassignBasicTest.java} | 50 +-
... PartitionBlockDataReassignMultiTimesTest.java} | 85 +-
.../uniffle/client/api/ShuffleManagerClient.java | 20 +-
.../client/impl/grpc/CoordinatorGrpcClient.java | 2 +-
.../client/impl/grpc/ShuffleManagerGrpcClient.java | 45 +-
.../client/impl/grpc/ShuffleServerGrpcClient.java | 23 +-
.../impl/grpc/ShuffleServerGrpcNettyClient.java | 19 +-
.../client/request/RssFetchClientConfRequest.java | 33 +
.../RssGetShuffleResultForMultiPartRequest.java | 18 +
.../client/request/RssGetShuffleResultRequest.java | 18 +
.../RssReassignFaultyShuffleServerRequest.java | 57 -
.../RssReassignOnBlockSendFailureRequest.java | 48 +
.../request/RssReportShuffleResultRequest.java | 29 +
.../response/RssGetShuffleResultResponse.java | 24 +
.../RssPartitionToShuffleServerResponse.java | 8 +-
.../RssReassignFaultyShuffleServerResponse.java | 49 -
... => RssReassignOnBlockSendFailureResponse.java} | 24 +-
.../response/RssReportShuffleResultResponse.java | 6 +
licenses-binary/LICENSE-rimraf.txt | 15 +
pom.xml | 11 +-
proto/src/main/proto/Rss.proto | 42 +-
rust/experimental/server/src/store/localfile.rs | 4 +-
server/pom.xml | 5 +-
.../org/apache/uniffle/server/HealthCheck.java | 2 +-
.../apache/uniffle/server/LocalStorageChecker.java | 37 +-
.../apache/uniffle/server/RegisterHeartBeat.java | 2 +-
.../apache/uniffle/server/ShuffleFlushManager.java | 2 +-
.../org/apache/uniffle/server/ShuffleServer.java | 21 +-
.../apache/uniffle/server/ShuffleServerConf.java | 21 +
.../uniffle/server/ShuffleServerMetrics.java | 32 +-
.../org/apache/uniffle/server/ShuffleTaskInfo.java | 4 +
.../apache/uniffle/server/ShuffleTaskManager.java | 12 +-
.../server/buffer/ShuffleBufferManager.java | 15 +
.../apache/uniffle/server/event/AppPurgeEvent.java | 14 +-
.../server/netty/ShuffleServerNettyHandler.java | 2 +-
.../apache/uniffle/server/netty/StreamServer.java | 45 +
.../server/storage/HadoopStorageManager.java | 8 +-
.../server/storage/LocalStorageManager.java | 4 +-
.../server/MockedShuffleServerGrpcService.java | 35 +
.../uniffle/server/ShuffleFlushManagerTest.java | 5 +
.../uniffle/server/ShuffleServerMetricsTest.java | 2 +-
.../apache/uniffle/server/ShuffleServerTest.java | 1 +
.../uniffle/server/ShuffleTaskManagerTest.java | 85 +-
.../TopNShuffleDataSizeOfAppCalcTaskTest.java | 6 -
.../server/buffer/ShuffleBufferManagerTest.java | 44 +
.../server/storage/HadoopStorageManagerTest.java | 87 +
.../server/storage/LocalStorageManagerTest.java | 42 +-
.../uniffle/storage/common/LocalStorage.java | 35 +-
.../uniffle/storage/common/LocalStorageMeta.java | 2 +-
.../storage/factory/ShuffleHandlerFactory.java | 4 +-
.../handler/impl/HadoopShuffleDeleteHandler.java | 31 +-
.../request/CreateShuffleDeleteHandlerRequest.java | 11 +
.../uniffle/storage/common/LocalStorageTest.java | 24 +-
173 files changed, 5442 insertions(+), 2526 deletions(-)
delete mode 100644
client-spark/common/src/main/java/org/apache/spark/shuffle/ShuffleHandleInfo.java
copy
common/src/main/java/org/apache/uniffle/common/security/NoOpSecurityContext.java
=>
client-spark/common/src/main/java/org/apache/spark/shuffle/ShuffleHandleInfoManager.java
(50%)
create mode 100644
client-spark/common/src/main/java/org/apache/spark/shuffle/handle/MutableShuffleHandleInfo.java
create mode 100644
client-spark/common/src/main/java/org/apache/spark/shuffle/handle/ShuffleHandleInfo.java
copy
storage/src/main/java/org/apache/uniffle/storage/common/StorageReadMetrics.java
=>
client-spark/common/src/main/java/org/apache/spark/shuffle/handle/ShuffleHandleInfoBase.java
(65%)
create mode 100644
client-spark/common/src/main/java/org/apache/spark/shuffle/handle/SimpleShuffleHandleInfo.java
create mode 100644
client-spark/common/src/main/java/org/apache/spark/shuffle/writer/TaskAttemptAssignment.java
create mode 100644
client-spark/common/src/main/java/org/apache/uniffle/shuffle/BlockIdManager.java
create mode 100644
client-spark/common/src/main/java/org/apache/uniffle/shuffle/BlockIdSelfManagedShuffleWriteClient.java
create mode 100644
client-spark/common/src/main/java/org/apache/uniffle/shuffle/RssShuffleClientFactory.java
rename
client-spark/common/src/test/java/org/apache/spark/shuffle/{ShuffleHandleInfoTest.java
=> handle/MutableShuffleHandleInfoTest.java} (59%)
create mode 100644
common/src/main/java/org/apache/uniffle/common/ReceivingFailureServer.java
copy
coordinator/src/main/java/org/apache/uniffle/coordinator/conf/{ClientConfParser.java
=> AbstractRssClientConfApplyStrategy.java} (68%)
copy
coordinator/src/main/java/org/apache/uniffle/coordinator/conf/{ClientConfParser.java
=> BypassRssClientConfApplyStrategy.java} (68%)
create mode 100644
coordinator/src/main/java/org/apache/uniffle/coordinator/conf/RssClientConfApplyManager.java
copy server/src/main/java/org/apache/uniffle/server/event/PurgeEvent.java =>
coordinator/src/main/java/org/apache/uniffle/coordinator/conf/RssClientConfFetchInfo.java
(52%)
create mode 100644
coordinator/src/test/java/org/apache/uniffle/coordinator/conf/RssClientConfApplyManagerTest.java
copy
integration-test/spark3/src/test/java/org/apache/uniffle/test/{PartitionBlockDataReassignTest.java
=> PartitionBlockDataReassignBasicTest.java} (69%)
rename
integration-test/spark3/src/test/java/org/apache/uniffle/test/{PartitionBlockDataReassignTest.java
=> PartitionBlockDataReassignMultiTimesTest.java} (53%)
delete mode 100644
internal-client/src/main/java/org/apache/uniffle/client/request/RssReassignFaultyShuffleServerRequest.java
create mode 100644
internal-client/src/main/java/org/apache/uniffle/client/request/RssReassignOnBlockSendFailureRequest.java
delete mode 100644
internal-client/src/main/java/org/apache/uniffle/client/response/RssReassignFaultyShuffleServerResponse.java
copy
internal-client/src/main/java/org/apache/uniffle/client/response/{RssReassignServersReponse.java
=> RssReassignOnBlockSendFailureResponse.java} (60%)
create mode 100644 licenses-binary/LICENSE-rimraf.txt
