somelovelanguage commented on code in PR #1510:
URL:
https://github.com/apache/incubator-uniffle/pull/1510#discussion_r1479564906
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java:
##########
@@ -99,6 +102,37 @@ public ShuffleServerGrpcService(ShuffleServer
shuffleServer) {
this.shuffleServer = shuffleServer;
}
+ @Override
+ public void unregisterApp(
+ RssProtos.AppUnregisterRequest request,
+ StreamObserver<RssProtos.AppUnregisterResponse> responseStreamObserver) {
+ String appId = request.getAppId();
+
+ StatusCode result = StatusCode.SUCCESS;
+ String responseMessage = "OK";
+ try {
+ Map<Integer, RangeMap<Integer, ShuffleBuffer>> shuffleIdToBuffers =
+ shuffleServer.getShuffleBufferManager().getBufferPool().get(appId);
+ if (shuffleIdToBuffers != null) {
+ HashSet<Integer> shuffleIds =
Sets.newHashSet(shuffleIdToBuffers.keySet());
+ shuffleIds.forEach(
+ shuffleId ->
+
shuffleServer.getShuffleTaskManager().removeShuffleDataAsync(appId, shuffleId));
Review Comment:
I tried to do this at first, but I found that when processing AppPurgeEvent,
it will first determine whether the App has expired.
```java
// the thread for clear expired resources
// 1
Runnable clearResourceRunnable =
() -> {
while (true) {
PurgeEvent event = null;
try {
event = expiredAppIdQueue.take();
long startTime = System.currentTimeMillis();
if (event instanceof AppPurgeEvent) {
removeResources(event.getAppId(), true);
double usedTime =
(System.currentTimeMillis() - startTime) /
Constants.MILLION_SECONDS_PER_SECOND;
ShuffleServerMetrics.summaryTotalRemoveResourceTime.observe(usedTime);
}
// 2
public void removeResources(String appId, boolean checkAppExpired) {
Lock lock = getAppLock(appId);
try {
lock.lock();
LOG.info("Start remove resource for appId[" + appId + "]");
if (checkAppExpired && !isAppExpired(appId)) {
LOG.info(
"It seems that this appId[{}] has registered a new shuffle, just
ignore this AppPurgeEvent event.",
appId);
return;
}
// 3
private boolean isAppExpired(String appId) {
if (shuffleTaskInfos.get(appId) == null) {
return true;
}
return System.currentTimeMillis() -
shuffleTaskInfos.get(appId).getCurrentTimes()
> appExpiredWithoutHB;
}
```
According to my understanding, in this case, the App is not necessarily
expired, so it may not be able to play the role of unregister shuffle. If my
understanding is wrong, please point me out.
```java
public void removeShuffleDataAsync(String appId) {
shuffleTaskInfos.remove(appId);
expiredAppIdQueue.add(
new AppPurgeEvent(appId, getUserByAppId(appId));
}
```
Maybe we can do this(remove the appId first and then do what you said) ,
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
