dingshun3016 opened a new issue, #2163:
URL: https://github.com/apache/incubator-uniffle/issues/2163

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the 
[issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Describe the bug
   
   The rss client was stuck when reading data in the memory of shufle server. 
From the container log, it can be seen that the task kept reading the block id 
`17595256313552` repeatedly
   
   `24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 26 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 10 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 15 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 11 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 26 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 29 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 28 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 32 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 13 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 17 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 8 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 9 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 6 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 28 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 21 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 17 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 54 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 38 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 13 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 6 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 23 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 40 ms
   24/10/09 04:54:29 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 21 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 26 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 4 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 7 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 56 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 64 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 4 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 50 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 9 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 20 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 16 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 59 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 18 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 37 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 7 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 6 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 16 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 6 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 5 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 7 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 19 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 24 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 8 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 24 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 33 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 69 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 58 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 63 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 17 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 15 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 6 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 35 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 23 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 12 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 25 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 17 ms
   24/10/09 04:54:30 INFO [Executor task launch worker for task 22297] 
ShuffleServerGrpcNettyClient: GetInMemoryShuffleData from x.x.x.x:49999 for 
appId[application_1726223638693_2862755_897411780-5594985], shuffleId[1], 
partitionId[366], lastBlockId[17595256313552] cost 27 ms`
   
   ### Affects Version(s)
   
   master
   
   ### Uniffle Server Log Output
   
   _No response_
   
   ### Uniffle Engine Log Output
   
   _No response_
   
   ### Uniffle Server Configurations
   
   _No response_
   
   ### Uniffle Engine Configurations
   
   _No response_
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to