leixm commented on issue #926: URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754586337
I think this exception will occur when duplicate blocks are sent. Assume that there are two blocks with equal blockIds in a ShuffleBuffer. Before flush occurs, getMemoryShuffleData will always be called and never end. As you can see from the log below, the first read is 5050999 bytes, subsequent reads are all 4625271 bytes, and the reading never ends. There should be duplicate blockIds. ``` 23/10/10 03:25:44 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 1114 ms, get 5050999 bytes 23/10/10 03:25:44 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 236 ms, get 4625271 bytes 23/10/10 03:25:45 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 642 ms, get 4625271 bytes 23/10/10 03:25:46 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 815 ms, get 4625271 bytes 23/10/10 03:25:47 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 868 ms, get 4625271 bytes ...... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
