rickyma commented on PR #1775:
URL:
https://github.com/apache/incubator-uniffle/pull/1775#issuecomment-2160358545
Why are the default values set this way? Please read this.
Currently, the most commonly used grpc interface should be the pre-allocated
memory interface (requireBuffer), and its request object structure is as
follows:
```
message RequireBufferRequest {
int32 requireSize = 1;
string appId = 2;
int32 shuffleId = 3;
repeated int32
partitionIds = 4;
}
```
Now, we can know the entire Proto object's size estimation in the Java heap:
Object overhead: 16 bytes
requireSize field: 4 bytes
appId field: about 130 bytes
shuffleId field: 4 bytes
partitionIds field: 16 bytes (assuming an empty array)
Total size: 16 + 4 + 130 + 4 + 16 = 170 bytes.
So it can be understood that the size of **_the RequireBufferRequest object
is basically equivalent to the size of the partitionIds array_**, with each
int32 element occupying 4 bytes. Then we can know the approximate
correspondence between the array size and the RequireBufferRequest object size:
| Number of partitions | Occupied size |
| -------- | -------- |
| 100 | About0.4KB |
| 1,000 | About4KB |
| 10,000 | About40KB |
| 100,000 | About400KB |
| 1,000,000| About4MB |
In order to consider the versatility, our chunkSize is set to 32KB, which
can basically cover most of the normal number of partition scenarios.
Therefore, because the minimum pageSize of Netty is 4096, the pageSize can
be set to 4KB, and the maxOrder can be set to 3, because 4KB * 2^3 = 32KB,
which can get the chunkSize of 32KB. This setting is reasonable because the
smallest memory block is 4KB, which will not cause too much memory
fragmentation and can meet most memory requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
