fx19880617 edited a comment on issue #6729: URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810468145
This is to have the tooling to estimate upfront size, just estimate. E.g. ```pinot-admin.sh EstimateTableSize -schema mySchema.json -tableConf myTable.json -columnStats xxxx``` @deemoliu E.g. I have a composite primary key with two string columns, estimated size is 24 bytes and 40 bytes, the unique combination is 16 million. Then for the Primary keymap size estimation(Per https://sourcegraph.com/github.com/apache/incubator-pinot@89a22f097c5ff26396e58950c90d764066a56121/-/blob/pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java#L72): - For keys: keySpace is (24+40)Bytes* 16 Million = 1GB - For values: we need to store RecordLocation, it contains <segmentName, DocId, Timestamp>, so maybe another <48+ 4 + 8>=64 bytes on it. It's another 1GB. Then we can estimate that the total memory overhead is about 2GB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
