theweipeng commented on code in PR #1508: URL: https://github.com/apache/incubator-fury/pull/1508#discussion_r1564459928
########## docs/specification/xlang_serialization_spec.md: ########## @@ -592,6 +592,21 @@ format will be: `KV header` will be a header marked by `MapFieldInfo` in java. For languages such as golang, this can be computed in advance for non-interface types most times. +#### Why serialize chunk by chunk? + +When fury will use first key-value pair to predict header optimistically, it can't know how many pairs have same +meta(tracking kef ref, key has null and so on). If we don't write chunk by chunk with max chunk size, we must write at +least `X` bytes to take up a place for later to update the number which has same elements, `X` is the num_bytes for +encoding varint encoding of map size. + +And most map size are smaller than 255, if all pairs have same data, the chunk will be 1. This is common in golang/rust, +which object are not reference by default. + +Also, if only one or two keys have different meta, we can make it into a different chunk, so that most pairs can share Review Comment: Am I understanding this correctly as follows? If the type prediction fails, we might initiate a new chunk. This implies that when streaming mode is disabled, all elements within the same chunk are of the same type. In this scenario, both the 3rd bit and the 7rd bit in the header will be set to zero. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
