chaokunyang commented on code in PR #1508: URL: https://github.com/apache/incubator-fury/pull/1508#discussion_r1564569469
########## docs/specification/xlang_serialization_spec.md: ########## @@ -592,6 +592,21 @@ format will be: `KV header` will be a header marked by `MapFieldInfo` in java. For languages such as golang, this can be computed in advance for non-interface types most times. +#### Why serialize chunk by chunk? + +When fury will use first key-value pair to predict header optimistically, it can't know how many pairs have same +meta(tracking kef ref, key has null and so on). If we don't write chunk by chunk with max chunk size, we must write at +least `X` bytes to take up a place for later to update the number which has same elements, `X` is the num_bytes for +encoding varint encoding of map size. + +And most map size are smaller than 255, if all pairs have same data, the chunk will be 1. This is common in golang/rust, +which object are not reference by default. + +Also, if only one or two keys have different meta, we can make it into a different chunk, so that most pairs can share Review Comment: Elements in a chunk can be different types too. For example, we can profile a a mapand if types are different but no null values, we can still use one chunk nut encode types for every item -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
