PragmaTwice opened a new issue, #1033: URL: https://github.com/apache/incubator-kvrocks/issues/1033
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues. ### Motivation [The current encoding](https://kvrocks.apache.org/docs/Design/design-structure-on-rocksdb) of kvrocks has several notable problems: 1. The length of `size` field is 32 bits, which makes it impossible to store large data exceeding `4Gi`; 2. The length of `expire` field is 32 bits, which means that kvrocks has the Y2038 problem, i.e. kvrocks will lose availability in 2038. ### Solution Since currently the higher 4 bits of `flags` field are still unused, we will use one of them to distinguish between the old encoding and the new one, i.e. ``` 0 0 0 0 <4-bit redis-type> -> old encoding 1 0 0 0 <4-bit redis-type> -> new encoding ``` The new protocol uses varint encoding to represent `expire` and `size` field, and all other fields remain the same as before. In this way, the new protocol can be extended to 64bit (or even higher) to solve the above two problems without significantly increasing the space (actually shrinking, because for small number, 4 bytes are not needed to storage). In fact, I discussed this solution with @torwig in slack before, but only considered the `size` field and used fixed-length 64bit encoding. But now I think varint encoding might be a better fit. I do not know if the `version` field is suitable for varint encoding, so some advice might be needed. cc @git-hulk @torwig @ShooterIT @caipengbo ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
