steFaiz opened a new issue, #6734: URL: https://github.com/apache/paimon/issues/6734
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation The SST file format is used in the following scenarios: 1. As the local storage format for LookupFile. 2. In the B-tree global index. 3. When query performance is a higher priority—for example, when used as a Flink dimension table—the data can be directly stored in SST format. ### Solution Based on current LookupStore implementation, we can introduce a generalized SST FileFormat which can store any binary key-value pairs. The file format is expected to provide below functionality： 1. fast lookup 2. fast range query 3. fast random access 4. efficient full-file scan ### Anything else? Please refer to [pip-25]( https://cwiki.apache.org/confluence/display/PAIMON/PIP-25%3A+Introduce+a+key-value+file+format+for+paimon+primary+key+table) as the related proposal and refer to [PR](https://github.com/apache/paimon/pull/6702) as current draft implementation. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
