liugs0213 opened a new issue, #11634: URL: https://github.com/apache/iceberg/issues/11634
### Feature Request / Improvement In large-scale feature datasets, adding new columns (or features) can be inefficient if the entire dataset needs to be rewritten. A more efficient method would involve associating only the new feature data with existing records, without rewriting the whole dataset. This could significantly optimize storage and improve iteration speeds, especially in machine learning and ETL pipelines. ## Key Considerations: 1. Storage Efficiency: Rather than rewriting the entire dataset when adding new columns, only the newly added columns should be written. This avoids duplication of existing data and minimizes storage consumption. 2. Efficient Query and Write Performance: Queries and writes should be optimized by linking new feature data to existing records without the need to reprocess or modify the entire dataset. This will maintain query performance while reducing unnecessary data movement. ## Current Approaches: Bytedance scheme: Whether the way of sorting according to the primary key is the best scheme. https://developer.volcengine.com/articles/7260058755952279606 ### Query engine Spark ### Willingness to contribute - [ ] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
