ajantha-bhat commented on PR #7105: URL: https://github.com/apache/iceberg/pull/7105#issuecomment-1482319205
> I guess there is the same argument also for things like manifest list, today rolling up manifest list is a bottleneck for write operations, and some kind of design backed by a key-value store could solve that bottleneck. Maybe we should think about that and try to solve these cases together? Just like we have FileIO that works really well with object storage semantics, we can have something like VersionedListStore that works well with any mutable but versioned list. Yes. I totally agree that storing metadata in files can be a bottleneck (applies to stats as well). I think now catalogs are maturing enough to store metadata in the DB (REST, maybe Nessie in future). But the migration from one catalog to another needs special handling in these cases. We should separately discuss this and handle it together for all the metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
