Hi Yufie, The original proposal did not seem to indicate that the metadata tables will be "materialized" (outside regular Iceberg metadata since most of those metadata tables are actually "views" on Iceberg metadata). However, in the last response, it seems metadata could potentially be written to designated tables/files. I think we need to resolve that question first (will metadata for those tables be separately materialized or just derived from existing Iceberg metadata?) before proceeding to the big metadata problem. Personally, I am not in favor of replicating this metadata (at least within Iceberg. See the other thread [1] for an "external system" argument).
One option is to leave the answer to this question outside the scope of this proposal, and keep the proposal for the APIs only. Implementations can decide how to store/compute this metadata. Also, regarding "HMS issues", I would suggest not to immediately reject any solution that has commonality with HMS: * HMS scaled very well at LinkedIn (and other companies) with some improvements. * The model itself (service + RDBMS) is still an accepted standard. Implementation details largely determine the scalability as opposed to the architecture itself. This is not to say that this is a recommended model, but just to say that "HMS similarity" alone should not influence the decision, and it is better to choose the solution that fits the use cases best without pre-rejecting an option. [1] https://lists.apache.org/thread/v4jf3djls06cq6q53lytbt9vj01l65kk Thanks, Walaa.
