aokolnychyi commented on pull request #2240: URL: https://github.com/apache/iceberg/pull/2240#issuecomment-786855993
> For example, what do we do when the sort field is removed and there is a metrics config property? Yeah, that's the point we won't be able to differentiate if we modified the props. I am convinced now. If we have enough confidence that this feature will be useful, I am ok if we want to modify all places where we write. I took a look at how hard will it be to move to `Table` as that would simplify our life in the future. Since `BaseTable` uses a proxy for serializing, we only serialize `FileIO` and the metadata location. We will need to read the metadata file back on each executor. In addition, `FileIO` may contain a full Hadoop conf. That's why we broadcast `FileIO` and `EncryptionManager` in Spark. We could consider broadcasting `Table` instead of `FileIO` and `EncryptionManager`. However, we will still have to read the metadata file from each executor. Is that a big deal? I am fine either way. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
