raghavyadav01 commented on issue #18297: URL: https://github.com/apache/iceberg/issues/18297#issuecomment-6375891975
One note on the open metrics question. Making this a primitive has a side effect that is easy to miss: - `MetricsConfig` decides eligibility with `type.isPrimitiveType() || type.isVariantType()` (`MetricsConfig.java:163`), so a new primitive is metrics-eligible by default, for the first 100 columns. - The default mode is `truncate(16)`, and truncation applies only to string and binary. Everything else is stored as-is. - So a `float[768]` column would write a full 768-float lower bound and upper bound into every manifest entry, roughly 6 KB per data file per vector column, in the metadata that scan planning reads. - The bounds would not buy much either. Lexicographic min/max over coordinates cannot prune a similarity search, which is the predicate these columns exist for. Might be worth the spec saying vector columns default to `none` rather than inheriting the default mode. If useful vector statistics turn up later, Puffin looks like the right home for them, next to the sketch work in #18516. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
