bhaskargurram-ai commented on PR #11549: URL: https://github.com/apache/arrow-rs/pull/11549#issuecomment-6549274129
Thanks @alamb for the review, and @adriangb for picking it up. I've pushed the change. ArrowSbbf now keeps the Arrow type alongside the physical type. It is either passed to ArrowSbbf::try_new(sbbf, column, Option<DataType>) or, if None, derived from the column with parquet_to_arrow_field, the same way PrimitiveArrayReader does it. The type is validated against the column once in try_new using the writer's arrow_to_parquet_type, and check then passes the values straight to the writer's visit_i32 / visit_i64 / visit_fixed_len_byte_array. That removes the second hand-written mapping and the per-array decimal checks. I also removed the type table from the docs.One special case is left: decimals use the column's own width, so files that store decimals in wider fixed-length arrays (for example from Impala) can still be checked. Since the constructor is now try_new and returns a Result, a mismatched type is reported up front instead of on the first check. Happy to adjust the naming or signature if you'd prefer something else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
