XiangpengHao commented on PR #8887: URL: https://github.com/apache/arrow-rs/pull/8887#issuecomment-3555090970
This test case failed: https://github.com/apache/arrow-rs/blob/fea605cb16f7524cb69a197bfa581a1d4f5fe5d0/parquet-variant-compute/src/variant_get.rs#L3754-L3781 It is because the perfectly shredded array itself is not a valid arrow array. I'm not sure if this is a well-defined behavior, I didn't check carefully, but I feel it is `Time64MicrosecondArray`'s responsibility to make sure the its data is valid. What do you think @alamb @klion26 @friendlymatthew ? My gut feeling is that this data integrity checking during read time can be very expensive; for example, validating utf8 every time we read a string array can be extremely slow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
