alamb opened a new issue, #10255: URL: https://github.com/apache/arrow-rs/issues/10255
PR #10206 adds an unsafe skip-validation option to arrow-flight decoding and threads it through IPC record batch decoding. However, Flight dictionary messages still use arrow_ipc::reader::read_dictionary, which currently constructs a fresh UnsafeFlag::new() and therefore continues to validate dictionary payloads even when FlightDataDecoder::with_skip_validation has been used. This means dictionary-encoded Flight streams do not fully benefit from the skip-validation path: dictionary values may still pay validation costs, while later record batches use the skipped validation setting. Future work: - Extend the IPC decoder API so dictionary decoding can be driven through the same RecordBatchDecoder-style path as record batches, for example with .with_skip_validation(...).read_dictionary(...). - Keep the existing free read_dictionary helper as a compatibility wrapper if needed. - Update arrow-flight dictionary batch handling to pass the FlightDataDecoder skip-validation setting through to dictionary decoding. - Add focused coverage for a Flight stream with dictionary batches and skip_validation enabled. Context: discovered while reviewing #10206. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
