HaoYang670 opened a new issue, #1770: URL: https://github.com/apache/arrow-rs/issues/1770
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** This is the follow-up of https://github.com/apache/arrow-rs/pull/1684#issuecomment-1123283421. **Describe the solution you'd like** ## How do we validate offsets now? 1. check if each offset can be converted to usize 2. check if **each** offset exceeds the `offset_limit` 3. check if offsets are monotonically increasing 4. validate each slot. ## How could we simplify this process? 1. check if the **last** offset exceeds the `offset_limit` 2. check if each offset can be converted to `usize` (actually, this could also be removed. But we need to convert the type of offset to `usize`, so reserve it) 3. check if offsets are monotonically increasing 4. validate each slot. ## What will be impacted? 1. We hope the validation could be somewhat faster (we need the benchmark result) 2. The Error message will change for some test cases. For example, given an offset buffer ``` [0, 1, 6, 3, 4, 5] ``` and `offset_limit == 5` We will get an error `"Offset invariant failure: offset at position 2 out of bounds: 6 > 5"` now, but `Offset invariant failure: non-monotonic offset at slot 2: 6 > 3` if we do the simplification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
