nevi-me commented on issue #1777: URL: https://github.com/apache/arrow-rs/issues/1777#issuecomment-1153271316
> One thing I stumbled across whilst reviewing the read side, is that the ColumnIndex has an implicit assumption that semantic records don't span page boundaries. That is the first repetition level of each page is 0. I think we may need to enforce this when writing file with a column index. https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L920 I think it should be considered a bug if part of a record is recorded over 2 page boundaries. It would create ambiguity on the record, and force a reader to potentially read the page in reverse to find the start of the record. Great catch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
