zeevm commented on code in PR #6649:
URL: https://github.com/apache/arrow-rs/pull/6649#discussion_r1822084410
##########
parquet/src/record/reader.rs:
##########
@@ -138,7 +138,15 @@ impl TreeBuilder {
.column_descr_ptr();
let col_reader = row_group_reader.get_column_reader(orig_index)?;
let column = TripletIter::new(col_descr, col_reader,
self.batch_size);
- Reader::PrimitiveReader(field, Box::new(column))
+ let reader = Reader::PrimitiveReader(field.clone(),
Box::new(column));
+ if repetition == Repetition::REPEATED && path.len() == 1 {
+ if curr_def_level != 1 || curr_rep_level != 1 {
+ return Err(ParquetError::General(format!("Top level
REPEATED primitve field {} should have definition and repetition levels of 1",
field.name())));
Review Comment:
This should handle these cases (from the
[format](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)
):
> A repeated field that is neither contained by a LIST- or MAP-annotated
group nor annotated by LIST or MAP should be interpreted as a required list of
required elements where the element type is the type of the field.
> Implementations should use either LIST and MAP annotations or unannotated
repeated fields, but not both. When using the annotations, no unannotated
repeated types are allowed.
IIUC we're talking about top level fields not contained in LIST groups, this
is a legacy schema for lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
