If that's the case (and according to the Format documentation it is) then we are doing it incorrectly in C++. How depressing
https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_writer.cc#L1097 This is unfortunately what happens when you don't have more rigorous integration tests. On Fri, May 22, 2020 at 3:14 AM Gabor Szadovszky <[email protected]> wrote: > > Hi ZJ, > > parquet-mr clearly writes repetition levels and definition levels according > to the specification. See the following code references. > For V1 pages: > https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnWriterV1.java#L60 > For V2 pages: > https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L655 > https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L221-L225 > > Regards, > Gabor > > > On Fri, May 22, 2020 at 6:35 AM Zhuo Jia Dai <[email protected]> wrote: > > > I raise this issue https://github.com/JuliaIO/Parquet.jl/issues/60 > > > > where the official parquet documentation states that repetition levels are > > written before definition levels. However, in the Julia Parquet package the > > parquet implementation reads definition before the repetition levels and > > the author insists on him being right but did not provide further evidence. > > > > I wanted to double-check this with the parquet dev community? Is it true > > that definitions levels need to be written before repetition levels? If > > true then the parquet documentation is wrong then I am happy to PR a fix. > > > > Regards > > -- > > ZJ > > > > [email protected] > >
