Forward compatibility and roundtripping/rewriting parquet files

[Jörn Horstmann]

Hello everyone,

Let's assume I want to create a program that rewrites and existing 
parquet file, for example by adding key/value metadata or adding 
additional columns. The input file might use some new features, like 
encodings, logical types or column orders.

Encodings are simple thrift enums, in most implementations these are 
handled as a simple wrapper around their id and not as static enums. So 
as long as the corresponding column data is copied byte-identical, 
roundtripping should work.

Logical types are optional in the thrift structures, so new variants 
should be skipped by the reading side, and written as absent values. 
Consumers of a file might then have to cast the values, but could 
otherwise still process the file.

Column orders however are contained in a list. Readers commonly map 
unknown values to an internal "Unknown" variant which it is then not 
able to serialize to thrift again. When rewriting a file that contained 
an unknown column order, we would either have to skip the whole list, or 
write a potentially incorrect order for some columns. Missing column 
orders then probably make statistics in that file unusable.

I wonder how any existing parquet rewriters handle these problems. And 
Maybe it would make sense to define a canonical "UNKNOWN" ColumnOrder 
that could be written in such cases.

Cheers,
Jörn