I just wanted to bounce an idea off of everyone. One thing I notice is that 
there are certain bugs that show up when using the parquet-cli that don't show 
up when using it as an SDK in a Java program, even when reading the same files. 
There appears to be some duplicated code between the CLI and the rest of the 
SDK. One example I noticed is how there are changes that were made to detect 
UUIDs as a special case of a fixed length byte array, and while all the 
necessary changes are made in the SDK, some are missing from the duplicated 
code in the CLI. One thing we need to do is stop relying on the duplicated code 
in the CLI and have it exist ONLY as a thin wrapper around the SDK. And one way 
perhaps to force us to do that would be to maintain the CLI as a separate 
project. Of course, I haven't figured out what all these code inconsistencies 
are, so perhaps it'll turn out to be easy to just fix the CLI as it is, but the 
point is to adopt policies that make it harder to break some parts of ParquetMR 
when adding enhancements.


Reply via email to