Although the python Parquet api is a wrapper of c++, there are some tuning knobs not included in python. For example, dictionary_pagesize_limit_. The dictionary page size will easily exceed the limit when any or many of the following happen: 1. The row_group_size is relatively large e.g. the default is 64M. 2. The size per entry is large e.g large string column 3. the repeatability of data is not so high. This may result in the dictionary encoding not being fully utilized if this parameter cannot be tuned. In C++, however, this parameter can be tuned to the optimized setting.
There are also other parameters not exposed in python, for example, max_statistics_size. There are also some unalignment between C++ and Python. For example, the parameter "chunk_size" in C++ WriteTable function is actually row group size. Calling it "chunk_size" without any explanation in the C++ doc is really confusing. In the Cython code it actually passes "row_group_size" to "chunk_size". Thanks in advance, Shawn Zeng
