nevi-me commented on pull request #8402: URL: https://github.com/apache/arrow/pull/8402#issuecomment-706556497
Parquet's dictionary encoding is a complexity on its own. My understanding's that after a certain size, the dictionary no longer grows, but the additional values are stored the normal way. I'm still to spend more time on parquet-mr and the format. I think the approach of not forcing Arrow dictionaries to have Parquet dictionary encoding is good. > also only supports Int32 index types in this commit, also removes NULLs Do you want to work on other index types and supporting primitive Arrow dictionaries? We could keep this PR open for longer; as long as it's not blocking any additional unit of work. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
