maccamlc commented on pull request #798: URL: https://github.com/apache/parquet-mr/pull/798#issuecomment-650527774
@gszadovszky before this gets merged, I just wanted to clarify something myself after looking more into the format spec, that might tidy this issue up further. * Is MAP_KEY_VALUE required to still be written as the Converted Type when creating new files? From what I could see from some older issues and the [backwards-compatibility rules](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1) it seems to have always been an optional type, and also used incorrectly in the past. It appears that older versions of Parquet would be able to read the Map type in the schema without MAP_KEY_VALUE. If that is true, I would probably suggest pushing this [additional commit](https://github.com/maccamlc/parquet-mr/commit/3f774d123997a4c63631185ca409550ca03b960d) that I tested, onto this PR. It would mean that any unexpected uses of LogicalType.MAP_KEY_VALUE would result in UNKNOWN being written to the file. But it is removed from the ConversionPatterns path, meaning that my case of this occuring when converting an Avro schema is still fixed, and tested. Let me know if believe this might be the preferred fix, or if what have already done is better. Thanks Matt ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
