maccamlc commented on pull request #798: URL: https://github.com/apache/parquet-mr/pull/798#issuecomment-649482253
> Thank you for creating the backward compatibility test for Map. It should have been existed already. > Unfortunately, this way you do not properly test backward compatibility. The problem is you cannot generate an "old" file with the "new" library. To be more precise the message parser is more for convenience and not used while reading/writing a parquet file. When you say you are testing converted type it is not really true because the parser tries to read logical types at the first place. Also the parquet writer writes both logical types and converted types so you cannot validate old files that have only converted types. > I would suggest adding tests that covers the examples in the [spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1) by creating the thrift generated format objects and convert them by `ParquetMetadataConverter` just like you did it in `TestParquetMetadataConverter.testMapLogicalType`. Maybe, these tests would fit better in that class as well. > I should have been described this before you've implemented this test. I am sorry about that. > > Please don't force push your changes because it makes harder to track the review. The committer will squash the PR before merging it anyway. > Thank you for creating the backward compatibility test for Map. It should have been existed already. > Unfortunately, this way you do not properly test backward compatibility. The problem is you cannot generate an "old" file with the "new" library. To be more precise the message parser is more for convenience and not used while reading/writing a parquet file. When you say you are testing converted type it is not really true because the parser tries to read logical types at the first place. Also the parquet writer writes both logical types and converted types so you cannot validate old files that have only converted types. > I would suggest adding tests that covers the examples in the [spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1) by creating the thrift generated format objects and convert them by `ParquetMetadataConverter` just like you did it in `TestParquetMetadataConverter.testMapLogicalType`. Maybe, these tests would fit better in that class as well. > I should have been described this before you've implemented this test. I am sorry about that. > > Please don't force push your changes because it makes harder to track the review. The committer will squash the PR before merging it anyway. Apologies for the force push. Good to know that squashed on commit. And thanks for the detailed reply. I think I got it this time :) Tests were moved into TestParquetMetadataConverter and for the old format test, building the metadata through Thrift SchemaElements. Regards, Matt ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
