yabola commented on PR #39950: URL: https://github.com/apache/spark/pull/39950#issuecomment-1437794387
@sunchao Thank you for your reply! Yes, I also noticed this, just before is for minimal changes. In the original implementation: The first `footerFileMetaData` use `SKIP_ROW_GROUPS` option (`SkipMetadataFilter`, return meta without rowGroup); The second `footerFileMetaData` use `RangeMetadataFilter`(return meta with rowGroup info). Actually the second `footerFileMetaData` contains all information used in the first `footerFileMetaData`(the detail implementation difference can see `ParquetMetadataConverter#readParquetMetadata`) So when in case that we need filter pushdown and also `enableVectorizedReader`, we can only create one `ParquetFileReader` and read parquet footer only once. Other situations can also be optimized to read footer. This needs to modify some more codes, do you think it is suitable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
