gengliangwang commented on issue #23635: [SPARK-26709][SQL] OptimizeMetadataOnlyQuery does not handle empty records correctly URL: https://github.com/apache/spark/pull/23635#issuecomment-457137369 @maropu @hvanhovell I thought about it. One way is to check if the partition directory is empty. But since https://github.com/apache/spark/pull/20525, there will be a parquet/orc metadata-only file under after writing a empty Dataframe. In such case, the regression still exists. It seems that we can't avoid it without actually read the files under partitioned directories.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
