felipepessoto opened a new issue, #55747: URL: https://github.com/apache/spark/issues/55747
According to some articles I found: - https://cloudsqale.com/2021/03/19/spark-reading-parquet-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/ - https://www.gresearch.com/news/parquet-files-know-your-scaling-limits/ Seems Spark can only parallelize the reads across row groups. Is this a known limitation? Is there any way to split it by row or page level? If a file has a single row group, it means all the tasks, except 1, would be idle? And that one task would read the entire file? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
