qlong commented on PR #16715: URL: https://github.com/apache/iceberg/pull/16715#issuecomment-4987809433
@steveloughran , I did not see performance related fix for variant in [Spark 4.2](https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%204.2.0%20and%20text%20~%20%22variant%22). Pending [Spark 4.3](https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%204.3.0%20and%20text%20~%20%22variant%22) has a few performance fix for variant, including variant filter pushdown (pre-requite for your row group skipping on iceberg side), and extraction pushdown enhancement. In short, file skipping/rowgroup skipping based on filter pushdown won't work until 4.3 is released. Extraction pushdown can work today in simple cases, will automatically get performance enhancement from Spark 4.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
