RussellSpitzer commented on issue #2527: URL: https://github.com/apache/iceberg/issues/2527#issuecomment-901586306
Iceberg doesn't implement SupportsRuntimeFiltering yet as added in that PR for Spark 3.2 (we actually don't support 3.2 at all yet) On Wed, Aug 18, 2021 at 9:06 PM vusal-iom ***@***.***> wrote: > I also hit the same problem while doing TPCDS 10TB benchmarking. The > reason was, only HadoopFsRelation had been taken account: > https://github.com/apache/spark/blob/fceabe2372ab2a53401059e6019f441d0580aeab/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala#L64 > > But, the latest code has V2 Datasource handling part already. See: > https://issues.apache.org/jira/browse/SPARK-35779 > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/issues/2527#issuecomment-901552057>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADE2YMJUHGQ6YF7O6JIPCLT5RRMHANCNFSM43U33RPQ> > . > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
