xabriel commented on pull request #1783: URL: https://github.com/apache/iceberg/pull/1783#issuecomment-745059522
> > One thing I wanted to make explicit. The test TestSparkSchema3 has been removed as SupportsCatalogOptions doesn't allow to specify the schema. This introduces a breaking change between Spark2.4 and Spark3 based IcebergSource. > > Thanks @rymurr. I think this is the only remaining blocker. > > @prodeezy & @xabriel, do you specify the schema when reading Iceberg tables in Spark 3? I think the feature that this PR would disable was added by you, so I want to make sure you're okay with this potential change. Thank you for the ping @rdblue. All of our current code is based on Spark 2.4.X branch. We meant to pursue this further and to suggest a new API on the Spark 3 branch, but other work took precedence. The discussion is over at #590 (TL;DR: seems like we should have a `df.project(StructType)` API). So yes this change will break us once we move to Spark 3. However I want to discuss internally whether we feel strongly about it or not. Will get back to this thread ASAP. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
