rdblue commented on issue #26005: [SPARK-29163][SQL] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-548913585 Not all of v2 is in Java, but the parts that users extend are Java interfaces. Writing these in Scala is problematic because Scala might convert a trait into an interface or might make it an abstract class. Plus, Scala visibility is only enforced in Scala. You can see how this would be a problem for this PR. If the trait gets turned into a Java abstract class because it needs instance fields to store its `broadcastedHadoopConf`, then that forces Java callers to choose between extending a base class (like `FileDataSourceV2`) and using the broadcasted configuration feature of the API. This problem still exists in Java, which is why it's hard to write this. Since you need to use an interface and not an abstract class, you will probably need to build the broadcast variable and then configure the implementation with it at runtime.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
