MaxGekk commented on pull request #31571: URL: https://github.com/apache/spark/pull/31571#issuecomment-780017986
> Given that this was there since 2.4.0, I'm not sure why Apache Spark 3.2.0 does require this suddenly. FYI, I have found this config while documenting other "internal" configs, see https://github.com/apache/spark/pull/31564#pullrequestreview-590877832 > I'm a little reluctant to advertise this option more explicitly in Apache Spark side. It has been already explicitly advertised in public docs: http://spark.apache.org/docs/latest/sql-data-sources-avro.html#compatibility-with-databricks-spark-avro http://spark.apache.org/docs/latest/sql-data-sources-avro.html#configuration > This additional exposure proposal puts us into a more difficult situation when we want to remove this completely in the future from the Apache Spark codebase. Do you consider a situation when you can just remove this config w/o deprecation only because it is marked as `.internal()`? From my point of view, even the config is "internal" de jure, it is external de facto. In this situation, it cannot be just removed hiddenly from users. I do believe we should make it as external de jure, so, maybe remove it in the future only via deprecation otherwise we can break users apps potentially. @HyukjinKwon @cloud-fan WDYT? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
