It's possible, but pretty unlikely to have an exact namespace collision. It's probably a best practice to clearly separates settings, etc that are downstream add-ons into a separate namespace, and I don't mind a sentence in a doc somewhere suggesting a convention, but I really think it's up to downstream projects to decide how they want to play it. If Spark did add a feature with this exact key, that'd be up to downstream projects to rationalize.
On Fri, Aug 30, 2019 at 9:36 AM Nicholas Chammas <nicholas.cham...@gmail.com> wrote: > > I discovered today that EMR provides its own optimizations for Spark. Some of > these optimizations are controlled by configuration settings with names like > `spark.sql.dynamicPartitionPruning.enabled` or > `spark.sql.optimizer.flattenScalarSubqueriesWithAggregates.enabled`. As far > as I can tell, these are EMR-specific configurations. > > Does this create a potential problem, since it's possible that future Apache > Spark configuration settings may end up colliding with these names selected > by EMR? > > Should we document some sort of third-party configuration namespace pattern > and encourage third parties to scope their custom configurations to that > area? e.g. Something like `spark.external.[vendor].[whatever]`. > > Nick --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
