So do we want to repurpose SPARK-30510 as an SQL config refactor? Alternatively, what’s the smallest step forward I can take to publicly document partitionOverwriteMode (which was my impetus for looking into this in the first place)?
2020년 1월 15일 (수) 오전 8:49, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > Resending to the dev list for archive purpose: > > I think automatically creating a configuration page isn't a bad idea > because I think we deprecate and remove configurations which are not > created via .internal() in SQLConf anyway. > > I already tried this automatic generation from the codes at SQL built-in > functions and I'm pretty sure we can do the similar thing for > configurations as well. > > We could perhaps mimic what hadoop does > https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml > > On Wed, 15 Jan 2020, 22:46 Hyukjin Kwon, <gurwls...@gmail.com> wrote: > >> I think automatically creating a configuration page isn't a bad idea >> because I think we deprecate and remove configurations which are not >> created via .internal() in SQLConf anyway. >> >> I already tried this automatic generation from the codes at SQL built-in >> functions and I'm pretty sure we can do the similar thing for >> configurations as well. >> >> We could perhaps mimic what hadoop does >> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml >> >> On Wed, 15 Jan 2020, 10:46 Sean Owen, <sro...@gmail.com> wrote: >> >>> Some of it is intentionally undocumented, as far as I know, as an >>> experimental option that may change, or legacy, or safety valve flag. >>> Certainly anything that's marked an internal conf. (That does raise >>> the question of who it's for, if you have to read source to find it.) >>> >>> I don't know if we need to overhaul the conf system, but there may >>> indeed be some confs that could legitimately be documented. I don't >>> know which. >>> >>> On Tue, Jan 14, 2020 at 7:32 PM Nicholas Chammas >>> <nicholas.cham...@gmail.com> wrote: >>> > >>> > I filed SPARK-30510 thinking that we had forgotten to document an >>> option, but it turns out that there's a whole bunch of stuff under >>> SQLConf.scala that has no public documentation under >>> http://spark.apache.org/docs. >>> > >>> > Would it be appropriate to somehow automatically generate a >>> documentation page from SQLConf.scala, as Hyukjin suggested on that ticket? >>> > >>> > Another thought that comes to mind is moving the config definitions >>> out of Scala and into a data format like YAML or JSON, and then sourcing >>> that both for SQLConf as well as for whatever documentation page we want to >>> generate. What do you think of that idea? >>> > >>> > Nick >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>>
