srowen commented on issue #26611: [SPARK-29939][CORE] Add spark.shuffle.mapStatus.compression.codec conf URL: https://github.com/apache/spark/pull/26611#issuecomment-558197515 It's an interesting philosophical question. If it's internal, why can users set it at all? I think it means something more like "developer API": use it if you know what you're doing, and it's subject to change without notice. It's not documented because, well, the audience is often reading the source code or consulting contributors directly for advanced, niche needs. These also often exist as safety-valve flags that are really not intended for use ever, unless something unforeseen goes wrong and we have to advertise a quick fix. So yeah, the semantics are a little fuzzy. Given that I generally prefer fewer configs anyway, anything we can do to keep this limited to the niche use cases that might need it probably helps avoid the info overload that is the Spark config page.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
