xCASx commented on issue #24732: [SPARK-27868][core] Better default value and documentation for socket server backlog. URL: https://github.com/apache/spark/pull/24732#issuecomment-573605559 This change, despite being > such a small change that it shouldn't affect anybody in fact broke our production configuration and led to almost a month of instability of our Spark workflows, while we were investigating the root cause (just because we had a totally different suspect and it took some time to realise that it wasn't a single contributing factor). In our set up we had the default JVM value overridden, and `spark.shuffle.io.backLog` left untouched. Of course with the change our set up got broken, because Sparks's new default of `64` got precedence and it is much less than we actually need. While we learned our own lesson out of this story, I'd like to emphasise, that one should not omit mentioning a change in release notes just because it is believed to be insignificant. Especially if you change names or values of default properties. [Release notes of Spark 2.4.4](https://spark.apache.org/releases/spark-release-2-4-4.html) has no even a hint of this change.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
