gaborgsomogyi commented on pull request #31818: URL: https://github.com/apache/spark/pull/31818#issuecomment-798312942
@viirya you see it well and you're right, it's a tradeoff which should be discussed in-depth to come up with balanced decision. Pro: * Used don't need to guess how much time listeners will consume and mustn't make guesses to set `spark.python.authenticate.socketTimeout` properly. Con: * If `ServerSocket` can't bind to localhost then an event will be dropped which is behavior change. * `2.4.8` is the last `2.4` release so sudden movements are not suggested. Couple of heavy users complained about this but that said `spark.python.authenticate.socketTimeout` can be increased as workaround. If we decide not to merge this then we must mention in our doc that such cases the timeout must be increased to a best guess timeout (this is only true for `2.4` line, because on `3.x` it runs in separate thread). I've seen cases where this timeout was between 20 to 60 seconds and the complain was exactly this wide range. I see their point because configuring huge and sensitive computations in PROD based on best guesses freaks out the guys. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
