zsxwing commented on pull request #33784: URL: https://github.com/apache/spark/pull/33784#issuecomment-902087268
> I'm under the impression that shuffle block fetching is in the hot-path, we have to block on fetches. The improvement doesn't sound noticeable since this is a RPC operation. > like if this is causing conflicts in practice It's hard to check this before the release goes out. But since [Caffeine](https://github.com/ben-manes/caffeine) is pretty popular (10.3k stars), the overlap of Spark and Caffeine users should not be small, and the chance to hit the conflict is not low. [SPARK-34309](https://issues.apache.org/jira/browse/SPARK-34309) also mentioned that > caffeine has been used in some open source projects like Cassandra, Hbase, Neo4j, Druid, Spring and so on. Not sure whether they use it in the client or the server (It would be great if someone can help confirm). If they use it in the client, it may be a problem when they use the connector for these library with Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
