On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <r...@databricks.com> wrote: > IIUC, the reason for that PR is that they found the string comparison to > increase the size in large shuffles. Maybe we should add the ability to > support the short name to Spark 1.6.2?
Is that something that really yields noticeable gains in performance? If it is, it seems like it would be simple to allow executors register with the full class name, and map the long names to short names in the shuffle service itself. You could even get fancy and have different ExecutorShuffleInfo implementations for each shuffle service, with an abstract "getBlockData" method that gets called instead of the current if/else in ExternalShuffleBlockResolver.java. -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
