[ https://issues.apache.org/jira/browse/SPARK-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993008#comment-15993008 ]
Steve Loughran commented on SPARK-20560: ---------------------------------------- {{FileSystem.getFileBlockLocations(path)}} is only invoked from from {{HdfsUtils.getFileSegmentLocations}}, and used as a source of data for {{RDD.preferredLocations}} I don't see anything explicit through the code that detects & reacts to the FS call returning localhost; I'll do some test downstream to see what surfaces against S3. Unless the scheduler has some explicit "localhost -> anywhere" map, it might make sense for HdfsUtils.getFileSegmentLocation to downgrade "localhost" to None, on the basis that in a cluster FS, the data clearly doesn't know where it is. > Review Spark's handling of filesystems returning "localhost" in > getFileBlockLocations > ------------------------------------------------------------------------------------- > > Key: SPARK-20560 > URL: https://issues.apache.org/jira/browse/SPARK-20560 > Project: Spark > Issue Type: Bug > Components: Scheduler > Affects Versions: 2.1.0 > Reporter: Steve Loughran > Priority: Minor > > Some filesystems (S3a, Azure WASB) return "localhost" as the response to > {{FileSystem.getFileBlockLocations(path)}}. If this is then used as the > preferred host when scheduling work, there's a risk that work will be queued > on one host, rather than spread across the cluster. > HIVE-14060 and TEZ-3291 have both seen it in their schedulers. > I don't know if Spark does it, someone needs to look at the code, maybe write > some tests -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org