I was able to get it working with "hostnames". thanks! To dig deeper, how much does Tez obey the hints provided? How are Vertex Location Hints handled ? What if YARN is not able to provide containers in requested locations ?
Raajay On Thu, Sep 10, 2015 at 10:19 AM, Hitesh Shah <[email protected]> wrote: > In almost all cases, this is usually hostnames. The general flow is find > the block locations for the data source, extract the hostname from there > and provide it to YARN so that it can provide a container on the same host > as the datanode having the data. As long as YARN is using hostnames, the > container locality matching should work correctly. I will need to go and > check the YARN codebase to see if it does some additional reverse dns > lookups for IPs to also function correctly but to be safe, hostnames should > work. > > I don’t believe Tez has yet introduced support for working with > application-level YARN node labels. > > thanks > — Hitesh > > On Sep 10, 2015, at 12:43 AM, Raajay <[email protected]> wrote: > > > While creating TaskLocationHints, using the static function > > > > TaskLocationHint.createTaskLocationHint(Set<String> nodes, Set<string> > racks) > > > > what should the Strings be ? IP address of the nodes ? Node labels ? Or > hostnames ? > > > > Thanks > > Raajay > >
