Re: Container settings at vertex level

2016-10-20 Thread Hitesh Shah
Hello Madhu, If you are using Tez via Hive, then this would need a fix in Hive. I don’t believe Hive supports different settings for each vertex in a given query today. However, for native jobs, Tez already supports different specs for each vertex: Vertex::setTaskResource() ( configuring yarn r

Container settings at vertex level

2016-10-20 Thread Madhusudan Ramanna
Hello Folks, Some vertices require more memory than other vertices. These vertices are memory intensive.  The graph, in general, takes a long(ish) time to complete.   Default allocation of a huge chunk of memory to this one DAG/application severely limits concurrent yarn containers that can be ru