Hi Aakash, This is an interesting topic. Now the scheduler is not aware of the underlying hardware differences. In my understanding, the placement constraints or label or node attributes together with application-level(MR/Spark/Hive to be cloud instance aware) changes could potentially achieve your goal.
BR, Zhankun On Wed, 18 Dec 2019 at 16:34, Aakash Sharma <[email protected]> wrote: > Hello Yarn Gurus, > > I am working on a YARN enhancement for optimizing utilization in AWS. > Is it possible to classify a container in resource manager as a Map or > Reduce task? I have looked at the source code of capacity scheduler, and I > dont think it is possible. > If I were to make some placement optimizations in RM to run certain reduce > tasks on a certain type of nodes, without using node labeling, how can I go > about doing this? > > Can anyone please give me some suggestions? > > -- > Thanks, > Aakash >
