Wangda Tan commented on YARN-3098:

Hi [~sunilg],
Thanks for thinking about this,.

I think two potential issues we need to take care regarding locks:
1) Deadlock caused by the new introduced locks.
2) Data inconsistency.

For 1),
I think it will be no problem if *a.* we don't try to acquire other locks from 
methods of QueueCapacities/ResourceUsage. And *b.* methods of QC/RU are pretty 
simple, all can terminate in finite time. 
If a/b meet at the same time. No new deadlock will happen. Agree?

For 2),
Data inconsistency also has two different types,
2.1 Data inconsistency while editing
When we edit data within QC/RU, it still under queue's lock. Which has no data 
inconsistency issue.

2.2 Data inconsistency while reading
When we read data in QC/RU from outside, it is possible QC in state1 (e.g. 
container-1 allocated) but RU in state2 (e.g. container-2 allocated). In the 
past we have same issue. We cannot avoid this except we wrap such data to a 
structure like "QueueStatusInfo" like AppReport.


> Create common QueueCapacities class in Capacity Scheduler to track 
> capacities-by-labels of queues
> -------------------------------------------------------------------------------------------------
>                 Key: YARN-3098
>                 URL: https://issues.apache.org/jira/browse/YARN-3098
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, 
> YARN-3098.4.patch
> Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) 
> need to track capacities-label (e.g. absolute-capacity, maximum-capacity, 
> absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a 
> class to encapsulate these capacities to make both better 
> maintainability/readability and fine-grained locking.

This message was sent by Atlassian JIRA

Reply via email to