[jira] [Commented] (YARN-3034) implement RM starting its ATS writer
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314763#comment-14314763 ] Sangjin Lee commented on YARN-3034: --- Thanks [~Naganarasimha]! I'll go over the patch today... bq. Whether we require Multithreaded Dispatcher as we are not publishing container life cycle events and if normal dispatcher is ok whether to use rmcontext.getDispatcher ? For publishing app lifecycle events only, I suspect a normal dispatcher might be OK. However, there could be more use cases in the future. If it is not too complicated, using a multi-threaded dispatcher might be bit preferable IMO. Thoughts? bq. AppAttempt needs to be Entity or event of ApplicationEntity ? i feel later option is better How is it today with the current ATS? If the same container can be part of different app attempts (e.g. successive AMs managing the same set of containers), then app attempts can't be separate entities? [~zjshen]? implement RM starting its ATS writer Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034.20150205-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) implement RM starting its ATS writer
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314797#comment-14314797 ] Sangjin Lee commented on YARN-3034: --- Some feedback on the patch... (1) this creates a dependency from RM to the timeline service; perhaps it is unavoidable... (2) RMTimelineAggregator.java - we need the license - annotate with @Private and @Unstable - line. 31: nit; spacing (3) SystemMetricsPublisher.java - instead of replacing the use of the existing ATS, I think we need to have both (the existing ATS calls as well as the new calls); we will need a global config that enables/disables the next gen timeline service implement RM starting its ATS writer Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034.20150205-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) implement RM starting its ATS writer
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315265#comment-14315265 ] Zhijie Shen commented on YARN-3034: --- bq. Whether we require Multithreaded Dispatcher as we are not publishing container life cycle events and if normal dispatcher is ok whether to use rmcontext.getDispatcher ? Previously, the reason why we use a separate dispatcher instead of the rmcontext dispatcher is that we want to make sure the timeline service I/O operations may not block the normal app life cycle management. The multi-threaded dispatcher is designed to increase concurrency. If aggregator is able to handle the requests in the async way, I'm okay to use rmcontext dispatcher. Otherwise, let's make sure at least we're using a separate async dispatcher. bq. How is it today with the current ATS? Currently, we don't have any special app/attempt/container entity. They're the payload of entity objects. I think it makes sense to have the special app/attempt/container entity for next gen, because given these are first class citizen and predefined, we have more chance to do the storage level optimization for them, instead of treating them generally. Thoughts? To Naga's question, I suggest that app, attempt and container are all entities instead of being events of the parent entity. Logically attempt or container is the sub-component within an app, and has it's related events. implement RM starting its ATS writer Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034.20150205-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) implement RM starting its ATS writer
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307194#comment-14307194 ] Naganarasimha G R commented on YARN-3034: - Modifications as part of the patch : 1) Created RMTimelineAggregator by extending BaseAggregatorService (and the required dependencies) 2) Refactored SystemMetricsPubliser and moved the event handling part to RMTimelineAggregator itself 3) Removed publishing of Container related life cycle events from RM. 4) Refactored the test cases to remove testcases for storing container related life cycle events. Pending 1) Once ATS reader interface and Dataobjects are finalized, can further stub and complete TestSystemMetricsPublisher some points which required further discussion 1) Whether we require Multithreaded Dispatcher as we are not publishing container life cycle events and if normal dispatcher is ok whether to use rmcontext.getDispatcher ? 2) AppAttempt needs to be Entity or event of ApplicationEntity ? i feel later option is better 3) It would better for BaseAggregatorService.postEntities postEntitiesAsync to throw YARNException and IOException as part of method signature. implement RM starting its ATS writer Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034.20150205-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
