[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.06.patch Rebasing patch and adding caller context from DagSubmittedEvent to otherInfo entity. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch, > TEZ-3611.04.patch, TEZ-3611.05.patch, TEZ-3611.06.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.05.patch Fixed ATSImportTool to be compatible with earlier and current logging formats. [~rajesh.balamohan] Please tell me if the fix is fine. Currently I have taken the approach of merging the data from TEZ_DAG_EXTRA_INFO into the "dag" json. Another approach is to add TEZ_DAG_EXTRA_INFO as another file in the zip and then change the processing in ATSFileParser/BaseParser to handle this case. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch, > TEZ-3611.04.patch, TEZ-3611.05.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.04.patch Thanks [~sseth]. Addressing comments: * HistoryEventTimelineConversion returns a list of events. * ATSHistoryLoggingService and ATSV15HistoryLoggingService log events are same now. [~sseth] I do not think we can remove ATSHistoryLoggingService, since hadoop 2.6 does not have acls. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch, > TEZ-3611.04.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: (was: TEZ-3627.04.patch) > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3627.04.patch Sorry [~sseth]. Removed the extra lock and made queueName volatile. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch, > TEZ-3627.04.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.03.patch Moved counters from DAGFinishedEvent into DAG_EXTRA_INFO entity. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.02.patch Changed the entity name to be TEZ_DAG_EXTRA_INFO, which we can use to move counters too. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3611) Create lightweight summary events for ATS.
[ https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3611: --- Attachment: TEZ-3611.01.patch Created TEZ_DAG_PLAN entity and split the DAGSubmittedEvent to be sent as two entities, TEZ_DAG_ID which contains existing data without dagPlan and TEZ_DAG_PLAN which contains only the dagPlan. This change is only made for ATSv15, since ATSv1 does not have concept of summary events. Should I make it consistent for both? Leaving the counters as suggested by [~sseth] to ensure the UI can display them in the "All Dags" page. > Create lightweight summary events for ATS. > -- > > Key: TEZ-3611 > URL: https://issues.apache.org/jira/browse/TEZ-3611 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3611.01.patch > > > All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and > DAGFinishedEvent has counters in it. And hence these events very large, which > makes the dag landing page on the tez-ui very expensive. > Split these events into two parts: > * One which contains only data used by the tez-ui "All Dags" page which will > go into summary log. > * Another which will contain the current object, which will be part of the > appropriate TimelineEntityGroupId. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
