Vrushali C updated YARN-3901:
    Attachment: YARN-3901-YARN-2928.WIP.patch

Uploading a work in progress patch. This patch is not yet rebased against the 
new branch. I would like to finish the patch and then rebase it.

- it has some new classes that deal with flow run table. 
- it adds in cell level tags to the cells being stored in the flow run table. 
- it has a coprocessor class that currently handles the put (prePut) and scan 
(preScannerOpen and postScannerOpen) operations. 
- it has a new AggregationScanner class that is invoked from the preprocessor, 
so any scans that hit this table will effectively go through via the 
AggregationScanner class methods
- the start time for a flow is defined as the lowest amongst the start times of 
all applications in that flow run. Similarly the end  time for a flow is 
defined as the biggest amongst the start times of all applications in that flow 
run. These are stored per flow run upon application creation and application 
finish events. The coprocessor prePut intercepts these and puts in only the 
right values. 
- for metrics, all metrics are stored as they come in. When a metric for a flow 
run is to be read back, a special scanner is used. This scanner reads all cells 
for that metric that belong to all applications. Only the latest cell per 
application is picked and added up to form a metric value for that flow run. 
The application states are also stored as running and finished in the cell tag 
for metrics

- TODO: 
- Working next to add in the get (very similar to scan), flush, compact 
- The applications that have finished can be compacted and a per flow run 
metric cell can be created.
- The finished application cells can be then removed

> Populate flow run data in the flow_run table
> --------------------------------------------
>                 Key: YARN-3901
>                 URL: https://issues.apache.org/jira/browse/YARN-3901
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>         Attachments: YARN-3901-YARN-2928.WIP.patch
> As per the schema proposed in YARN-3815 in 
> https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf
> filing jira to track creation and population of data in the flow run table. 
> Some points that are being  considered:
> - Stores per flow run information aggregated across applications, flow version
> RM’s collector writes to on app creation and app completion
> - Per App collector writes to it for metric updates at a slower frequency 
> than the metric updates to application table
> primary key: cluster ! user ! flow ! flow run id
> - Only the latest version of flow-level aggregated metrics will be kept, even 
> if the entity and application level keep a timeseries.
> - The running_apps column will be incremented on app creation, and 
> decremented on app completion.
> - For min_start_time the RM writer will simply write a value with the tag for 
> the applicationId. A coprocessor will return the min value of all written 
> values. - 
> - Upon flush and compactions, the min value between all the cells of this 
> column will be written to the cell without any tag (empty tag) and all the 
> other cells will be discarded.
> - Ditto for the max_end_time, but then the max will be kept.
> - Tags are represented as #type:value. The type can be not set (0), or can 
> indicate running (1) or complete (2). In those cases (for metrics) only 
> complete app metrics are collapsed on compaction.
> - The m! values are aggregated (summed) upon read. Only when applications are 
> completed (indicated by tag type 2) can the values be collapsed.
> - The application ids that have completed and been aggregated into the flow 
> numbers are retained in a separate column for historical tracking: we don’t 
> want to re-aggregate for those upon replay

This message was sent by Atlassian JIRA

Reply via email to