Sangjin Lee updated YARN-4074:
    Attachment: YARN-4074-YARN-2928.POC.003.patch

POC v.3 patch posted.

Key changes include
- switched from Get.setMaxResultSize() to PageFilter (more on that below)
- major refactoring of HBaseTimelineReaderImpl
-- introduced TimelineEntityReader and the hierarchy of classes to isolate 
proper reading per type
- added unit tests to test HBaseTimelineReaderImpl for flow activity and flow 
- fixed an issue with FlowScanner where the cells were returned in the wrong 
order so it was breaking Column.readResult()
- made *RowKey classes real object classes, and added the parseRowKey method 
that returns an instance of the RowKey
- fixed the order of the add and pollLast
- renamed FlowEntity to FlowRunEntity
- added the compareTo() method for FlowActivityEntity
- passed the type into the FlowActivityEntity constructor
- set configs for FlowActivityEntity and FlowRunEntity to null
- improved the way we get string values from info for FlowActivityEntity and 
- added getNumberOfRuns() to FlowActivityEntity

It is actually pretty close to being ready, but since YARN-3901 is still 
outstanding, I'm not making it an official patch yet.

As for the PageFilter issue, I concluded setMaxResultSize() is not the right 
API to use to limit the number of rows. I believe the PageFilter is the right 
thing to use. I also added the counting logic to get the right number of 
records even if the result iterator advances.

As for the FlowScanner issue mentioned above, [~vrushalic] and [~jrottinghuis] 
debugged this to track down a bug in YARN-3901. As such, this change will 
likely be made in the final YARN-3901 patch. I just included it here for 
completeness and to make the unit code pass.

You should be able to apply the YARN-3901 v.3 patch and then this patch 
cleanly. Let me know if you have any questions.

I'd greatly appreciate review feedback. I understand it's a lot of code...

> [timeline reader] implement support for querying for flows and flow runs
> ------------------------------------------------------------------------
>                 Key: YARN-4074
>                 URL: https://issues.apache.org/jira/browse/YARN-4074
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-4074-YARN-2928.POC.001.patch, 
> YARN-4074-YARN-2928.POC.002.patch, YARN-4074-YARN-2928.POC.003.patch
> Implement support for querying for flows and flow runs.
> We should be able to query for the most recent N flows, etc.
> This includes changes to the {{TimelineReader}} API if necessary, as well as 
> implementation of the API.

This message was sent by Atlassian JIRA

Reply via email to