[
https://issues.apache.org/jira/browse/YARN-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645313#comment-14645313
]
Sangjin Lee commented on YARN-3904:
-----------------------------------
I went through the patch at a high level. I need to do another more detailed
pass later, but wanted to share the high level feedback first.
Regarding {{PhoenixOfflineAggregatorWriterImpl}}, does it have to implement the
{{TimelineWriter}} interface?
It is no longer plugged into the real-time write path, and as such,
implementing {{TimelineWriter}} seems unnecessary. If we envision using this in
a separate mechanism such as mapreduce, I think we ought to come up with a new
interface for aggregation. For example, in l.106-114, seeing which field in
{{TimelineCollectorContext}} is not null and triggering aggregation that way
seems pretty awkward, and that might be because we're trying to use the
{{TimelineWriter}} interface and work with {{TimelineCollectorContext}}, but
{{TimelineCollectorContext}} wasn't really designed for that purpose, and we
would be using it in a unexpected manner. This to me is an unnecessary
constraint.
Also, the actual work of reading the HBase tables (eventually the flow run
table) and invoking the offline aggregator is not captured here. I presume it
would come later?
> Refactor timelineservice.storage to add support to online and offline
> aggregation writers
> -----------------------------------------------------------------------------------------
>
> Key: YARN-3904
> URL: https://issues.apache.org/jira/browse/YARN-3904
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Li Lu
> Attachments: YARN-3904-YARN-2928.001.patch,
> YARN-3904-YARN-2928.002.patch, YARN-3904-YARN-2928.003.patch,
> YARN-3904-YARN-2928.004.patch
>
>
> After we finished the design for time-based aggregation, we can adopt our
> existing Phoenix storage into the storage of the aggregated data. In this
> JIRA, I'm proposing to refactor writers to add support to aggregation
> writers. Offline aggregation writers typically has less contextual
> information. We can distinguish these writers by special naming. We can also
> use CollectorContexts to model all contextual information and use it in our
> writer interfaces.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
