Sangjin Lee commented on YARN-3904:

I agree it is appealing to centralize table creations. After putting some 
thoughts here I think what we really want is a centralized workflow for storage 
schema creations. That is to say, when setting up a v2 timeline server, users 
can simply run data schema creator for once to create necessary data storage 
schemas. With this in mind, I added Phoenix schema creation into the existing 
data schema creator, with a separate option -p. However, I'm keeping the SQL 
statements for table creation inside the writer file so that we also have a 
centralized place for the Phoenix storage schema.

I'm fine with that approach.

We can definitely reuse this PreparedStatement (as well as the connections) 
after we integrated the aggregation writer with the aggregator. My plan is to 
use this (relatively) stable writer to unblock the future patch on flow and 
user level offline aggregation. After we have the whole workflow, we can 
gradually add optimizations. Thoughts?

Yes, that sounds fine. Thanks!

> Refactor timelineservice.storage to add support to online and offline 
> aggregation writers
> -----------------------------------------------------------------------------------------
>                 Key: YARN-3904
>                 URL: https://issues.apache.org/jira/browse/YARN-3904
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-3904-YARN-2928.001.patch, 
> YARN-3904-YARN-2928.002.patch, YARN-3904-YARN-2928.003.patch, 
> YARN-3904-YARN-2928.004.patch, YARN-3904-YARN-2928.005.patch, 
> YARN-3904-YARN-2928.006.patch, YARN-3904-YARN-2928.007.patch
> After we finished the design for time-based aggregation, we can adopt our 
> existing Phoenix storage into the storage of the aggregated data. In this 
> JIRA, I'm proposing to refactor writers to add support to aggregation 
> writers. Offline aggregation writers typically has less contextual 
> information. We can distinguish these writers by special naming. We can also 
> use CollectorContexts to model all contextual information and use it in our 
> writer interfaces. 

This message was sent by Atlassian JIRA

Reply via email to