Zhijie Shen commented on YARN-3411:

bq. A flow can be uniquely identified with the flow name and run id (and of 
course cluster and user id).

I think in Phoenix impl, we have treated version as part of identifier of a 
unique flow.

bq. Hmm, so schema creation happens more or less once in the lifetime of the 
hbase cluster like during cluster setup (or perhaps if we decide to drop and 
recreate it, which is rare in production). I believe writers will come to life 
and cease to exist with each yarn application lifecycle but cluster is more or 
less eternal, so adding to this step to the lifecycle of a Writer Impl object 
seems somewhat out of place to me.

Fair point. And this is another place different from Phoenix impl, which 
creates table if they don't exist. My perspective is more about automation, and 
it's better to leave fewer steps for users to setup the service. Perhaps we can 
find somewhere else to invoke the table initialization once if the service is 
setup for YARN cluster, and HBase/Phoenix is used as the backend.

> [Storage implementation] explore the native HBase write schema for storage
> --------------------------------------------------------------------------
>                 Key: YARN-3411
>                 URL: https://issues.apache.org/jira/browse/YARN-3411
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Vrushali C
>            Priority: Critical
>         Attachments: ATSv2BackendHBaseSchemaproposal.pdf, 
> YARN-3411-YARN-2928.001.patch, YARN-3411-YARN-2928.002.patch, 
> YARN-3411-YARN-2928.003.patch, YARN-3411-YARN-2928.004.patch, 
> YARN-3411-YARN-2928.005.patch, YARN-3411-YARN-2928.006.patch, 
> YARN-3411.poc.2.txt, YARN-3411.poc.3.txt, YARN-3411.poc.4.txt, 
> YARN-3411.poc.5.txt, YARN-3411.poc.6.txt, YARN-3411.poc.7.txt, 
> YARN-3411.poc.txt
> There is work that's in progress to implement the storage based on a Phoenix 
> schema (YARN-3134).
> In parallel, we would like to explore an implementation based on a native 
> HBase schema for the write path. Such a schema does not exclude using 
> Phoenix, especially for reads and offline queries.
> Once we have basic implementations of both options, we could evaluate them in 
> terms of performance, scalability, usability, etc. and make a call.

This message was sent by Atlassian JIRA

Reply via email to