Joep Rottinghuis commented on YARN-3411:

I think it is reasonable that two implementations can differ in their backing 
schema as long as they both can write the data and retrieve the data with the 
same key information. Phoenix may need to add somethings to the rowkey in order 
to work properly, it may have to add some things, and ditto for the raw HBase 
implementation, some additional secondary lookups may be needed etc. That is 
part of the performance comparison to see.

[~djp] with respect to adding the flow version in the key, I think the problem 
with that is that you now require the caller to know what the version is in 
order to query back. I don't think that is a natural requirement. I know that I 
ran the "ComputeUniqueUsers" flow on the cluster, so I have user cluster and 
flowname, but I don't need to know the version to just query the last few runs 
right? If you do have the version (for reducer estimation and you want the last 
runs of the same flow back) then it should be possible to query by flow _and_ 
by version, but I don't think it should be mandatory.
Therefore I don't think that flow version must perse be a rowkey in all 

I think we'll find that with certain schema choices some things will be more 
performant while others will be somewhat slower. It will be a mater of finding 
those schema choices that will give good enough write performance to handle 
scale and give good read performance for the most common use cases, while 
maintaining reasonable performance for other queries.

> [Storage implementation] explore the native HBase write schema for storage
> --------------------------------------------------------------------------
>                 Key: YARN-3411
>                 URL: https://issues.apache.org/jira/browse/YARN-3411
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Vrushali C
>            Priority: Critical
>         Attachments: ATSv2BackendHBaseSchemaproposal.pdf, 
> YARN-3411-YARN-2928.001.patch, YARN-3411-YARN-2928.002.patch, 
> YARN-3411-YARN-2928.003.patch, YARN-3411-YARN-2928.004.patch, 
> YARN-3411-YARN-2928.005.patch, YARN-3411-YARN-2928.006.patch, 
> YARN-3411-YARN-2928.007.patch, YARN-3411.poc.2.txt, YARN-3411.poc.3.txt, 
> YARN-3411.poc.4.txt, YARN-3411.poc.5.txt, YARN-3411.poc.6.txt, 
> YARN-3411.poc.7.txt, YARN-3411.poc.txt
> There is work that's in progress to implement the storage based on a Phoenix 
> schema (YARN-3134).
> In parallel, we would like to explore an implementation based on a native 
> HBase schema for the write path. Such a schema does not exclude using 
> Phoenix, especially for reads and offline queries.
> Once we have basic implementations of both options, we could evaluate them in 
> terms of performance, scalability, usability, etc. and make a call.

This message was sent by Atlassian JIRA

Reply via email to