Joep Rottinghuis commented on YARN-3699:

The question really seems to be whether version is an identifying feature of a 
By and large the version increases linearly (in hRaven I think we accomplished 
that by having a version table with the version identifying, ie a hash of the 
Pig script or a hash of the class that runs the scalding code. Then the 
timestamp that we first see a new version becomes the version identifier).

In any case, we can ask give me the last 10 runs of this flow. Often you don't 
care about the versions. Leaving the version out of the rowkey, that will allow 
a quick linear range scan over 10 values.
If version is part of the key, and if it is possible for two versions to run 
overlapping (user submits a run of the old version, and then a newer version, 
then the older version again. More complex scenarios are possible), then one 
cannot do a quick range scan over just the 10 runs starting with the latest.
Once has to scan over all the runs up to the next flow. In theory it is 
possible that any version still had a run.

The reverse is true if one makes version a column. In that case getting the 
last 3 runs of a version becomes more expensive. In hRaven the usecase for this 
is when we want to do reducer estimation. We want to query back the last 3 runs 
of a flow and compare inputs and outputs of the various jobs in a flow. 
However, if one changes the Pig script or Scalding code then the DAG changes 
and one can no longer safely use inputs and outputs from those previous runs 
(with a different version). In theory we could scan the entire flow to see if 
there were ever any runs for this version. Particularly for the reducer 
estimation this is not needed. We can simply add a limit and say, give me only 
the last 3 runs in the last week. Even if we set a limit of let's say 100 runs 
that is fine. The query then becomes, give me the last 3 runs of this flow of 
this specific version out of the last 100 runs (of any version).
We would error on the side of returning fewer runs. Specifically for the 
reducer estimation case, that is essentially no worse than the first three runs 
of any flow version, so we simply fall back on the sampling based estimator or 
a size based heuristic.
Even if we do have to get an accurate answer, this still is a larger range scan 
over all the runs of a flow with an additional column value filter.

The more common case (give me the last few runs for the UI, for costing, for 
stats etc) should be the cheaper one, and the more rare (and selective one) 
with the version specified can be the more expensive one if one is not ready to 
tolerate some leniency in getting fewer rows back in return for guaranteed 
reasonable perf.

Another (but admittedly much much weaker) argument would be that storing 
version as a column if cheaper, since that part of the rowkey is not repeatedly 
stored for each column. In OpenTSDB for example, there are many lookup codes in 
order to make the rowkeys compact.
We could for example codify the cluster and instead of using a larger (human 
readable) name such as cluster@datacenter, one could use an integer value with 
a lookup table. For hRaven we never ended up making that optimization because 
the additional complexity of lookups, not having more readable keys etc. did 
not seem to be needed. The performance was simply good enough.

This view has a skew to the use-cases that we have seen in our production 
environment with hRaven. I could see somebody arguing for a different set of 
usecases and priorities. As with all schema design, there is probably no right 
or wrong answers in the abstract without considering the relative priority of 
the various uses.

> Decide if  flow version should be part of row key or column
> -----------------------------------------------------------
>                 Key: YARN-3699
>                 URL: https://issues.apache.org/jira/browse/YARN-3699
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Vrushali C
> Based on discussions in YARN-3411 with [~djp], filing jira for continuing 
> discussion on putting the flow version in rowkey or column. 
> Either phoenix/hbase approach will update the jira with the conclusions..

This message was sent by Atlassian JIRA

Reply via email to