Hi Lee, regarding how to depict the error distribution in a way that's easy to explain and shows differences between LRA and HRA and relative and uniform guarantees: I suggest to simply use re[i] = r[i] - tr[i]. This is actually a uniform error, and for our sketch the plots should roughly look like a linear function (decreasing for HRA and increasing for LRA). So, the plots shouldn't look like any of the four plots you sent on Friday. Cheers, Pavel
On 2020-09-12 20:26, leerho wrote: Hi Pavel, I think we are in agreement. I take it as a positive that you haven't found any flaw in the implementation of the relative error quantiles algorithm. What we are discussing now is definitions in how to define rank especially with respect to HRA and LRA, and philosophically, what kind of error distribution as a function of rank will users want and be easiest to specify and explain. My intuition is that all plots should look roughly the same, up to mirroring along the Y axis. This statement is a little puzzling as it will only be true if we choose definitions of rank appropriately if the user selects LRA or HRA. As my plots reveal, if we keep the definition of rank the same, switching from HRA to LRA has dramatically different error distribution as a f(rank). I agree with your formulas for relative error, except that all of our ranks are already normalized by SL, so I would replace SL in your formulas by 1.0. I still want to add to these plots your a priori calculation of error to see where it lies compared to the measured error. I gather from your comments that what you had in mind when writing the paper was a rank error distribution that looks like plots 3 and 4 above and not plots 1 and 2. However, I feel that the rank error distribution shown in plots 1 and 2 would be easier for users to understand. We will probably leave the decision totally up to the user as to what they prefer, however, this will require more documentation to explain all the different error behaviors :( . Lee.
