For some reason my overall comment on this PR failed to be copied to the list:

This is a really nice new feature. It works and is clean. It rates a +1. But I 
think we are likely to be processing thousands or perhaps millions of data 
points at any given time, and so the constant re-parsing of the bounds list is 
troublesome. Also for the STATS_BIN function, the percentile bounds list should 
be transformed once into a value bounds list, rather than applying the 
boundFunc many times to (on average) half the bounds in the list, per input 
value.

Unlike the Profiler, there is no config caching between calls to the Stellar 
Function, because this "configuration" is done every time, in-line, rather than 
in ZK. But we don't want all the complexity of ZK just for this little binning 
function. And we want multiple different binning functions to be in use at the 
same time, without needing complex scope management.

One solution would be to treat it like Python does with regex, and provide a 
compiler function. What if we have
COMPILE_BIN(bounds) and COMPILE_STATS_BIN(stats, bounds)
Each would return an opaque key (or an integer) that references a cached 
pre-parsed setup; it can be thread-safe as it would be read-only. Then we would 
invoke with
BIN(key, value) and STATS_BIN(key, value)

Normally I would say this is an optimization and we should leave it for later. 
But then we would be stuck with the inefficient, non-compiled form of the BIN 
and STATS_BIN functions.

Your call, @cestella . I don't want to get in the way of progress, I just feel 
obligated to bring it up.
--Matt



On 12/21/16, 11:50 PM, "mattf-horton" <[email protected]> wrote:

    Github user mattf-horton commented on a diff in the pull request:
    
        https://github.com/apache/incubator-metron/pull/401#discussion_r93574334
      
        --- Diff: 
metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
 ---
        @@ -425,4 +428,61 @@ public Object apply(List<Object> args) {
               return result;
             }
           }
        +
        +  /**
        +   * Calculates the statistical bin that a value falls in.
        +   */
        +  @Stellar(namespace = "STATS", name = "BIN"
        +          , description = "Computes the bin that the value is in based 
on the statistical distribution."
        +          , params = {
        +          "stats - The Stellar statistics object"
        +          , "value - The value to bin"
        +          , "bounds? - A list of percentile bin bounds (excluding min 
and max) or a string representing a known and common set of bins.  " +
        +          "For convenience, we have provided QUARTILE, QUINTILE, and 
DECILE which you can pass in as a string arg." +
        +          " If this argument is omitted, then we assume a Quartile bin 
split."
        +                    }
        +          ,returns = "Which bin N the value falls in such that 
bound(N-1) < value <= bound(N). " +
        +          "No min and max bounds are provided, so values smaller than 
the 0'th bound go in the 0'th bin, " +
        +          "and values greater than the last bound go in the M'th bin."
        +  )
        +  public static class StatsBin extends BaseStellarFunction {
        +    public enum BinSplits {
        +      QUARTILE(ImmutableList.of(25.0, 50.0, 75.0)),
        +      QUINTILE(ImmutableList.of(20.0, 40.0, 60.0, 80.0)),
        +      DECILE(ImmutableList.of(10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 
70.0, 80.0, 90.0))
        +      ;
        +      public final List<Double> split;
        +      BinSplits(List<Double> split) {
        +        this.split = split;
        +      }
        +
        +      public static List<Double> getSplit(Object o) {
        +        if(o instanceof String) {
        +          return BinSplits.valueOf((String)o).split;
        +        }
        +        else if(o instanceof List) {
        +          List<Double> ret = new ArrayList<>();
        +          for(Object valO : (List<Object>)o) {
        +            ret.add(ConversionUtils.convert(valO, Double.class));
        --- End diff --
        
        During this step we need to validate that the bounds list is strictly 
increasing.
    
    
    ---
    If your project is set up for it, you can reply to this email and have your
    reply appear on GitHub as well. If your project does not have this feature
    enabled and wishes so, or if the feature is enabled but not working, please
    contact infrastructure at [email protected] or file a JIRA ticket
    with INFRA.
    ---
    
    


Reply via email to