Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/401
  
    Ok, so a couple of things.  I ran a quick perf test on this function as it 
stands.  On my macbook from 4 years ago, I ran the `STATS_BIN` function a 
million times with random values and it took ~5.5s.  Even at a throughput of 1M 
messages per second, if we assume that the messages are spread across the 
cluster, I think this keeps up.
    
    Now, *EVEN GIVEN THIS*, normally I would go through the effort here of 
adding a caching layer to save us the computation of the percentile, but it's 
actually quite difficult to figure out when two `StatisticsProvider` objects 
are equivalent without resorting to actually calling percentiles.  I could do 
things like use non-computed attributes (number of data points, sum of data 
points, average and sum of data points), but the former Math grad student in me 
was uncomfortable in that.  It's just very hard to not be absolutely sure that 
you couldn't have all of those attributes the same and be different 
distributions.
    
    I think given these things together that I'm going to recommend to cross 
the caching bridge when we come to it.  Now, it won't take much to convince me 
to go the other direction, so if you (or anyone else, really ;) feels strongly 
@mattf-horton , I'll go ahead and tackle that dude as best I can.
    
    I'm going to correct the rest of your comments and check in the performance 
test I ran, so you can see that I have nothing up my sleeves and so it can be 
run periodically.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---

Reply via email to