pranav.lo...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 )
Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() ...................................................................... Patch Set 26: (3 comments) > Patch Set 25: Code-Review+1 > > (3 comments) > > Math seems good to me http://gerrit.cloudera.org:8080/#/c/18413/25/be/src/exprs/aggregate-functions-ir.cc File be/src/exprs/aggregate-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/18413/25/be/src/exprs/aggregate-functions-ir.cc@300 PS25, Line 300: struct CorrState { : int64_t count; // number of elements : double xavg; // average of x elements : double yavg; // average of y elements : double xvar; // n times the variance of x elements : double yvar; // n times the variance of y elements : double covar; // n times the covariance : }; > As I understand, both CorrState and CovarState calculates xavg,yavg and cov Yeah, that's right, we did something similar in our 12th patch where we didn't create a new state for covar_samp() and covar_pop() but two fields were not getting used in calculating covar_samp() and covar_pop() and it was causing a lot of unnecessary computation so we separated it into two different states. http://gerrit.cloudera.org:8080/#/c/18413/25/be/src/exprs/aggregate-functions-ir.cc@324 PS25, Line 324: if (state->count > 1 > nit: do we need this? for first items state->xavg = deltaX = x, so we will Yeah, you're right, Quanlong and I had a discussion on this, we decided to add this check as it saves some floating point computation. More computation would means more floating point rounding error. http://gerrit.cloudera.org:8080/#/c/18413/25/be/src/exprs/aggregate-functions-ir.cc@331 PS25, Line 331: // vx_n = vx_(n - 1) + (x_n - mx_(n - 1)) * (x_n - mx_n) > nit: the order is a bit misleading as deltax=(x_n - mx_(n - 1)) Yeah, I just changed it. -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 26 Gerrit-Owner: Anonymous Coward <pranav.lo...@cloudera.com> Gerrit-Reviewer: Anonymous Coward <pranav.lo...@cloudera.com> Gerrit-Reviewer: Gergely Fürnstáhl <gfurnst...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Jian Zhang <zjsar...@gmail.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Wed, 15 Jun 2022 15:28:31 +0000 Gerrit-HasComments: Yes
