Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10977 )
Change subject: IMPALA-7296: bytes limit for row batch queue ...................................................................... Patch Set 5: (9 comments) http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue-test.cc File be/src/util/blocking-queue-test.cc: http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue-test.cc@97 PS5, Line 97: LONG_TIMEOUT_MICROS); > Does it make the test timing sensitive ? Should we use a larger timeout ? Bumped to 1 min http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h File be/src/util/blocking-queue.h: http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@117 PS5, Line 117: ElemBytesFn()(*out); > It's a bit unfortunate that each element goes through two indirect function I thought about doing that and didn't like the idea of storing the size when it's not needed. That probably isn't a big deal in practice though - the overhead won't matter for RowBatchQueue since RowBatch is much larger than an int64 and probably isn't a big deal for ThreadPool, which is the other user of BlockingQueue. The templated function doesn't require an indirect call so I think it's a trade-off between complexity of templated functors vs the space overhead. I guess the one other advantage of the template functor is that it's possible to optimise out most of the added logic if the functor always returns 0. I don't feel that strongly about which alternative is better, open to suggestions. http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@117 PS5, Line 117: ElemBytesFn()(*out); > Actually, this may not actually be an indirect call after all, depending on Yeah it's a direct call to ElemBytesFn::operator(), there's no function pointer or virtual function so no way for an indirect call to get generated. http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@142 PS5, Line 142: int64_t val_bytes = ElemBytesFn()(val); > this can go up outside the lock, right? Done http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@152 PS5, Line 152: if (val_bytes > 0) > Not sure why the need for if() here. I can only assume this is to avoid acc My thought was that if ElemBytesFn() always returns 0 this branch can be optimised out, but it is totally a premature optimisation. http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@170 PS5, Line 170: int64_t val_bytes = ElemBytesFn()(val); > same, can compute outside the lock Done http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@185 PS5, Line 185: if (val_bytes > 0) > See comments above. Done http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@237 PS5, Line 237: bool HasCapacityInternal(const boost::unique_lock<boost::mutex>& lock, int64_t val_bytes) { > nit: long line Done http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@249 PS5, Line 249: get_bytes_dequeued_.Load(); : if (dequeued == 0) return false; : put_bytes_enqueued_ -= dequeued; : get_bytes_dequeued_.Add(-dequeued); > could you just Exchange(0) rather than loading and decrementing? seems like Done -- To view, visit http://gerrit.cloudera.org:8080/10977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa06d1d8da2a6d101efda08f620c0bf84a71e681 Gerrit-Change-Number: 10977 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Wed, 25 Jul 2018 18:56:50 +0000 Gerrit-HasComments: Yes
