Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8363 )
Change subject: IMPALA-4456: Address scalability issues of qs_map_lock_ and client_request_state_map_lock_ ...................................................................... Patch Set 4: (4 comments) > Feel free to ignore, but I'm curious if you tried out SpinLock too. > be/src/benchmarks/lock-benchmark.cc shows that it is somehow more > efficient than boost::mutex under contention. Using SpinLock shows better performance in the general case. So, I've switched qs_map_lock_ to use spin locks. I haven't done the same for client_request_state_map_lock_, since the access patterns may be different, and it would be best to confirm with a benchmark first before making a decision. I can add a benchmark in a follow on patch. http://gerrit.cloudera.org:8080/#/c/8363/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8363/3//COMMIT_MSG@30 PS3, Line 30: sharded to a default of 4 buckets initally. > nit: sharded Done http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/runtime/query-exec-mgr.cc File be/src/runtime/query-exec-mgr.cc: http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/runtime/query-exec-mgr.cc@84 PS3, Line 84: DEFINE_int32(log_mem_usage_interval, 0, "If non-zero, impalad will output memory usage " : "every log_mem_usage_interval'th fragment complet > Would it make sense to write a helper so that this code could look like: Yup, that would make more sense. I've added a ScopedQueryStateMap {} class that locks/unlocks the appropriate map(bucket) on scope entry/exit respectively. I added a similar class ScopedCRSMap in impala-server.h/cc. http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/service/impala-http-handler.cc@244 PS3, Line 244: stringstream ss; > 4 should be the constant here, no? Yup, changed it. http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/8363/3/be/src/service/impala-server.h@562 PS3, Line 562: /// ClientRequestState's deletion. Also writes the query profile to the profile log > nit: corresponding Done -- To view, visit http://gerrit.cloudera.org:8080/8363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I61089090e1095da45a8a64ed3ccc78bd310807f1 Gerrit-Change-Number: 8363 Gerrit-PatchSet: 4 Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Wed, 25 Oct 2017 22:48:30 +0000 Gerrit-HasComments: Yes
