Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15472
Change subject: WIP - IMPALA-9457: reduce # of unused threads ...................................................................... WIP - IMPALA-9457: reduce # of unused threads TODO: * comments. * perf * need a thread pool test This reduces the number of threads created in ThreadPool and for the Disk I/O manager. Before this an idle Impala has 100s of unused threads across all the different thread pools - 260 backend threads on my system. After this change the Impalad has 57 threads on startup and it ramps up to ~100 when running some queries and back to ~80 when idle. ThreadPool threads are created lazily only when there aren't enough idle threads to serve all the queued requests. After this patch, only 1 thread is created per disk queue initially. Additional threads are created only after ranges are submitted to the queue. This dramatically reduces the number of I/O threads on typical systems because only 1 thread per remote FS is created. In both cases I chose to always create the first thread because it sidesteps any issues with liveness that might occur if thread creation faled - with one thread, we can always make progress on the requests in the queue. Also switch from raw pointers to unique pointers in some places in the I/O manager. Change-Id: I249f48a0c1155bc6551966e2ee9e1b0faefde022 --- M be/src/common/logging.h M be/src/kudu/util/logging.h M be/src/runtime/io/disk-io-mgr-internal.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/util/blocking-queue.h M be/src/util/thread-pool.h 9 files changed, 213 insertions(+), 64 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/15472/4 -- To view, visit http://gerrit.cloudera.org:8080/15472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I249f48a0c1155bc6551966e2ee9e1b0faefde022 Gerrit-Change-Number: 15472 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong <[email protected]>
