Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20920 )
Change subject: IMPALA-12727: Reduce IO threads for non-TARGET_FILESYSTEM filesystems ...................................................................... IMPALA-12727: Reduce IO threads for non-TARGET_FILESYSTEM filesystems The DiskIoMgr starts a large number of threads for each different type of object store, most of which are idle. For development, this slows down processing minidumps and debugging with gdb. This adds an option "reduce_disk_io_threads" to bin/start-impala-cluster.py that sets the thread count startup parameter to one for any filesystem that is not the TARGET_FILESYSTEM. On a typical development setup running against HDFS, this reduces the number of DiskIoMgr threads by 150 and the HDFS monitoring threads by 150 as well. This option is enabled by default. It can disabled by setting --reduce_disk_io_threads=False for bin/start-impala-cluster.py. Separately, DiskIoMgr should be modified to reduce the number of threads it spawns in general. Testing: - Hand tested this on my local development system Change-Id: Ic8ee1fb1f9b9fe65d542d024573562b3bb120b76 Reviewed-on: http://gerrit.cloudera.org:8080/20920 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M bin/start-impala-cluster.py 1 file changed, 33 insertions(+), 0 deletions(-) Approvals: Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/20920 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic8ee1fb1f9b9fe65d542d024573562b3bb120b76 Gerrit-Change-Number: 20920 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
