Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15202 )
Change subject: IMPALA-9373: Trial run of include-what-you-use ...................................................................... IMPALA-9373: Trial run of include-what-you-use Implemented recommendations from IWYU in a subset of files, mostly in util. Did a few cleanups related to systematic problems that I noticed as a result. I noticed that uid-util.h was pulling in boost UUID headers to a lot of compilation units, so refactored that a little bit, including pulling out the hash functions into unique-id-hash.h and moving some inline functions into client-request-state-map.cc. Systematically replaced the general boost mutex header with the internal pthread-based one. This is equivalent for us, since we assume that boost::mutex is implemented by pthread_mutex_t, e.g. for the implementation of ConditionVariable. Switch include guards to pragma once just as general cleanup. Prefix string with std:: consistently in headers so that they don't depend on "using" declarations pulled in from random headers. Look at includes of C++ stream headers, including iostream and stringstream, and replaced them with iosfwd or removed them if possible. Compile time: Measured a full ASAN build of the impalad binary on an 8 core machine with cccache enabled, but cleared. It used very slightly less CPU, probably because we are still pulling in most of the same system headers. Before: real 9m27.502s user 64m39.775s sys 2m49.002s After: real 9m26.561s user 64m28.948s sys 2m48.252s So for the moment, the only significant wins are on incremental builds, where touching header files should not require as many recompilations. Compile times should start to drop meaningfully once we thin out more unnecessary includes - currently it seems like most compile units end up with large chunks of boost/std code included via transitive header dependencies. Change-Id: I3450e0ffcb8b183e18ac59c8b33b9ecbd3f60e20 Reviewed-on: http://gerrit.cloudera.org:8080/15202 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/benchmarks/lock-benchmark.cc M be/src/benchmarks/process-wide-locks-benchmark.cc M be/src/catalog/catalog-server.h M be/src/codegen/llvm-codegen.cc M be/src/common/logging.cc M be/src/common/names.h M be/src/common/object-pool.h M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/data-sink.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hash-table.h M be/src/exec/hbase-table-scanner.h M be/src/exec/hdfs-plugin-text-scanner.h M be/src/exec/hdfs-scan-node.h M be/src/exec/kudu-scanner.h M be/src/exec/parquet/parquet-column-chunk-reader.h M be/src/exec/parquet/parquet-column-stats.inline.h M be/src/exec/parquet/parquet-level-decoder.h M be/src/exec/parquet/parquet-metadata-utils.h M be/src/exec/parquet/parquet-page-reader.h M be/src/exec/plan-root-sink.cc M be/src/exec/read-write-util.h M be/src/exec/scanner-context.h M be/src/exec/write-stream.h M be/src/exec/write-stream.inline.h M be/src/experiments/data-provider.h M be/src/exprs/operators-ir.cc M be/src/rpc/auth-provider.h M be/src/rpc/impala-service-pool.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/client-cache.h M be/src/runtime/collection-value-builder.h M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/date-parse-util.h M be/src/runtime/decimal-test.cc M be/src/runtime/decimal-value.inline.h M be/src/runtime/descriptors.h M be/src/runtime/dml-exec-state.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/hdfs-fs-cache.h M be/src/runtime/initial-reservations.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/handle-cache.h M be/src/runtime/io/local-file-system-with-fault-injection.h M be/src/runtime/io/request-ranges.h M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/krpc-data-stream-recvr.h M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/lib-cache.h M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/raw-value.h M be/src/runtime/row-batch.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter.h M be/src/runtime/scanner-mem-limiter.h M be/src/runtime/sorted-run-merger.h M be/src/runtime/string-search.h M be/src/runtime/thread-resource-mgr.h M be/src/runtime/tmp-file-mgr.h M be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.h M be/src/service/CMakeLists.txt M be/src/service/child-query.h A be/src/service/client-request-state-map.cc M be/src/service/client-request-state-map.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/data-stream-service.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/statestore/failure-detector.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.h M be/src/testutil/mini-kdc-wrapper.h M be/src/thirdparty/mustache/mustache.cc M be/src/thirdparty/mustache/mustache.h M be/src/transport/TSaslServerTransport.h M be/src/udf/uda-test-harness.h M be/src/udf/udf-internal.h M be/src/udf/udf-test-harness.h M be/src/util/CMakeLists.txt M be/src/util/auth-util.cc M be/src/util/auth-util.h M be/src/util/bit-stream-utils.h M be/src/util/bit-util.cc M be/src/util/bit-util.h M be/src/util/blocking-queue-test.cc M be/src/util/blocking-queue.h M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/buffer-builder.h M be/src/util/cgroup-util.cc M be/src/util/codec.cc M be/src/util/codec.h M be/src/util/coding-util.cc M be/src/util/coding-util.h M be/src/util/collection-metrics.cc M be/src/util/collection-metrics.h M be/src/util/common-metrics.cc M be/src/util/common-metrics.h M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/condition-variable.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h M be/src/util/cyclic-barrier.cc M be/src/util/cyclic-barrier.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/decimal-util.cc M be/src/util/decimal-util.h M be/src/util/decompress.cc M be/src/util/decompress.h M be/src/util/default-path-handlers.cc M be/src/util/disk-info.cc M be/src/util/disk-info.h M be/src/util/dynamic-util.cc M be/src/util/event-metrics.cc M be/src/util/event-metrics.h M be/src/util/filesystem-util.cc M be/src/util/flat_buffer.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/internal-queue-test.cc M be/src/util/mem-info.h M be/src/util/memory-metrics.h M be/src/util/memusage-path-handlers.cc M be/src/util/metrics.h M be/src/util/parse-util.h M be/src/util/periodic-counter-updater.h M be/src/util/process-state-info.h M be/src/util/progress-updater.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.h M be/src/util/sharded-query-map-util.h M be/src/util/simple-logger.h M be/src/util/stopwatch.h M be/src/util/streaming-sampler.h M be/src/util/string-parser-test.cc M be/src/util/system-state-info.h M be/src/util/thread-pool-test.cc M be/src/util/thread-pool.h M be/src/util/uid-util-test.cc A be/src/util/uid-util.cc M be/src/util/uid-util.h A be/src/util/unique-id-hash.h M be/src/util/webserver.h 169 files changed, 700 insertions(+), 732 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3450e0ffcb8b183e18ac59c8b33b9ecbd3f60e20 Gerrit-Change-Number: 15202 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
