Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11191 )
Change subject: IMPALA-7408: add a debugging flag to disable reading fs data from catalogd ...................................................................... IMPALA-7408: add a debugging flag to disable reading fs data from catalogd Add the flag: --disable_catalog_data_ops_debug_only that skips loading files from the file-system from catalogd. The flag is by default false and its hidden. Its intent is to avoid time-consuming accesses to the file-system when debugging metadata issues and the file-system contents are not available. For example, a recent ~18 GB catalog takes 10 hours to load without the flag set vs. 1 hour to load with the flag. The extra time comes from accessing the file-system, failing, and logging exceptions. This flag specifically disables copying jars from the fs when loading Java functions and it skips loading avro schema files. Additional cases can be added under this flag if more are needed. Testing: - manually confirmed that jars and avro schema files are skipped. - added a test to check the same behavior in a custom cluster test. - ran core tests. Change-Id: I15789fb489b285e2a6565025eb17c63cdc726354 Reviewed-on: http://gerrit.cloudera.org:8080/11191 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/AvroSchemaUtils.java M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_disable_catalog_data_ops.py 8 files changed, 94 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I15789fb489b285e2a6565025eb17c63cdc726354 Gerrit-Change-Number: 11191 Gerrit-PatchSet: 9 Gerrit-Owner: Vuk Ercegovac <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
