Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18811 )
Change subject: IMPALA-11469: Make prefix of ignored staging dirs configurable ...................................................................... IMPALA-11469: Make prefix of ignored staging dirs configurable External systems like Hive or Spark will write temporary or "non-data" files in the table location. Catalogd will skip them when loading file metadata. However, the prefix is currently hard coded. We recently found that Spark streaming will generated a _spark_metadata dir which is not handled correctly. To avoid future code changes when interact with more systems, this patch adds a new startup flag, ignored_dir_prefix_list, for catalogd. It's a comma separated list for the prefix of ignored dirs. Currently, the default value is ".,_tmp.,_spark_metadata". Users can add more in the future. Tests: - Add a case for _spark_metadata in FileSystemUtilTest Change-Id: I108bfa823281a35d28932f7ccce0b12a0c5af57d Reviewed-on: http://gerrit.cloudera.org:8080/18811 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java 5 files changed, 49 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I108bfa823281a35d28932f7ccce0b12a0c5af57d Gerrit-Change-Number: 18811 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
