Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default ...................................................................... IMPALA-8454 (part 3): enable recursive file listing by default This enables recursive listing of files within partition directories by default. This is a behavior change, but in fact makes Impala consistent with modern versions of Hive, Spark, Presto, etc. In fact, this is necessary for querying certain Hive tables which have been written out by a query containing a UNION ALL clause if that query is executed by Tez (see HIVE-12812 for example). Technically, this is an incompatible change. Although it's unlikely people were relying on the non-recursive listing, this patch offers two escape hatches: - an individual table may be marked with the 'impala.disable.recursive.listing' property - impala may be globally configured with --recursively_list_partitions=false Given that we know this behavior is inconsistent with other SQL engines, and that there is no performance benefit to not recursing in the common case that there _are_ no subdirectories, I made the flag "hidden" and did not document the new table property. These are only "chicken bit" flags. Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Reviewed-on: http://gerrit.cloudera.org:8080/13127 Reviewed-by: Bharath Vissapragada <bhara...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A tests/metadata/test_recursive_listing.py 8 files changed, 151 insertions(+), 2 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Sudhanshu Arora <sudhan...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
