Hello Vihang Karajgaonkar, Sudhanshu Arora,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/13127
to review the following change.
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
IMPALA-8454 (part 3): enable recursive file listing by default
This enables recursive listing of files within partition directories by
default. This is a behavior change, but in fact makes Impala consistent
with modern versions of Hive, Spark, Presto, etc.
In fact, this is necessary for querying certain Hive tables which have
been written out by a query containing a UNION ALL clause if that query
is executed by Tez (see HIVE-12812 for example).
Technically, this is an incompatible change. Although it's unlikely
people were relying on the non-recursive listing, this patch offers two
escape hatches:
- an individual table may be marked with the
'impala.disable.recursive.listing' property
- impala may be globally configured with
--recursively_list_partitions=false
Given that we know this behavior is inconsistent with other SQL engines,
and that there is no performance benefit to not recursing in the common
case that there _are_ no subdirectories, I made the flag "hidden" and
did not document the new table property. These are only "chicken bit"
flags.
Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A tests/metadata/test_recursive_listing.py
8 files changed, 150 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/1
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Sudhanshu Arora <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
