Hello Vihang Karajgaonkar,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/13251
to review the following change.
Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset
loading
......................................................................
IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
This fixes three issues for functional dataset loading:
- works around HIVE-21675, a bug in which 'CREATE VIEW IF NOT EXISTS'
does not function correctly in our current Hive build. This has been
fixed already, but the workaround is pretty simple, and actually the
'drop and recreate' pattern is used more widely for data-loading than
the 'create if not exists' one.
- adds the ability to specify version restrictions for tables to load.
The restrictions use the Python "requirements.txt" syntax. This new
functionality is used to skip creating a hive "INDEX" table on Hive 3,
where this functionality has been removed.
- Moving from MR to Tez execution changed the behavior of data loading by
disabling the auto-merging of small files. With Hive-on-MR, this
behavior defaulted to true, but with Hive-on-Tez it defaults false. The
change is likely motivated by the fact that Tez automatically groups
small splits on the _input_ side and thus is less likely to produce lots
of small files. However, that grouping functionality doesn't work
properly in localhost clusters (TEZ-3310) so we aren't seeing the
benefit. So, this patch enables the post-process merging of small
files.
Prior to this change, the 'alltypesaggmultifilesnopart' test table was
getting 40+ files inside it, which broke various planner tests. With the
change, it gets the expected 4 files.
Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd
---
M fe/src/test/resources/hive-site.xml.py
M testdata/bin/generate-schema-statements.py
M testdata/bin/load-dependent-tables.sql
M testdata/datasets/README
M testdata/datasets/functional/functional_schema_template.sql
5 files changed, 119 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13251/1
--
To view, visit http://gerrit.cloudera.org:8080/13251
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd
Gerrit-Change-Number: 13251
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
