Hello Philip Zeyliger,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8260
to look at the new patch set (#4).
Change subject: IMPALA-6052: Change HDFS layout for test tables
......................................................................
IMPALA-6052: Change HDFS layout for test tables
Currently, every table in Impala's test data has an HDFS
directory directly underneath /test-warehouse. Tables from
different databases are distinguished by their directory
name.
functional.alltypes => /test-warehouse/alltypes
functional_parquet.alltypes => /test-warehouse/alltypes_parquet
This makes the HDFS filesystem difficult to navigate.
The /test-warehouse directory has 800+ subdirectories. Listing
only the tables from a single database is difficult.
This adds a level to the directory structure for the database
and places all table directories in these databases directories.
This corresponds to the default placement for Impala-created
tables when LOCATION is not specified.
functional.alltypes => /test-warehouse/functional.db/alltypes
functional_parquet.alltypes => /test-warehouse/functional_parquet.db/alltypes
After this change, the /test-warehouse directory now has about 60
subdirectories.
In existing dataload, some tables are created at the top
level /test-warehouse/* and some are created under the
database directories. Some have hdfs directory names different
from the table name. This corrects those idiosyncracies and
makes all table locations consistent. This allows for consistent
detection of tables with existing data.
This directory structure change required updating paths in
a variety of tests.
Testing:
Ran exhaustive tests with full dataload.
Verified single_node_perf_run.py still works.
Change-Id: I3ba27ba6d3c7e445795e750281070963bbe1bb51
---
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/resources/authz-policy.ini.template
M testdata/avro_schema_resolution/create_table.sql
M testdata/bin/create-load-data.sh
M testdata/bin/create-table-many-blocks.sh
M testdata/bin/generate-schema-statements.py
M testdata/bin/load-dependent-tables.sql
M testdata/common/text_delims_table.py
M testdata/datasets/functional/functional_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M
testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M
testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/s3.test
M testdata/workloads/functional-planner/queries/PlannerTest/topn.test
M testdata/workloads/functional-planner/queries/PlannerTest/union.test
M testdata/workloads/functional-query/queries/DataErrorsTest/avro-errors.test
M
testdata/workloads/functional-query/queries/QueryTest/alter-table-set-column-stats.test
M testdata/workloads/functional-query/queries/QueryTest/alter-table.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M
testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test
M
testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test
M
testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test
M
testdata/workloads/functional-query/queries/QueryTest/parquet-continue-on-error.test
M
testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test
M testdata/workloads/functional-query/queries/QueryTest/parquet.test
M testdata/workloads/functional-query/queries/QueryTest/show-stats.test
M testdata/workloads/functional-query/queries/QueryTest/show.test
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_load.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_stale_metadata.py
M tests/query_test/test_chars.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_file_mods.py
41 files changed, 776 insertions(+), 811 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/8260/4
--
To view, visit http://gerrit.cloudera.org:8080/8260
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3ba27ba6d3c7e445795e750281070963bbe1bb51
Gerrit-Change-Number: 8260
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
