Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5527: Add nested testdata flattener ......................................................................
IMPALA-5527: Add nested testdata flattener The TableFlattener takes a nested dataset and creates an equivalent unnested dataset. The unnested dataset is saved as Parquet. When an array or map is encountered in the original table, the flattener creates a new table and adds an id column to it which references the row in the parent table. Joining on the id column should produce the original dataset. The flattened dataset should be loaded into Postgres in order to run the query generator (in nested types mode) on it. There is a script that automates generaration, flattening and loading random data into Postgres and Impala: testdata/bin/generate-load-nested.sh -f Testing: - ran ./testdata/bin/generate-load-nested.sh -f and random nested data was generated and flattened as expected. Change-Id: I7e7a8e53ada9274759a3e2128b97bec292c129c6 Reviewed-on: http://gerrit.cloudera.org:8080/5787 Reviewed-by: Alex Behm <[email protected]> Tested-by: Impala Public Jenkins --- A testdata/TableFlattener/.gitignore A testdata/TableFlattener/README A testdata/TableFlattener/pom.xml A testdata/TableFlattener/src/main/java/org/apache/impala/infra/tableflattener/FileMigrator.java A testdata/TableFlattener/src/main/java/org/apache/impala/infra/tableflattener/FlattenedSchema.java A testdata/TableFlattener/src/main/java/org/apache/impala/infra/tableflattener/Main.java A testdata/TableFlattener/src/main/java/org/apache/impala/infra/tableflattener/SchemaFlattener.java A testdata/TableFlattener/src/main/java/org/apache/impala/infra/tableflattener/SchemaUtil.java M testdata/bin/generate-load-nested.sh 9 files changed, 863 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/5787 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7e7a8e53ada9274759a3e2128b97bec292c129c6 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown <[email protected]> Gerrit-Reviewer: Taras Bobrovytsky <[email protected]>
