Michael Brown has uploaded a new patch set (#4). Change subject: IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL ......................................................................
IMPALA-4338: test infra data migrator: include tables' primary keys in PostgreSQL This patch adds the ability for the test infrastructure's Impala-to-PostgreSQL data migration tool to recognize whether the Impala source tables have primary keys, and if so, CREATE the tables in PostgreSQL with the same primary keys. This is needed especially for performing CRUD operations by the random query generator for comparison with Impala/Kudu tables and equivalent PostgreSQL tables. I modified the make_create_table_sql() implementation to check the "universal" Python object model of the table's columns. We generate CREATE TABLE statements with, or without, a PRIMARY KEY clause. For Impala-side tables that this tool may create, we also ensure that we only write such a clause when the table's format supports primary keys (currently Kudu). When the random query generator runs, it needs to know that the tables it's examining in both databases are equivalent. It does this by examining the tables' names, column names, and column types. I have added whether the column is a primary key as part of this equivalence test. Testing: - The patch includes some unit and system tests for the tool. - Actually migrated a few small Kudu and HDFS tables from Impala into both PostgreSQL 9.3 and 9.5 and examined the tables in PostgreSQL to make sure they had primary keys (or not) as expected. - Very short discrepancy_searcher.py --explain-only runs to test positive and negative cases of Impala/PostgreSQL equivalency. Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8 --- M tests/comparison/db_connection.py M tests/comparison/tests/fake_query.py A tests/comparison/tests/test_cursor.py 3 files changed, 271 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/4951/4 -- To view, visit http://gerrit.cloudera.org:8080/4951 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I447f022e2dc3d4fc8373b7f388c7875a869921b8 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Brown <mi...@cloudera.com> Gerrit-Reviewer: David Knupp <dkn...@cloudera.com> Gerrit-Reviewer: Michael Brown <mi...@cloudera.com> Gerrit-Reviewer: Taras Bobrovytsky <tbobrovyt...@cloudera.com>
