Jim Apple has submitted this change and it was merged. Change subject: IMPALA-3776: fix 'describe formatted' for Avro tables ......................................................................
IMPALA-3776: fix 'describe formatted' for Avro tables For Avro tables the column information in the underlying database of the Hive metastore can be different from what is specified in the avro schema. HIVE-6308 aimed to improve upon this, but for older tables the two don't necessarily align. There are two possible cases: 1) Hive's underlying database contains a column which is not present in the Avro schema file. In this case we encounter a NullPointerException in DescribeResultFactory.java#L189 when trying to look up the column in the internal table object. 2) The Avro schema contains a column, which is not present in the underlying database. In this case the column will not be displayed in describe formatted. In addition to the automatic tests I verified this manually by creating an Avro table with an external schema file in Hive. This populated the underlying database with the column information. I then either removed a column from the Avro schema file (case 1) or cleared the column information from the "COLUMNS_V2" table in the underlying database (case 2) and verified that the change fixed both cases. Change-Id: Ieb69d3678e662465d40aee80ba23132ea13871a0 Reviewed-on: http://gerrit.cloudera.org:8080/4126 Reviewed-by: Lars Volker <[email protected]> Tested-by: Internal Jenkins Reviewed-by: Jim Apple <[email protected]> --- M fe/src/main/java/com/cloudera/impala/catalog/Column.java M fe/src/main/java/com/cloudera/impala/catalog/Table.java M fe/src/main/java/com/cloudera/impala/service/DescribeResultFactory.java M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test M tests/query_test/test_avro_schema_resolution.py 5 files changed, 101 insertions(+), 15 deletions(-) Approvals: Jim Apple: Looks good to me, approved Lars Volker: Looks good to me, but someone else must approve Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/4126 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ieb69d3678e662465d40aee80ba23132ea13871a0 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Jim Apple <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]>
