Skye Wanderman-Milne has posted comments on this change. Change subject: IMPALA-3092: Avro table un-queriable after add columns without a default value ......................................................................
Patch Set 1: (6 comments) http://gerrit.cloudera.org:8080/#/c/2361/1//COMMIT_MSG Commit Message: Line 8: without a default value Instead of copying the JIRA title, describe what this patch actually does. e.g. IMPALA-3092: return NULL for Avro fields missing defaults Line 11: ****(can someone help fill this?) slots. With this patch We used to require that any missing fields missing in an Avro file have a corresponding default value in the table schema. http://gerrit.cloudera.org:8080/#/c/2361/1/be/src/exec/hdfs-avro-scanner.cc File be/src/exec/hdfs-avro-scanner.cc: Line 217: "WriteDefaultValue() doesn't support default records yet, should have failed"; You'll hit this DCHECK if there's a missing record with a default value, since WriteDefaultValue() will return without an error. I would add the following above the WriteDefaultValue() call: if (is_avro_complex_type(table_record->schema)) { return Status(TErrorCode::AVRO_UNSUPPORTED_DEFAULT_VALUE, field_name, avro_type_name(table_record->schema->type)); } We should add a test case for this if we don't already. Line 252: if (!default_value) { if (default_value == NULL) Line 253: RawValue::Write(NULL, avro_header_->template_tuple, slot_desc, NULL); We usually do template_tuple->SetNull(slot_desc->null_indicator_offset()) directly http://gerrit.cloudera.org:8080/#/c/2361/1/tests/query_test/test_avro.py File tests/query_test/test_avro.py: Line 12: class TestAvroAddColumn(ImpalaTestSuite): Instead of creating a new test class, let's add to query_test/test_avro_schema_resolution.py. Check out $IMPALA_HOME/testdata/avro_schema_resolution. In create_table.sql, the schema_resolution_test table defines default values for every column, and then contains a file with extra columns and file with missing columns. You should add a new similar table that's the same except it doesn't contain any default values, and then add queries to avro-schema-resolution.test that select from this table. Please add a record field to this table as well. -- To view, visit http://gerrit.cloudera.org:8080/2361 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ie86421fc3da51086e566998af317faa62ba9789b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Huaisi Xu <[email protected]> Gerrit-Reviewer: Skye Wanderman-Milne <[email protected]> Gerrit-HasComments: Yes
