This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit d1d28c0eec52c212b4b8cd09080327c542f24bfa
Author: Joe McDonnell <joemcdonn...@cloudera.com>
AuthorDate: Tue May 7 12:50:23 2024 -0700
IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
HIVE-24157 introduces a restriction to prohibit casting DATE/TIMESTAMP
types to and from NUMERIC. It's enabled by default and can be turned off
by set hive.strict.timestamp.conversion=false.
This restriction breaks the data loading on avro_coldef and
avro_extra_coldef tables, which results in empty data set and finally
fails TestAvroSchemaResolution.test_avro_schema_resolution.
This patch explicitly disables the restriction in loading these two avro
tables.
The Hive version currently used for development does not have HIVE-24157,
but upstream Hive does have it. Adding hive.strict.timestamp.conversion
does not cause problems for Hive versions that don't have HIVE-24157.
Tests:
- Run the data loading and test_avro_schema_resolution locally using
a Hive that has HIVE-24157.
- Run CORE tests
- Run data loading with a Hive that doesn't have HIVE-24157.
Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
Reviewed-on: http://gerrit.cloudera.org:8080/21413
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Joe McDonnell <joemcdonn...@cloudera.com>
---
testdata/avro_schema_resolution/create_table.sql | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/testdata/avro_schema_resolution/create_table.sql
b/testdata/avro_schema_resolution/create_table.sql
index 27ae8e97c..01a31fdb7 100644
--- a/testdata/avro_schema_resolution/create_table.sql
+++ b/testdata/avro_schema_resolution/create_table.sql
@@ -166,6 +166,10 @@ TBLPROPERTIES ('avro.schema.literal'='{
-- any preexisting data files, which in turn will fail the query.
MSCK REPAIR TABLE avro_coldef;
+-- Disable the restriction of HIVE-24157, otherwise casting TIMESTAMP to
BIGINT is
+-- prohibited. Note that type of timestamp_col is long in
'avro.schema.literal'.
+SET hive.strict.timestamp.conversion=false;
+
INSERT OVERWRITE TABLE avro_coldef PARTITION(year=2014, month=1)
SELECT bool_col, tinyint_col, smallint_col, int_col, bigint_col,
float_col, double_col, date_string_col, string_col, timestamp_col
