Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24585 )
Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1 ...................................................................... IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1 The TPC-DS data generator (dsdgen) emits some string columns in ISO-8859-1 (Latin-1) rather than UTF-8. For example, customer.c_birth_country holds country names whose accented letters are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE D'IVOIRE") that are not valid UTF-8, which will cause data-load failure if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675). Tag the TPC-DS text tables with the serde property 'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data files to UTF-8 on read -- including when the derived Parquet (and other format) tables are populated from the text tables via INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and this exercises the serialization.encoding feature (IMPALA-10319) in the standard data load. The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the LOAD section rather than at CREATE: Impala's CREATE does not accept a custom SERDE, and 'serialization.encoding' is honored only from serde properties (not TBLPROPERTIES). The LOAD section is the text-only load phase, so the ALTER runs before the derived tables read the data. Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c Assisted-by: Claude Opus 4.8 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24585 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M testdata/datasets/tpcds/tpcds_schema_template.sql M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test 3 files changed, 28 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c Gerrit-Change-Number: 24585 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
