Daniel Vanko has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24585 )
Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1 ...................................................................... IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1 The TPC-DS data generator (dsdgen) emits some string columns in ISO-8859-1 (Latin-1) rather than UTF-8. For example, customer.c_birth_country holds country names whose accented letters are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE D'IVOIRE") that are not valid UTF-8, which will cause data-load failure if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675). Tag the TPC-DS text tables with the serde property 'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data files to UTF-8 on read -- including when the derived Parquet (and other format) tables are populated from the text tables via INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and this exercises the serialization.encoding feature (IMPALA-10319) in the standard data load. The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the LOAD section rather than at CREATE: Impala's CREATE does not accept a custom SERDE, and 'serialization.encoding' is honored only from serde properties (not TBLPROPERTIES). The LOAD section is the text-only load phase, so the ALTER runs before the derived tables read the data. Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c Assisted-by: Claude Opus 4.8 (Claude Code) --- M testdata/datasets/tpcds/tpcds_schema_template.sql 1 file changed, 24 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24585/3 -- To view, visit http://gerrit.cloudera.org:8080/24585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c Gerrit-Change-Number: 24585 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
