Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )
Change subject: IMPALA-5961: Convert generated TPC-DS test data to UTF-8 ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload File testdata/datasets/tpcds/preload: http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@116 PS1, Line 116: Did you check how much time this step takes? http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@117 PS1, Line 117: # dsdgen emits string data in ISO-8859-1 (Latin-1), which can contain bytes that are : # not valid UTF-8 (e.g. the 0xD4 in customer.c_birth_country). Convert the generated : # files to UTF-8 so the loaded TPC-DS data is valid Unicode (IMPALA-5961). : echo "Converting generated data files from ISO-8859-1 to UTF-8" : for data_file in *.dat; do : iconv -f ISO-8859-1 -t UTF-8 "${data_file}" -o "${data_file}.utf8" : mv "${data_file}.utf8" "${data_file}" : done An alternative would be do the conversion when reading with Impala. This can be done by setting TBLPROPERTIES("serialization.encoding"="ISO-8859-1") for the text tables. I would prefer this approach because it would give more testing for this relatively new encoding feature. -- To view, visit http://gerrit.cloudera.org:8080/24585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c Gerrit-Change-Number: 24585 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Mihaly Szjatinya <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Comment-Date: Wed, 15 Jul 2026 15:56:09 +0000 Gerrit-HasComments: Yes
