[
https://issues.apache.org/jira/browse/IMPALA-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098515#comment-18098515
]
ASF subversion and git services commented on IMPALA-10319:
----------------------------------------------------------
Commit 60a0e9a03e382c63d8be479c83bf46a74920e284 in impala's branch
refs/heads/master from Daniel Vanko
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=60a0e9a03 ]
IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).
Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.
The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.
Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24585
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Support arbitrary encodings on Text files
> -----------------------------------------
>
> Key: IMPALA-10319
> URL: https://issues.apache.org/jira/browse/IMPALA-10319
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Mihaly Szjatinya
> Priority: Critical
> Attachments: gbk_names.txt
>
>
> ORC/Parquet/Avro files store strings in UTF-8 encoded bytes. However, Text
> and Sequence files can be in arbitrary encodings. Hive supports specifying
> arbitrary encoding on tables using LazySimpleSerDe with the
> "serialization.encoding" table property (HIVE-7142). Impala is currently not
> aware of this table property and treate all strings as byte arrays. It's good
> to support at least reading from these text/sequence files.
> *Example*
> Create a text table in Hive using GBK encoding and load a GBK encoded text
> file into it:
> {code:sql}
> hive> create table gbk_names (name string) stored as textfile
> tblproperties("serialization.encoding"="GBK");
> hive> load data local inpath '/home/quanlong/workspace/Impala/gbk_names.txt'
> into table gbk_names;
> hive> select * from gbk_names;
> +-----------------+
> | gbk_names.name |
> +-----------------+
> | 张三 |
> | 李四 |
> | 王五 |
> +-----------------+
> {code}
> Impala read strings as byte arrays so can't decode them correctly:
> {code:sql}
> impala-shell> invalidate metadata gbk_names;
> impala-shell> select * from gbk_names;
> +------+
> | name |
> +------+
> | ���� |
> | ���� |
> | ���� |
> +------+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
