[
https://issues.apache.org/jira/browse/IMPALA-13748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955583#comment-17955583
]
ASF subversion and git services commented on IMPALA-13748:
----------------------------------------------------------
Commit 4837cedc795017aa1e7b69ef0914020a3022ca88 in impala's branch
refs/heads/master from Mihaly Szjatinya
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4837cedc7 ]
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Reviewed-on: http://gerrit.cloudera.org:8080/22049
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Make Serde property handling consistent with Hive
> -------------------------------------------------
>
> Key: IMPALA-13748
> URL: https://issues.apache.org/jira/browse/IMPALA-13748
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Csaba Ringhofer
> Priority: Critical
>
> Hive seems to both consider serde and table properties with the same name for
> serde properties, while Impala only considers serde properties:
> {code}
> -- from Impala
> create table t (i int, s string);
> insert into t values (1, "a");
> alter table t set tblproperties ('field.delim' = ':');
> refresh t;
> select * from t; -- returns 1, "a", which means that 'field.delim' was
> ignored
> -- from Hive
> select * from t -- returns NULL, NULL, which means that 'field.delim' was
> respected
> -- from Impala
> alter table t set serdeproperties ('field.delim' = ':');
> refresh t; -- for some reason refresh is needed for serdeproperty change to
> take effect
> select * from t; -- returns NULL, NULL + warnings
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
