[
https://issues.apache.org/jira/browse/IMPALA-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831286#comment-17831286
]
Zihao Ye commented on IMPALA-12927:
-----------------------------------
Hi, [~csringhofer] , thank you for the information and suggestions, they are
very helpful. I agree with the approach of checking the table property
"json.binary.format", it makes sense. I will take some time to investigate how
Hive handles rawstring. If there isn't a suitable method for handling it, I
think disabling binary columns in this case would be acceptable, as it should
be very rare to use a binary type to store string data.
> Support reading BINARY columns in JSON tables
> ---------------------------------------------
>
> Key: IMPALA-12927
> URL: https://issues.apache.org/jira/browse/IMPALA-12927
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Csaba Ringhofer
> Assignee: Zihao Ye
> Priority: Major
>
> Currently Impala cannot read BINARY columns in JSON files written by Hive
> correctly and returns runtime errors:
> {code}
> select * from functional_json.binary_tbl;
> +----+--------------+------------+
> | id | string_col | binary_col |
> +----+--------------+------------+
> | 1 | ascii | NULL |
> | 2 | ascii | NULL |
> | 3 | null | NULL |
> | 4 | empty | |
> | 5 | valid utf8 | NULL |
> | 6 | valid utf8 | NULL |
> | 7 | invalid utf8 | NULL |
> | 8 | invalid utf8 | NULL |
> +----+--------------+------------+
> WARNINGS: Error converting column: functional_json.binary_tbl.binary_col,
> type: STRING, data: 'binary1'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> Error converting column: functional_json.binary_tbl.binary_col, type: STRING,
> data: 'binary2'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> Error converting column: functional_json.binary_tbl.binary_col, type: STRING,
> data: 'árvíztűrőtükörfúró'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> Error converting column: functional_json.binary_tbl.binary_col, type: STRING,
> data: '你好hello'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> Error converting column: functional_json.binary_tbl.binary_col, type: STRING,
> data: '��'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> Error converting column: functional_json.binary_tbl.binary_col, type: STRING,
> data: '�D3"'
> Error parsing row: file:
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0, before
> offset: 481
> {code}
> The single file in the table looks like this:
> {code}
> hdfs://localhost:20500/test-warehouse/binary_tbl_json/000000_0
> {"id":1,"string_col":"ascii","binary_col":"binary1"}
> {"id":2,"string_col":"ascii","binary_col":"binary2"}
> {"id":3,"string_col":"null","binary_col":null}
> {"id":4,"string_col":"empty","binary_col":""}
> {"id":5,"string_col":"valid utf8","binary_col":"árvíztűrőtükörfúró"}
> {"id":6,"string_col":"valid utf8","binary_col":"你好hello"}
> {"id":7,"string_col":"invalid utf8","binary_col":"\u0000�\u0000�"}
> {"id":8,"string_col":"invalid utf8","binary_col":"�D3\"\u0011\u0000"}
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
