[
https://issues.apache.org/jira/browse/HAWQ-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei Chang updated HAWQ-315:
---------------------------
Fix Version/s: 2.0.0
> Invalid Byte Sequence Error when loading a large(100MB+) csv file
> -----------------------------------------------------------------
>
> Key: HAWQ-315
> URL: https://issues.apache.org/jira/browse/HAWQ-315
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
> Reporter: Goden Yao
> Assignee: Ruilong Huo
> Fix For: 2.0.0
>
>
> This bug occurs when copying or reading a large csv file. The reproducible
> file we tried is 100MB+ so cannot be uploaded to JIRA.
> *Repro steps*
> The large CSV file size needs to be over 100MB at least.
> The pattern in the csv file should contain the following:
> {code:actionscript}
> ..., "dummy data text1
> dummy data text2,
> dummy data text3,
> dummy data text4"
> {code}
> basically , a long text broken into multiple lines but within quotes.
> This doesn't cause issue in a smaller size file though.
> {code:SQL}
> DROP TABLE IF EXISTS <_test table name_>;
> CREATE TABLE <_test table name_>
> (
> <_define test table schema_>
> ...
> );
> COPY <_test table name_> FROM '<_csv file path_>'
> DELIMITER ','
> NULL ''
> ESCAPE '"'
> CSV QUOTE '"'
> LOG ERRORS INTO <_error reject table name_>
> SEGMENT REJECT LIMIT 10 rows
> ;
> {code}
> *Errors*
> Error in first line with quoted data:
> {code}
> DEBUG5: invalid byte sequence for encoding "UTF8": 0x00
> HINT: This error can also happen if the byte sequence does not match the
> encoding expected by the server, which is controlled by "client_encoding".
> CONTEXT: COPY addresses_heap, line 604932
> {code}
> Error in second line with quoted data: this is due to wrong formatting (as
> the first half line within the quotes was mishandled.
> {code}
> DEBUG5: missing data for column "sourceid"
> CONTEXT: COPY addresses_heap, line 604933: "...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
