Goden Yao created HAWQ-315:
------------------------------
Summary: Invalid Byte Sequence Error when loading a large(100MB+)
csv file
Key: HAWQ-315
URL: https://issues.apache.org/jira/browse/HAWQ-315
Project: Apache HAWQ
Issue Type: Bug
Components: Core
Reporter: Goden Yao
Assignee: Lei Chang
This bug occurs when copying or reading a large csv file. The reproducible file
we tried is 100MB+ so cannot be uploaded to JIRA.
*repro steps*
The large CSV file size needs to be over 100MB at least.
The pattern in the csv file should contain the following:
{code:csv}
..., "dummy data text1
dummy data text2,
dummy data text3,
dummy data text4"
{code}
basically , a long text broken into multiple lines but within quotes.
This doesn't cause issue in a smaller size file though.
{code:SQL}
DROP TABLE IF EXISTS <_test table name_>;
CREATE TABLE <_test table name_>
(
<_define test table schema_>
...
);
COPY <_test table name_> FROM '<_csv file path_>'
DELIMITER ','
NULL ''
ESCAPE '"'
CSV QUOTE '"'
LOG ERRORS INTO <_error reject table name_>
SEGMENT REJECT LIMIT 10 rows
;
{code}
*Errors*
Error in first line with quoted data:
{code}
DEBUG5: invalid byte sequence for encoding "UTF8": 0x00
HINT: This error can also happen if the byte sequence does not match the
encoding expected by the server, which is controlled by "client_encoding".
CONTEXT: COPY addresses_heap, line 604932
{code}
Error in second line with quoted data: this is due to wrong formatting (as the
first half line within the quotes was mishandled.
{code}
DEBUG5: missing data for column "sourceid"
CONTEXT: COPY addresses_heap, line 604933: "...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
