[
https://issues.apache.org/jira/browse/HAWQ-956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Diachenko updated HAWQ-956:
-------------------------------------
Description:
As for now COPY command is transactional for native HAWQ tables, but it's not
for external tables.
This command involves communication with underlying HDFS layer which isn't
under HAWQ's control.
If something happens to HDFS during COPY data in table ending up being
corrupted.
STR:
{code}
1) Create two external tables:
create writable external table store_t ( a text, b text, c text, d text )
LOCATION ('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT'
(DELIMITER ',');
create external table read_t ( a text, b text, c text, d text ) LOCATION
('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT' (DELIMITER
',');
{code}
2) Copy big file(~ 1Gb) from local fs to store_t:
COPY store_table from '/tmp/data/1Gb.txt' DELIMITER ',';
3) Restart HDFS while COPY is in progress.
was:
As for now COPY command is transactional for native HAWQ tables, but it's not
for external tables.
This command involves communication with underlying HDFS layer which isn't
under HAWQ's control.
If something happens to HDFS during COPY data in table ending up being
corrupted.
STR:
{code}
1) Create two external tables:
create writable external table store_t ( a text, b text, c text, d text )
LOCATION ('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT'
(DELIMITER ',');
create external table read_t ( a text, b text, c text, d text ) LOCATION
('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT' (DELIMITER
',');
{code}
> Make COPY command transactional for external tables
> ---------------------------------------------------
>
> Key: HAWQ-956
> URL: https://issues.apache.org/jira/browse/HAWQ-956
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: External Tables, PXF
> Reporter: Oleksandr Diachenko
> Assignee: Goden Yao
>
> As for now COPY command is transactional for native HAWQ tables, but it's not
> for external tables.
> This command involves communication with underlying HDFS layer which isn't
> under HAWQ's control.
> If something happens to HDFS during COPY data in table ending up being
> corrupted.
> STR:
> {code}
> 1) Create two external tables:
> create writable external table store_t ( a text, b text, c text, d text )
> LOCATION ('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT'
> (DELIMITER ',');
> create external table read_t ( a text, b text, c text, d text ) LOCATION
> ('pxf://localhost:51200/data?Profile=HdfsTextSimple') FORMAT 'TEXT'
> (DELIMITER ',');
> {code}
> 2) Copy big file(~ 1Gb) from local fs to store_t:
> COPY store_table from '/tmp/data/1Gb.txt' DELIMITER ',';
> 3) Restart HDFS while COPY is in progress.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
