[
https://issues.apache.org/jira/browse/IMPALA-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894780#comment-16894780
]
ASF subversion and git services commented on IMPALA-8636:
---------------------------------------------------------
Commit 6360657cb4d3b7655d9ff80958b2694ae4609370 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6360657 ]
IMPALA-8636: Implement INSERT for insert-only ACID tables
This commit adds INSERT support for insert-only ACID tables.
The Frontend opens a transaction for INSERT statements when the target
table is transactional. It also allocates a write ID for the target
table. The Frontend aborts the transaction if an error occurs during
analysis/planning.
The Backend gets the transaction id and the write id in TFinalizeParams.
The write id is also set the for the HDFS table sinks. The sinks write
the files at their final destination which is an ACID base or delta
directory. There is no need for finalization of transactional INSERTS.
When the sinks finished with writing the data, the Coordinator invokes
updateCatalog() on catalogd which also commits the transaction if
everything went well, otherwise the Coordinator aborts the transaction.
Testing:
* added new tables during dataload
* added acid-insert.test file with INSERT statements against the new
tables
* test insertions between ACID and non-ACID tables
* test error scenarios via debug actions
* added integration test with Hive to test_hms_integration.py. The test
inserts data with Impala and reads with Hive. (These integration
tests only run with exhaustive exploration strategy)
TODO in following commits:
* add locks and heartbeats (without heartbeats long-running transactions
might be aborted by HMS)
* implement TRUNCATE
* CTAS creates files in the 'root' directory of the table/partition. It
is handled correctly during SELECT, but would be better to create a
base directory from the beginning. Hive creates a delta directory
for CTAS.
Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad
Reviewed-on: http://gerrit.cloudera.org:8080/13559
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Implement INSERT for insert-only ACID tables
> --------------------------------------------
>
> Key: IMPALA-8636
> URL: https://issues.apache.org/jira/browse/IMPALA-8636
> Project: IMPALA
> Issue Type: New Feature
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: impala-acid
>
> Impala should support insertion for insert-only ACID tables.
> For this we need to allocate a write ID for the target table, and write the
> data into the base/delta directories.
> INSERT operation should create a new delta directory with the allocated write
> ID.
> INSERT OVERWRITE should create a new base directory with the allocated write
> ID. This new base directory will only contain the data coming from this
> operation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
