[
https://issues.apache.org/jira/browse/IMPALA-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204114#comment-17204114
]
ASF subversion and git services commented on IMPALA-9664:
---------------------------------------------------------
Commit e53d649f8a88f42a70237fe7c2663baa126fed1a in impala's branch
refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e53d649 ]
IMPALA-9664: Support hive replication
This patch makes some improvements to the INSERT
event generated by Impala. Specifically, the INSERT event
will now include new file information when Impala inserts
into a table. This information can be used by external
tools like Hive Replication to replicate the changes made
by Impala in their source databases.
Additionally, this patch modifies the truncate table
execution so that it uses HMS API to truncate the table
instead of deleting the files directly on the filesystem.
Following changes were made.
1. Fires insert events for insert overwrite.
2. Has the names of the new files in the events. In case of insert
overwrite, this is just a list of files which were added by
the insert overwrite operation.
3. In case of ACID tables, fires transactional notification API
for all the partitions in which data is inserted.
4. For tables which have replication enabled, the truncate table
operation now uses a HMS API to truncate the table. This is
necessary since HMS API moves the files to a replication change
manager location if needed. Additionally, it generates
ALTER_TABLE events with truncate flag set to true.
TODO:
1. For external tables, replication does not seem to work in the
dev environment. This will be done as a followup.
Testing:
1. Created a new test in test_events_processing.py which inserts
into managed tables which are being replicated. It makes sure that
hive replication detects the new rows which are added into the
tables. The test also exercises insert overwrite and truncate
statements and makes sure that the table is replicated correctly.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Reviewed-on: http://gerrit.cloudera.org:8080/16439
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Vihang Karajgaonkar <[email protected]>
> Support Hive replication for ACID tables
> ----------------------------------------
>
> Key: IMPALA-9664
> URL: https://issues.apache.org/jira/browse/IMPALA-9664
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Critical
> Fix For: Impala 4.0
>
>
> According to what we see in Hive source code, for transactional tables, the
> insert events are fired with a different API {{addWriteNotificationLog}}.
> Currently Impala fires {{firelistenerEvent}} for both transactional and
> non-transactional tables. We should look at what is the difference between
> the two APIs and see if we need to handle transactional tables differently.
> References:
> https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2402
> https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2236
> These insert events are used to replicate the changes in the ACID tables by
> tools like Hive replication. With the ability of insert data into ACID tables
> from Impala, we should also generate the insert events appropriately so that
> the replication works seemlessly. Additionally, the {{truncate table}}
> command should use the HMS API to truncate the table instead of deleteing the
> files directly from filesystem since it takes care of moving the files to a
> replication change management directory so that replication can have access
> to dropped data files.
> Note that for external tables, Hive replication doesn't need to keep track of
> the files. It only replicates the table metadata based on events and the data
> files are "distcped" to the target cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
