[
https://issues.apache.org/jira/browse/IMPALA-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated IMPALA-9664:
----------------------------------------
Description:
According to what we see in Hive source code, for transactional tables, the
insert events are fired with a different API {{addWriteNotificationLog}}.
Currently Impala fires {{firelistenerEvent}} for both transactional and
non-transactional tables. We should look at what is the difference between the
two APIs and see if we need to handle transactional tables differently.
References:
https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2402
https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2236
These insert events are used to replicate the changes in the ACID tables by
tools like Hive replication. With the ability of insert data into ACID tables
from Impala, we should also generate the insert events appropriately so that
the replication works seemlessly. Additionally, the {{truncate table}} command
should use the HMS API to truncate the table instead of deleteing the files
directly from filesystem since it takes care of moving the files to a
replication change management directory so that replication can have access to
dropped data files.
was:
According to what we see in Hive source code, for transactional tables, the
insert events are fired with a different API {{addWriteNotificationLog}}.
Currently Impala fires {{firelistenerEvent}} for both transactional and
non-transactional tables. We should look at what is the difference between the
two APIs and see if we need to handle transactional tables differently.
References:
https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2402
https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2236
> Support Hive replication
> ------------------------
>
> Key: IMPALA-9664
> URL: https://issues.apache.org/jira/browse/IMPALA-9664
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Critical
>
> According to what we see in Hive source code, for transactional tables, the
> insert events are fired with a different API {{addWriteNotificationLog}}.
> Currently Impala fires {{firelistenerEvent}} for both transactional and
> non-transactional tables. We should look at what is the difference between
> the two APIs and see if we need to handle transactional tables differently.
> References:
> https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2402
> https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2236
> These insert events are used to replicate the changes in the ACID tables by
> tools like Hive replication. With the ability of insert data into ACID tables
> from Impala, we should also generate the insert events appropriately so that
> the replication works seemlessly. Additionally, the {{truncate table}}
> command should use the HMS API to truncate the table instead of deleteing the
> files directly from filesystem since it takes care of moving the files to a
> replication change management directory so that replication can have access
> to dropped data files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
