Anurag Mantripragada has uploaded a new patch set (#16). ( 
http://gerrit.cloudera.org:8080/12889 )

Change subject: IMPALA-7971: Add support for insert events in event processor.
......................................................................

IMPALA-7971: Add support for insert events in event processor.

This patch adds support for detecting and processing insert events
triggered by impala as well as external engines (eg.Hive).

Inserts from Impala will fire an insert event notification.
Using this event, event-processor will refresh table/partition.
Both insert into and overwrite are supported for tables/partitions.

Known Issues:
1. Inserts into tables from Hive are ignored by the event processor
   as these inserts create an ALTER event first followed by an
   INSERT event. The alter will invalidate table making the refresh
   a no-op. Insert into partitions from hive will create an INSERT
   event first followed by an ALTER event. In this case, there is
   an unnecessary table invalidate after a refresh.
2. Existing self-events logic cannot be used for insert events since
   firing insert event does not allow us to modify table parameters in
   HMS. This means we cannot get the CatalogServiceIdentifiers in insert
   events. Therefore, the event-processor will also refresh the tables
   for which insert operation is performed through Impala.

Testing:
1. Added new custom cluster tests to run different insert commands from
hive and verified new data is available in Impala without invalidate
metadata.

2. Added a test in MetastoreEventsProcessor for testing insert events.

Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A tests/custom_cluster/test_event_processing.py
9 files changed, 528 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/16
--
To view, visit http://gerrit.cloudera.org:8080/12889
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
Gerrit-Change-Number: 12889
Gerrit-PatchSet: 16
Gerrit-Owner: Anurag Mantripragada <anu...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bhar...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <prog...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>

Reply via email to