[
https://issues.apache.org/jira/browse/ATLAS-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320301#comment-17320301
]
ASF subversion and git services commented on ATLAS-4204:
--------------------------------------------------------
Commit 6a7e2e41188ca8545c061f34e1b16fbe84c46a8d in atlas's branch
refs/heads/branch-2.0 from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6a7e2e4 ]
ATLAS-4204: Configure HS2 hook to send lineage-only messages. Part-2
> Hive Hook: Configure HiveServer2 Hook to send Lineage-only Messages
> -------------------------------------------------------------------
>
> Key: ATLAS-4204
> URL: https://issues.apache.org/jira/browse/ATLAS-4204
> Project: Atlas
> Issue Type: Improvement
> Components: hive-integration
> Affects Versions: 2.1.0
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> *Background*
> HiveServer2 hook for Atlas sends notification message for both metadata (DDL
> operations) and lineage (DML operations).
> Hive Metastore (HMS) hook already sends metadata information to Atlas. These
> messages are all DDL operations.
> So duplicate messages about object updates are sent to Atlas.
> Atlas processes these messages like any other.
> This is additional processing time and increased volume. There is also a
> potential of incorrect data being updated within Atlas if the sequence of
> messages from HMS and HS2 gets changed.
> *Solution*
> This improvement will send only lineage messages from HS2 hook. All the DDL
> (schema definition) messages will continue be sent from HMS hook (no change
> here).
> This will also reduce the volume of messages sent to Atlas from hive server2
> and will help improve performance by avoiding processing duplicate messages.
> The improvement can be used via a configuration parameter. That way existing
> behavior continues as is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
