[
https://issues.apache.org/jira/browse/ATLAS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shwetha G S updated ATLAS-1111:
-------------------------------
Attachment: ATLAS-1111.3.patch
Patch with UTs fixed, and end to end tested
kafkaconsumer.next() always returns the next message irrespective of auto
commit or not. With auto commit disabled, the the offset has to be committed
manually, as opposed to auto commit on next(). The last saved offset is used
across consumer restarts.
During graceful shutdown, because of graph shutdown, all the message processing
failed and we did commits even in case of failures. So, after restart, atlas
started at last committed offset and ignored message failures during shutdown
and hence lost messages.
The patch has the following changes:
1. No commit in case of message failure
2. Failed messages are written to different log file
> Data loss is observed when atlas is restarted while hive_table metadata
> ingestion into kafka topic is in-progress
> -----------------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-1111
> URL: https://issues.apache.org/jira/browse/ATLAS-1111
> Project: Atlas
> Issue Type: Bug
> Affects Versions: 0.8-incubating
> Reporter: Sharmadha Sainath
> Assignee: Shwetha G S
> Priority: Critical
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-1111.1.patch, ATLAS-1111.2.patch,
> ATLAS-1111.3.patch, ATLAS-1111.patch
>
>
> During atlas stop, the graph is shutdown first and then the services like
> NotificationHookConsumer is shutdown. After graph is shutdown and before
> NotificationHookConsumer is shutdown, the message handling fails as the graph
> is down, but the NotificationHookConsumer commits the message(updating the
> offset). So, the messages during this time are lost
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
