[
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439680#comment-16439680
]
Sankar Hariappan edited comment on HIVE-19219 at 4/16/18 4:27 PM:
------------------------------------------------------------------
Added 01.patch
* REPL DUMP throw IllegalStateException if events found missing in source.
* Modified testIncrementalLoadWithVariableLengthEventId to rename the dumped
event directory names instead of changing event id while dumping.
Stack trace:
{code:java}
2018-04-16T09:21:22,417 WARN [main] metastore.HiveMetaStoreClient: Requested
events are found missing in NOTIFICATION_LOG table. Probably, cleaner would've
cleaned it up. Try setting higher value for
hive.metastore.event.db.listener.timetolive. Also, bootstrap the system again
to get back the consistent replicated state.
2018-04-16T09:21:22,418 ERROR [main] repl.ReplDumpTask: failed
java.lang.IllegalStateException: Notification events are missing.
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:2587)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_112]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy40.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:94)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:146)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:176)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:160)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:111)
[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]{code}
[~thejas], [~maheshk114], [~anishek],
Can you please take a look?
was (Author: sankarh):
Added 01.patch
* REPL DUMP throw IllegalStateException if events found missing in source.
* Modified testIncrementalLoadWithVariableLengthEventId to rename the dumped
event directory names instead of changing event id while dumping.
Stack trace:
{quote}
2018-04-16T09:21:22,417 WARN [main] metastore.HiveMetaStoreClient: Requested
events are found missing in NOTIFICATION_LOG table. Probably, cleaner would've
cleaned it up. Try setting higher value for
hive.metastore.event.db.listener.timetolive. Also, bootstrap the system again
to get back the consistent replicated state.
2018-04-16T09:21:22,418 ERROR [main] repl.ReplDumpTask: failed
java.lang.IllegalStateException: Notification events are missing.
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:2587)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_112]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy40.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:94)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:146)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:176)
~[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:160)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:111)
[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
{quote}
[~thejas], [~maheshk114], [~anishek],
Can you please take a look?
> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> ----------------------------------------------------------------------------
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Sankar Hariappan
> Assignee: Sankar Hariappan
> Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch
>
>
> This is the case where the events were deleted on source because of old event
> purging and hence min(source event id) > target event id (last replicated
> event id).
> Repl dump should fail in this case so that user can drop the database and
> bootstrap again.
> The next incremental repl dump could check if the events fetched from source
> notification_log table is retrieved in continuous sequence with no events
> missing. If any event is missing, it should throw error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
