----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62221/#review185134 -----------------------------------------------------------
sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/DBUpdateForwarder.java Line 117 (original), 138 (patched) <https://reviews.apache.org/r/62221/#comment261378> I noticed this problem in my recent testing. With Brian's fix (Hold HDFS full snapshot while HMS snapshot is happening), the log is being cluttered (every 0.5sec) with these messages while the HMS snapshot is happening (which usually is long if we have lot of data e.g citi) ```bash 2017-09-11 18:50:12,585 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:12,620 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:12,621 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:13,091 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:13,125 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:13,126 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:13,593 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:13,628 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:13,629 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:14,095 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:14,132 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:14,132 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:14,598 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:14,635 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:14,636 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:15,101 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:15,139 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:15,144 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:15,603 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:15,647 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:15,647 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:16,105 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:16,150 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:16,151 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:16,606 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:16,654 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:16,654 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:17,108 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:17,157 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:17,158 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:17,611 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:17,661 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:17,662 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:18,113 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:18,165 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:18,166 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 9 2017-09-11 18:50:18,615 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:18,670 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:18,671 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 10 2017-09-11 18:50:19,118 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:19,187 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:19,188 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 10 2017-09-11 18:50:19,620 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:19,691 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:19,692 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A newer full update is found with image number: 10 2017-09-11 18:50:20,459 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 2017-09-11 18:50:20,459 INFO org.apache.sentry.hdfs.DBUpdateForwarder: A full update is returned due to an unavailable sequence number: 0 ``` We should ideally print this log message just before we get the snapshot but not before we do the SentryStateStore check. It is giving wrong notion that we are getting too many full snapshots which is not the case really. - Vamsee Yarlagadda On Sept. 11, 2017, 8:54 p.m., Arjun Mishra wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62221/ > ----------------------------------------------------------- > > (Updated Sept. 11, 2017, 8:54 p.m.) > > > Review request for sentry, Alexander Kolbasov, Brian Towles, Na Li, Sergio > Pena, and Vamsee Yarlagadda. > > > Repository: sentry > > > Description > ------- > > Sentry Supportability improvement. > > 1. HMSFollower > * Print confirmation message (at INFO level) once full snapshot is persisted > in the DB. > * Print the message that HMSFollower is completely ready (after the initial > pass of HMSFollower is done) > > 2. DBUpdateForwarder > * Every log message should explicitly mention which type of events is it > referring to (PERM or PATH) otherwise there is no way for us to differentiate > between calls. > > > Diffs > ----- > > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/DBUpdateForwarder.java > 8a34d5623 > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryPlugin.java > 1318082d3 > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java > b600487dc > > > Diff: https://reviews.apache.org/r/62221/diff/3/ > > > Testing > ------- > > > Thanks, > > Arjun Mishra > >
