> On March 10, 2023, 2:24 a.m., Madhan Neethiraj wrote: > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java > > Lines 353 (patched) > > <https://reviews.apache.org/r/74344/diff/1/?file=2274915#file2274915line357> > > > > Is it necessary to commit each unhandled message? Wouldn't committing > > the last message be enough? > > > > If that works, it would be necessary to only hold the last message in > > memory i.e. eliminate unhandledMessages. Also, commitToKafka() should clear > > lastUnhandledMessage - as it shouldn't be call commit on earlier message > > offset.
tagsync supports a configuration parameter for specifying the size of the batch of messages read from the Kafka queue (which may be greater than 1) and then processed. In that case, the batch of messages being may contain a mix of handled and unhandled messages and each message has to be committed individually. - Abhay ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74344/#review225258 ----------------------------------------------------------- On March 10, 2023, 12:17 a.m., Abhay Kulkarni wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74344/ > ----------------------------------------------------------- > > (Updated March 10, 2023, 12:17 a.m.) > > > Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, > Ramesh Mani, and Velmurugan Periasamy. > > > Bugs: RANGER-4129 > https://issues.apache.org/jira/browse/RANGER-4129 > > > Repository: ranger > > > Description > ------- > > If AtlasTagSource.buildAndUploadServiceTags() is called with empty > AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds > exception is thrown when AtlasTagSource.messages list is read. This may > happen when the first notification processed by tagsync process is of type > ENTITY_DELETE. > > This Jira also fixes a potential issue if the Kafka queue contains only a > large number of notifications not expected to be handled by tagsync. In such > cases, tagsync will not acknowledge receipt and it may cause back-pressure on > Kafka queue. > > > Diffs > ----- > > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java > a7c456b3d > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java > 1a3ddecb5 > > > Diff: https://reviews.apache.org/r/74344/diff/1/ > > > Testing > ------- > > Compiled clean and passed all unit tests > > > Thanks, > > Abhay Kulkarni > >
