> On March 10, 2023, 2:24 a.m., Madhan Neethiraj wrote: > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java > > Lines 353 (patched) > > <https://reviews.apache.org/r/74344/diff/1/?file=2274915#file2274915line357> > > > > Is it necessary to commit each unhandled message? Wouldn't committing > > the last message be enough? > > > > If that works, it would be necessary to only hold the last message in > > memory i.e. eliminate unhandledMessages. Also, commitToKafka() should clear > > lastUnhandledMessage - as it shouldn't be call commit on earlier message > > offset. > > Abhay Kulkarni wrote: > tagsync supports a configuration parameter for specifying the size of the > batch of messages read from the Kafka queue (which may be greater than 1) and > then processed. In that case, the batch of messages being may contain a mix > of handled and unhandled messages and each message has to be committed > individually.
This fix now only addresses the Array out-of-bounds issue. For optimizing the commits to Kafka, another JIRA (RANGER-4130) is created. - Abhay ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74344/#review225258 ----------------------------------------------------------- On March 10, 2023, 12:17 a.m., Abhay Kulkarni wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74344/ > ----------------------------------------------------------- > > (Updated March 10, 2023, 12:17 a.m.) > > > Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, > Ramesh Mani, and Velmurugan Periasamy. > > > Bugs: RANGER-4129 > https://issues.apache.org/jira/browse/RANGER-4129 > > > Repository: ranger > > > Description > ------- > > If AtlasTagSource.buildAndUploadServiceTags() is called with empty > AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds > exception is thrown when AtlasTagSource.messages list is read. This may > happen when the first notification processed by tagsync process is of type > ENTITY_DELETE. > > This Jira also fixes a potential issue if the Kafka queue contains only a > large number of notifications not expected to be handled by tagsync. In such > cases, tagsync will not acknowledge receipt and it may cause back-pressure on > Kafka queue. > > > Diffs > ----- > > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java > a7c456b3d > > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java > 1a3ddecb5 > > > Diff: https://reviews.apache.org/r/74344/diff/1/ > > > Testing > ------- > > Compiled clean and passed all unit tests > > > Thanks, > > Abhay Kulkarni > >
