Adding reference link <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java>(to the code).
On Mon, Jul 29, 2019 at 10:21 AM Ameer Mawia <[email protected]> wrote: > Thanks for reply. > > Hmm, that should explain the behavior we noted. > > But I see(here > <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceTextWithMapping.java>) > an instance level lock which is protecting the update Mapping method. > *Shouldn't > that eventually block other threads from accessing the old mapping?* > > Or may that this locking was added later - version 1.9 or something? We > are using 1.8. > > Thanks, > Ameer Mawia > > On Thu, Jul 25, 2019 at 3:51 AM Koji Kawamura <[email protected]> > wrote: > >> Hi Ameer, >> >> Is the ReplaceTextWithMapping's 'Concurrent Tasks' set to grater than 1? >> Since ReplaceTextWithMapping only reload at a single thread, other >> threads may use old mapping until the loading thread complete >> refreshing mapping definition. >> >> Thanks, >> Koji >> >> On Wed, Jul 24, 2019 at 4:28 AM Ameer Mawia <[email protected]> >> wrote: >> > >> > Inline. >> > >> > On Mon, Jul 22, 2019 at 2:17 AM Koji Kawamura <[email protected]> >> wrote: >> >> >> >> Hi Ameer, >> >> >> >> How is ReplaceTextWithMapping 'Mapping File Refresh Interval' >> configured? >> > >> > [Ameer] It is configured to 1sec - the lowest value allowed. >> >> >> >> By default, it's set to '60s'. So, >> >> 1. If ReplaceTextWithMapping ran with the old mapping file >> > >> > [Ameer] First Processing took place on Day-1. A new Mapping was dropped >> on Day-1, after Day-1 Processing was over. >> >> >> >> 2. and the mapping file was updated for the next processing >> > >> > [Ameer] Second Processing took place on Day-2. >> > [Ameer] Here assumption was CACHE will be refreshed from the new >> mapping file dropped a day earlier. But ti diddnt happend. Cache got >> refreshed in the middle of the flow - not at the very beginnning. Thus few >> flowfile got old value and later flowfile got new value. >> >> >> >> 3. then the flow started processing another CSV file right away line >> by line >> >> >> >> In above scenario, some lines in the CSV might get processed with the >> >> old mapping file. After 60s passed from 1, some other lines may get >> >> processed with the new mappings. Is that what you're seeing? >> >> >> > [Ameer] This is what is happening. But it shouldn't have - becuase new >> mapping file was already existing before the next processing begin. It >> should have refresh right at the start - as also suggested by the code of >> the ReplaceTextWithMapping processor. >> >> >> >> BTW, please avoid posting the same question to users and dev at the >> >> same time. I've removed dev address. >> >> [Ameer] Got it. >> >> Thanks, >> >> Koji >> >> >> >> On Sat, Jul 20, 2019 at 3:08 AM Ameer Mawia <[email protected]> >> wrote: >> >> > >> >> > Correcting Typo. >> >> > >> >> > On Fri, Jul 19, 2019 at 2:03 PM Ameer Mawia <[email protected]> >> wrote: >> >> >> >> >> >> Guys, >> >> >> >> >> >> It seems that NIFI ReplaceTextWithMapping Processors has a BUG >> with Refreshing its Mapped file. We are using its functionality in PROD and >> getting odd behaviour. >> >> >> >> >> >> Our USAGE Scenario: >> >> >> >> >> >> We use NIFI primarily as a TRANSFORMATION Tool. >> >> >> Our flow involves: >> >> >> >> >> >> Getting a raw csv file. >> >> >> Split the file on per line basis: >> >> >> >> >> >> So from one source flowfile - we may have 10000 flowfile >> generated/splitted out. >> >> >> >> >> >> For each of the splitted flow file(flowfiles for individual lines) >> we perform transformation on the attributes. >> >> >> We merge these flowfiles back and write the Output file. >> >> >> >> >> >> >> >> >> As part of the transformation in Step#3, we do some mapping for one >> of the field in the csv. For this we use ReplaceTextWithMapping Processor. >> Also to note we update our mapping file just before starting our flow(ie. >> Step #1) >> >> >> >> >> >> Our Issue: >> >> >> >> >> >> We have noted for SAME key we get two DIFFERENT values in two >> different flowfiles. >> >> >> We noted that one of the value mapped, existed in an older Mapping >> file. >> >> >> So in essence: ReplaceTextWithMapping Processor didn't refresh its >> cash uptill certain time. And thus return the old value for few mapping >> file and then - once in the meanwhile it has refreshed it cache - returned >> new updated value. >> >> >> And this cause the issue? >> >> >> >> >> >> Question: >> >> >> >> >> >> Is this a known issue with ReplaceTextWithMapping Processor? >> >> >> If not how can I create an issue for this? >> >> >> How can I confirm this behaviour? >> >> >> >> >> >> Thanks, >> >> >> Ameer Mawia >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> http://ca.linkedin.com/in/ameermawia >> >> >> Toronto, ON >> >> >> >> >> > >> >> > >> >> > -- >> >> > http://ca.linkedin.com/in/ameermawia >> >> > Toronto, ON >> >> > >> > >> > >> > >> > -- >> > http://ca.linkedin.com/in/ameermawia >> > Toronto, ON >> > >> > > > -- > http://ca.linkedin.com/in/ameermawia > Toronto, ON > > -- http://ca.linkedin.com/in/ameermawia Toronto, ON
