[
https://issues.apache.org/jira/browse/NIFI-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklas updated NIFI-5861:
-------------------------
Description:
Since the Notify processor is writing to the cache it would make sense to have
a "age of duration" property like the DetectDuplicate processor.
For example: I've 2 seperate flows which put Data from one DB to another DB.
Flow 2 depends on Flow 1 due to Foreign Key. Both flows have GenerateTableFetch
which listen to Updates in source DB.
At the end of Flow 1 it'll put all INSERTED IDs to a Notify Processor which put
everything in the Cache.
Flow 2 runs simultaneously and it'll check if the Foreign Key Value exist in
the DB that it'll stream the data. If the key doesn't exist it'll be routed to
a DetectDuplicate and if it's a Duplicate (this ID already run in Flow 1 but
isn't inserted yet) it'll be routed to a Wait Processor. The wait processor now
waits for the Notify node on Flow 1.
This whole construct works fine. The wait processor removes the key from the
cache if it's match but I'm writing all IDs to the cache, even IDs that doesn't
hang in a wait processor. I've tried to do a workaround with DetectDuplicate
Processor that it'll only use Notify for things that hang in a wait processor
but it didn't work that great due to bad timings.
It would be good to have a TTL setting similar to the DetectDuplicate processor
for the Notify processor. Additionally to this issue is that the Nifi Cache
Server doesn't have a default TTL option to clear the cache automatically.
was:
Since the Notify processor is writing to the cache it would make sense to have
a "age of duration" property like the DetectDuplicate processor.
For example: I've 2 seperate flows which put Data from one DB to another DB.
Flow 2 depends on Flow 1 due to Foreign Key. Both flows have GenerateTableFetch
which listen to Updates in source DB.
At the end of Flow 1 it'll put all INSERTED IDs to a Notify Processor which put
everything in the Cache.
Flow 2 runs simultaneously and it'll check if the Foreign Key Value exist in
the DB that it'll stream the data. If the key doesn't exist it'll be routed to
a DetectDuplicate and if it's a Duplicate (this ID already run in Flow 1 but
isn't inserted yet) it'll be routed to a Wait Processor. The wait processor now
waits for the Notify node on Flow 1.
This whole construct works fine. The wait processor removes the key from the
cache if it's match but I'm writing all IDs to the cache, even IDs that doesn't
hang in a wait processor.
I've tried to do a workaround with DetectDuplicate Processor that it'll only
use Notify for things that hang in a wait processor but it didn't work that
great due to bad timings.
It would be good to have a TTL setting similar to the DetectDuplicate processor
for the Notify processor.
Additionally to this issue is that the Nifi Cache Server doesn't have a default
TTL option to clear the cache automatically.
> Notify: Add support for age of duration
> ---------------------------------------
>
> Key: NIFI-5861
> URL: https://issues.apache.org/jira/browse/NIFI-5861
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Niklas
> Priority: Major
> Original Estimate: 4h
> Remaining Estimate: 4h
>
>
> Since the Notify processor is writing to the cache it would make sense to
> have a "age of duration" property like the DetectDuplicate processor.
>
> For example: I've 2 seperate flows which put Data from one DB to another DB.
> Flow 2 depends on Flow 1 due to Foreign Key. Both flows have
> GenerateTableFetch which listen to Updates in source DB.
> At the end of Flow 1 it'll put all INSERTED IDs to a Notify Processor which
> put everything in the Cache.
> Flow 2 runs simultaneously and it'll check if the Foreign Key Value exist in
> the DB that it'll stream the data. If the key doesn't exist it'll be routed
> to a DetectDuplicate and if it's a Duplicate (this ID already run in Flow 1
> but isn't inserted yet) it'll be routed to a Wait Processor. The wait
> processor now waits for the Notify node on Flow 1.
>
> This whole construct works fine. The wait processor removes the key from the
> cache if it's match but I'm writing all IDs to the cache, even IDs that
> doesn't hang in a wait processor. I've tried to do a workaround with
> DetectDuplicate Processor that it'll only use Notify for things that hang in
> a wait processor but it didn't work that great due to bad timings.
>
> It would be good to have a TTL setting similar to the DetectDuplicate
> processor for the Notify processor. Additionally to this issue is that the
> Nifi Cache Server doesn't have a default TTL option to clear the cache
> automatically.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
