[jira] [Updated] (NIFI-2853) Improve ListHDFS state tracking
[ https://issues.apache.org/jira/browse/NIFI-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-2853: -- Description: Currently ListHDFS tracks two properties in state management, "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the directory property now supports expression language which means the directory being listed could dynamically change on any execution of the processor. The processor should be changed to store state specific to the directory that was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1". This would also help in a clustered scenario... currently ListHDFS has to be run on primary node only, otherwise each node will be overwriting each others state and producing unexpected results. With the above improvement, if the directory evaluated to a unique path for each node, it would store the state of each of those path separately. was: Currently ListHDFS tracks two properties in state management, "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the directory property now supports expression language which means the directory being listed could dynamically change on any execution of the processor. The processor should store state specific to the directory that was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1". This would also help in a clustered scenario... currently ListHDFS has to be run on primary node only, otherwise each node will be overwriting each others state and producing unexpected results. With the above improvement, if the directory evaluated to a unique path for each node, it would store the state of each of those path separately. > Improve ListHDFS state tracking > --- > > Key: NIFI-2853 > URL: https://issues.apache.org/jira/browse/NIFI-2853 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bryan Bende >Priority: Minor > > Currently ListHDFS tracks two properties in state management, > "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the > directory property now supports expression language which means the directory > being listed could dynamically change on any execution of the processor. > The processor should be changed to store state specific to the directory that > was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1". > This would also help in a clustered scenario... currently ListHDFS has to be > run on primary node only, otherwise each node will be overwriting each others > state and producing unexpected results. With the above improvement, if the > directory evaluated to a unique path for each node, it would store the state > of each of those path separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2853) Improve ListHDFS state tracking
[ https://issues.apache.org/jira/browse/NIFI-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-2853: -- Description: Currently ListHDFS tracks two properties in state management, "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the directory property now supports expression language which means the directory being listed could dynamically change on any execution of the processor. The processor should store state specific to the directory that was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1". This would also help in a clustered scenario... currently ListHDFS has to be run on primary node only, otherwise each node will be overwriting each others state and producing unexpected results. With the above improvement, if the directory evaluated to a unique path for each node, it would store the state of each of those path separately. was: Currently ListHDFS tracks two properties in state management, "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the directory property now supports expression language which means the directory being listed could dynamically change on any execution of the processors. The processor should store state specific to the directory that was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1". This would also help in a clustered scenario... currently ListHDFS has to be run on primary node only, otherwise each node will be overwriting each others state and producing unexpected results. With the above improvement, if the directory evaluated to a unique path for each node, it would store the state of each of those path separately. > Improve ListHDFS state tracking > --- > > Key: NIFI-2853 > URL: https://issues.apache.org/jira/browse/NIFI-2853 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bryan Bende >Priority: Minor > > Currently ListHDFS tracks two properties in state management, > "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the > directory property now supports expression language which means the directory > being listed could dynamically change on any execution of the processor. The > processor should store state specific to the directory that was listed, for > example "listing.timestamp.dir1" and "emitted.timestamp.dir1". > This would also help in a clustered scenario... currently ListHDFS has to be > run on primary node only, otherwise each node will be overwriting each others > state and producing unexpected results. With the above improvement, if the > directory evaluated to a unique path for each node, it would store the state > of each of those path separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
