Re: proper way in nifi to sync status between custom processors

Hi Ben, The one thing that looks strange in the screenshot is the ExecuteSqlCommand having FlowFiles queued in its incoming connection. Those should be transferred to 'failure' relationship. Following executeSql() method, shouldn't it re-throw the caught exception? try (Connection

Re: proper way in nifi to sync status between custom processors

Hi Koji, no problem. You could check the code of processor WaitBatch at the link: https://drive.google.com/open?id=1DMpW5GMiXpyZQdui989Rr3D9rlchQfWQ I also uploaded a snapshot of part of NiFi flow which includes the ExecuteSqlCommand and WaitBatch, you could check the picture at the link:

Re: proper way in nifi to sync status between custom processors

Hi Ben, I was referring these two log messages in your previous email. These two messages are both written by ExecuteSqlCommand, it does not mean 'it was executed again'. ``` 2017-12-26 07:00:01,312 INFO [Timer-Driven Process Thread-1] c.z.nifi.processors.ExecuteSqlCommand

Re: proper way in nifi to sync status between custom processors

Hi Koji, I will print the sql before actually executing it, but I checked the error log line you mentioned in your reply, this error was thrown by NiFi from within another processor called WaitBatch. I didn't find similar errors as the one from the ExecuteSqlCommand processor, I think it's because

Re: proper way in nifi to sync status between custom processors

Hi Ben, The following two log messages are very close in terms of written timestamp, but have different log level. 2017-12-26 07:00:01,312 INFO 2017-12-26 07:00:01,315 ERROR I guess those are logged within a single onTrigger of your ExecuteSqlCommand custom processor, one is before executing,

Re: proper way in nifi to sync status between custom processors

Thanks Koji, I will look into this article about the record model. By the way, that error I previously mentioned to you occurred again, I could see the sql query was executed twice in the log, this time I had turned on the verbose NiFi logging, the sql query is as below: 2017-12-26 07:00:01,312

Re: proper way in nifi to sync status between custom processors

Hi Ben, This blog post written by Mark, would be a good starting point to get familiar with NiFi Record model. https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi HA for DistributedMapCacheClientService and DistributedMapCacheServer pair is not supported at the moment. If you need

Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Steven Understood. That it now shows you this is a good thing in 1.4.0 because what was happening in the older versions is one of the sets of processors was being loaded. You are probably seeing a warning in the app log at startup. So depending on which nar is loaded first (standard or your

RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Yes, I agree. Unfortunately, our schedule may not allow for that at this time so we will probably have to put off migrating for now. Thanks to you and Bryan for your replies. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional

Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Steven To Bryan's point if it will be best if you simply copy the code/base classes over to your Nar rather than extending from existing concrete implemented processors. Thanks On Tue, Dec 26, 2017 at 1:50 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US)

RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Bryan, We have extended a few processors from standard processors. For example, one of the processors we are extending is: org.apache.nifi.processors.standard.JmsConsumer In another processor we are importing org.apache.nifi.processors.standard.util.JdbcCommon. I tried making the

Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Can you give some more details about what the dependency is for? If it is some utility code that exists in standard processors then we should be looking to move that to other reusable modules so that you can depend on it without depending on the processors. If it is because you extended a

RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

Thanks for the reply, Bryan, Yes, two of our custom processors have a dependency on the standard processors. The dependency cannot be removed or those processors will not compile. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional

Re: Moving from version 1.1.2 to 1.4.0

Hello, This means your custom NAR is bundling the standard processors jar and as a result they are getting discovered twice, once from your NAR and once from the standard NAR. You’ll have to look at your maven dependencies for your custom NARs and figure out why the dependency on standard

Moving from version 1.1.2 to 1.4.0

Hi devs, I'm looking into moving from NiFi 1.1.2 to 1.4.0. We have several custom processors and services. Everything is compiling without any problems but when I put the services into the 1.4.0 instance, NiFi shows in the list of processors a 1.1.2 and 1.4.0 version of all processors including

Re: proper way in nifi to sync status between custom processors

Thanks for your quick response, Koji, I haven't heard and seen anything about the NiFi record data model when I was reading the NiFi documentations,could you tell me where this model is documented? Thanks. By the way, to my knowledge, when you need to use the DistributedMapCacheServer from

Re: proper way in nifi to sync status between custom processors

Hi Ben, As you found from existing code, DistributedMapCache is used to share state among different processors, and it can be used by your custom processors, too. However, I'd recommend to avoid such tight dependencies between FlowFiles if possible, or minimize the part in flow that requires that

Re: NIFI-4715 : ListS3 list duplicate files when incoming file throughput to S3 is high

Hi Milan, Thanks for your contribution! I reviewed the PR and posted a comment there. Would you check that? Koji On Sat, Dec 23, 2017 at 7:15 AM, Milan Das wrote: > I have logged a defect in NIFI. ListS3 is generation duplicate flows when > S3 throughput is high. > > > >

proper way in nifi to sync status between custom processors

Hi guys, I'm currently trying to find a proper way in nifi which could sync status between my custom processors. our requirement is like this, we're doing some ETL work using nifi and I'm extracting the data from DB into batches of FlowFiles(each batch of FlowFile has a flag FlowFile indicating