Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Peter Moberg
Think I might have found the issue. Will report tonight. Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far. Thanks, Peter On Oct 18, 2019, 2:21 PM -0500, Peter Moberg , wrote: > Here it is: > > > 2019-10-18

Re: Curious about Best Practices for Deployment of Workflows

2019-10-18 Thread Andy LoPresto
Ken, There are a lot of disjoint resources that discuss some of these concepts, but they are not well-organized at the moment. I would recommend Kevin Doran’s presentation at Dataworks Summit 2018 [1] as a good starting point, specifically including some activity hooks which can automate much

Curious about Best Practices for Deployment of Workflows

2019-10-18 Thread Ken Swanson
Hi all, My group is getting started with NiFi and I'm hoping to draw on people's experience. I was curious if there were any best practices for NiFi when deploying workflows. When I deploy new code to production, there's often pull requests, code review, and then CI/CD to actually put it in

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Peter Moberg
Here it is: 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000--65536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Mike Thomsen
Can you share the stacktrace from the logs? On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg wrote: > Mike, > > The SSLContextService only had the Trust store configured. I think I seen > that ticket before but didn’t pay attention to the fact it wasn’t merged in > to the code I am running. > >

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Matt Burgess
Tim, The "secondary flow" issue is something I wanted to address as well, so I decoupled the formatting of data from the transmission of that data into a new paradigm/service I'm calling a RecordServiceSink. A ReportingTask can use RecordServiceSink to allow the user to choose where the reporting

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Peter Moberg
Mike, The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running. However, I configured the service to have a KeyStore now but I am getting the same errors… Thanks, Peter On

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Tim Dean
Thanks Mike, I understand what you are saying but I am really trying to avoid having a secondary flow in NiFi if I can avoid it. It seems like NiFi was designed to allow this kind of custom reporting task and that I should ideally be able to do this without using the NiFi-provided S2S

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Mike Thomsen
I would recommend using the site to site task instead of your own because it will give you a very scalable way to asynchronously handle your provenance tracking. There's nothing preventing you from having the tracking instance of NiFi be responsible for publishing the events to your tracking

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Juan Pablo Gardella
I have an issue to validate, reported at before: http://apache-nifi.1125220.n5.nabble.com/Error-instantiating-template-on-cluster-The-specified-observer-identifier-already-exists-td12973.html I reproduced it at apache nifi 1.5.0. I will try to check at nifi 1.9.2. The template to load is near to

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Joe Witt
is a daily effort at this point. i am close to pushing first rc. have been watching for stability on bug fixes. On Fri, Oct 18, 2019 at 1:10 PM Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > Any ETA for Nifi 1.10 release? > > On Fri, 18 Oct 2019 at 13:39, Mike Thomsen wrote: > >>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Juan Pablo Gardella
Any ETA for Nifi 1.10 release? On Fri, 18 Oct 2019 at 13:39, Mike Thomsen wrote: > Peter, > > Are you configuring the service as a trust-only configuration? If so, > that's been addressed in the 1.10 which is due for release in the near(ish) > future. > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Mike Thomsen
Peter, Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future. https://issues.apache.org/jira/browse/NIFI-6228 Thanks, Mike On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg wrote: > As a

Implementing a custom provenance reporting task

2019-10-18 Thread Tim Dean
I would like to implement some custom monitoring logic that captures certain information from the provenance repository. It would be similar in some ways to the existing SiteToSiteProvenanceReporting task, but it will not be sending information to another NiFi node but instead sending things to

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Peter Moberg
As a follow-up. On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate. Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the

Re: Re: MergeRecord can not guarantee the ordering of the input sequence?

2019-10-18 Thread wangl...@geekplus.com.cn
Seems it is because of the balance strategy that is used. The balance will not guarantee the the order. Thanks, Lei wangl...@geekplus.com.cn From: wangl...@geekplus.com.cn Date: 2019-10-16 10:21 To: dev; users CC: dev Subject: Re: Re: MergeRecord can not guarantee the ordering of the input