Re: nifi deploy automation - variables pass through (incl controller services)

2019-08-08 Thread Bryan Bende
Hello, A lot depends on whether you are doing the promotion with the UI or with scripts using something like NiFi CLI or NiPyApi. 1) Currently references to controller services outside the versioned process group need to be set after initial import. In the next release it will auto-select by

Re: Component State

2019-07-31 Thread Bryan Bende
Hello, I'm not sure specifically how to do it with nipyapi, but the REST end-point is /processors//{id}/state where id is the processor uuid. Thanks, Bryan On Wed, Jul 31, 2019 at 7:34 AM ashwinreddyc . wrote: > > Hello, > > I'm looking to get the component state of say QueryDatabaseTables

Re: Nifi registry - extension-repository, extensions, bundles

2019-07-31 Thread Bryan Bende
Hi Mike, The ability to version control extension bundles is a new feature that the community has been working towards. The dream vision is to pull a flow flow registry and also be able to retrieve any extension bundles that go with it. Right now, the 0.4.0 release of registry has the ability to

Re: Required work to close out 1.10

2019-07-29 Thread Bryan Bende
There are two really helpful bug fixes [1][2] related to version controlling flows that I would like to see make it in to 1.10.0, but they are dependent on releasing registry 0.5.0 first, since part of the fixes are in the registry flow-diff code which NiFi depends on. So I was hoping to release

Re: Creating no argument record path functions

2019-07-26 Thread Bryan Bende
I know this doesn’t really answer the question, but in most cases couldn’t you use EL ${now()} in the record path statement and it would get evaluated before the record path? On Fri, Jul 26, 2019 at 6:42 PM Mike Thomsen wrote: > I tried to creating a simple now() timestamp function and got

Re: CM of nar via toolkit cli and regisry

2019-07-22 Thread Bryan Bende
One more thing, make sure the on the dependencies stays the same, meaning nifi-api should still be provided. You don't want to end up bundling the 1.10.0 nifi-api into your NARs. On Mon, Jul 22, 2019 at 2:21 PM Bryan Bende wrote: > > I think you are correct that you would need to also bum

Re: CM of nar via toolkit cli and regisry

2019-07-22 Thread Bryan Bende
ltipart.TestStandardMyService.testService(TestStandardMyService.java:36) > > > > > > Do I need to updated nifi-mock to 1.10 also? > > > > > > Thanks. > > > > > > -Aaron > > > > > > On Sun, Jul 21, 2019 at 9:59 PM Aaron Rich wrote: > >

Re: CM of nar via toolkit cli and regisry

2019-07-21 Thread Bryan Bende
Hello, The issue is that in order to correctly generate the extension manifest with the new NAR plugin, it requires changes from nifi-api that are not released yet. You should be able to build NiFi on the master branch, really just the nifi-api module, doing a mvn clean install. Then in your

Re: Auto cleanup custom Controller Service properties?

2019-07-19 Thread Bryan Bende
I believe you should still be able to delete the property with a delete icon next to it. This should be the same behavior for processors when a property is removed, regardless of dynamic properties. On Fri, Jul 19, 2019 at 11:01 AM Peter Wicks (pwicks) wrote: > > I ran into a fairly rare

Re: NiFi - Cluster crashes while running CaptureChangeMySQL for CDC

2019-07-15 Thread Bryan Bende
I'm not very familiar with this processor, but I think we should probably set a size on the blocking queue so that it can't grow indefinitely, and possibly make the size configurable as a property of the processor. On Sat, Jul 13, 2019 at 2:05 PM Purushotham Pushpavanthar wrote: > > Hi, > > I've

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Bryan Bende
> > I agree that this looks great. I think Mike’s idea is worth > > > > considering > > > > > > as > > > > > > > well. I would hope, that as part of this effort some thought will > > > be > > > > > > given > > > > > >

Re: NiFi Toolkit CLI Token Creation

2019-06-14 Thread Bryan Bende
plementing a Kerberos > solution that either allows you to configure a keytab and principal or pulls > from the current subject if your already logged in. > > > > I created NIFI-6378 and NIFIREG-281. Can one of you assign the registry > one to me as I'm not a contributor ther

Re: NiFi Toolkit CLI Token Creation

2019-06-13 Thread Bryan Bende
t;> request. I'll have to do some testing but that might not require too many > >> changes. In theory it could also support username/password auth as well > >> doing it the same way. > >> > >> https://stackoverflow.com/questions/29056051/adding-authorization

Re: NiFi Toolkit CLI Token Creation

2019-06-13 Thread Bryan Bende
rname/password auth as well doing > it the same way. > > https://stackoverflow.com/questions/29056051/adding-authorization-header-to-jersey-sse-client-request > > Thanks > Shawn > > On 6/13/19, 1:04 PM, "Bryan Bende" wrote: > > I'm not sure if I confused t

Re: NiFi Toolkit CLI Token Creation

2019-06-13 Thread Bryan Bende
icies and we don't have a way > to get SSL Certificates issued to individuals only servers. > > Thanks > Shawn > > On 6/13/19, 12:30 PM, "Bryan Bende" wrote: > > Just to further elaborate, within the CLI there are commands that work > against registry

Re: NiFi Toolkit CLI Token Creation

2019-06-13 Thread Bryan Bende
gt;> Kerberos Token. > >>> > >>> As I looked through the tool kit CLI I was seeing that auth token being > >>> passed into all the rest calls so I was hoping I could hijack wherever > >>> that was being generated via 2way ssl and add an option to

Re: NiFi Toolkit CLI Token Creation

2019-06-12 Thread Bryan Bende
I meant to say that you obviously could generate certs for CLI users, but I was just mentioning an alternative where you can proxy an identity. Right now the CLI never obtains a token because it is all cert based. On Wed, Jun 12, 2019 at 1:03 PM Bryan Bende wrote: > Right now the i

Re: NiFi Toolkit CLI Token Creation

2019-06-12 Thread Bryan Bende
Right now the idea is that whoever is running the CLI would have access to a NiFi server certificate and then you can proxy any user you want. There should be examples of this in the readme or toolkit guide. Supporting Kerberos auth was something I wanted to do, but it’s definitely not a trivial

Re: [EXT] Re: GitHub Stuff

2019-06-12 Thread Bryan Bende
; alopre...@apache.org > > > alopresto.apa...@gmail.com > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > On Jun 11, 2019, at 12:55 PM, Bryan Bende wrote: > > > > > > > > I will admi

Re: [EXT] Re: GitHub Stuff

2019-06-11 Thread Bryan Bende
ggestions. Thanks for starting the > > conversation Bryan. GPG signing is very important for security and for > > encouraging the rest of the community to adopt these practices as well. > > > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...

GitHub Stuff

2019-06-11 Thread Bryan Bende
I had two thoughts related to our GitHub usage that I wanted to throw out there for PMC members and committers... 1) I think it would be helpful if everyone setup the link between their Apache id and github [1]. Setting up this link puts you into the nifi-committers group in Apache (currently 17

Re: First contribution - did I do it right?

2019-06-05 Thread Bryan Bende
-Evan > > > > On Wed, Jun 5, 2019 at 2:08 PM Bryan Bende wrote: > > > Hi Evan, > > > > It looks correct to me. What did you mean by 'the ticket is not visible"? > > I was able to view the JIRA. > > > > Thanks, > > > > Bryan > &g

Re: First contribution - did I do it right?

2019-06-05 Thread Bryan Bende
Hi Evan, It looks correct to me. What did you mean by 'the ticket is not visible"? I was able to view the JIRA. Thanks, Bryan On Wed, Jun 5, 2019 at 4:59 PM Evan Reynolds wrote: > I had a problem with the MergeRecord processor which I tracked down to a > bug, so I fixed it. I thought I'd

Re: NAR extensions warning during build

2019-05-30 Thread Bryan Bende
wrote: > > I'm sorry. Too much copying and pasting. I'm using 1.3.1 of the > nifi-nar-maven-plugin. Would you reconsider the question on that basis? > > On 5/30/19 11:05 AM, Bryan Bende wrote: > > 3.5.1 is not a valid version of the NAR plugin. The current released > >

Re: NAR extensions warning during build

2019-05-30 Thread Bryan Bende
3.5.1 is not a valid version of the NAR plugin. The current released version used by NiFi master is 1.3.1 ... https://github.com/apache/nifi/blob/master/pom.xml#L492 Does the same thing happen using 1.3.1? On Thu, May 30, 2019 at 1:02 PM Russell Bateman wrote: > > In /pom.xml/, I have: > > >

Re: MODERATE for annou...@apache.org

2019-05-23 Thread Bryan Bende
cept-1558618900.41479.goiokbocdnegaihbb...@apache.org > > > > > > To approve: > > announce-accept-1558618900.41479.goiokbocdnegaihbb...@apache.org > > To reject: > > announce-reject-1558618900.41479.goiokbocdnegaihbb...@apache.org > > To give a reason to reject: >

[ANNOUNCE] Apache NiFi Registry 0.4.0 Release

2019-05-22 Thread Bryan Bende
Hello The Apache NiFi team would like to announce the release of Apache NiFi Registry 0.4.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing,

Re: Registry Root Template

2019-05-22 Thread Bryan Bende
Craig, You should be able to just create one additional process group to wrap everything that would have been on your root canvas. For example, if you normally have 10 process groups on your root canvas, then put all of them inside one process group and have just that one process group on your

[RESULT] [VOTE] Release Apache NiFi Registry 0.4.0

2019-05-21 Thread Bryan Bende
://lists.apache.org/thread.html/23cf575e9942d92294fdacb0a2eee05b8e4f7fcf046cf6e855e66b36@%3Cdev.nifi.apache.org%3E On Tue, May 21, 2019 at 9:24 AM Bryan Bende wrote: > > +1 binding > > On Mon, May 20, 2019 at 7:21 AM Pierre Villard > wrote: > > > > +1, binding > > > > Went t

Re: [VOTE] Release Apache NiFi Registry 0.4.0

2019-05-21 Thread Bryan Bende
ase this package as nifi-registry-0.4.0 > > > > Ran through the release helper guide... > > * Built with a clean local maven repository > > * Ran the binary, everything running well. > > * Ran TestRestAPI.java to exercise the API. > > * UI testing, no issues

Apache NiFi Registry 0.4.0 RC1 Release Helper Guide

2019-05-16 Thread Bryan Bende
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven

[VOTE] Release Apache NiFi Registry 0.4.0

2019-05-16 Thread Bryan Bende
Hello, I am pleased to be calling this vote for the source release of Apache NiFi Registry nifi-registry-0.4.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1145 The Git tag is nifi-registry-0.4.0-RC1 The

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

2019-05-16 Thread Bryan Bende
The last few items tagged for 0.4.0 have been merged so we should be good to put out an RC for 0.4.0. I'll start working on getting everything together and try to get something out soon. On Fri, May 10, 2019 at 9:00 AM Bryan Bende wrote: > > I think we should be able to kick 0.4.0 ver

[RESULT] [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.1

2019-05-13 Thread Bryan Bende
://lists.apache.org/thread.html/a222ddab888bc42aab86f090ab4aab0fbfc746366f3b9e2df252e7d9@%3Cdev.nifi.apache.org%3E On Mon, May 13, 2019 at 9:49 AM Bryan Bende wrote: > > +1 binding > > On Fri, May 10, 2019 at 2:11 PM Pierre Villard > wrote: > > > > +1, binding > &

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.1

2019-05-13 Thread Bryan Bende
> > > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.1 > > > > Follow helper guide. Cleared local repository while building the maven > > plugin and nifi. Verified build and resulting artifacts working as > > expected. > > > > Thanks fo

Re: Apache NiFi NAR Maven Plugin

2019-05-09 Thread Bryan Bende
Hello, There is generally a minimum set of NARs you would need/want to leave like framework, provenance, standard api, standard processors, but anything else you don’t use you can remove. Thanks, Bryan On Thu, May 9, 2019 at 7:28 PM Craig Knell wrote: > Hi Folks > > with the NAR's is it

Apache NiFi NAR Maven Plugin 1.3.1 RC1 Release Helper Guide

2019-05-09 Thread Bryan Bende
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven

[VOTE] Release Apache NiFi NAR Maven Plugin 1.3.1

2019-05-09 Thread Bryan Bende
Hello, I am pleased to be calling this vote for the source release of Apache NiFi NAR Maven Plugin 1.3.1. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1144 The Git tag is nifi-nar-maven-plugin-1.3.1-RC1 The

Re: [DISCUSS] Release NAR Maven Plugin

2019-05-07 Thread Bryan Bende
seems to solve the issue, but will continue to test [3]. [1] https://github.com/apache/nifi/pull/3463 [2] https://issues.apache.org/jira/browse/NIFI-6272 [3] https://github.com/apache/nifi-maven/pull/11 On Mon, Apr 29, 2019 at 10:43 AM Bryan Bende wrote: > > I'll start pulling together the r

[ANNOUNCE] Apache NiFi NAR Maven Plugin 1.3.0 Release

2019-05-07 Thread Bryan Bende
Hello The Apache NiFi team would like to announce the release of Apache NiFi nifi-nar-maven-plugin 1.3.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data

Re: [ANNOUNCE] Apache NiFi NAR Maven Plugin 1.3.0 Release

2019-05-06 Thread Bryan Bende
The announce was rejected because apparently you are not supposed to use "repository.apache.org" as a link for downstream consumers, and instead should be linked to Maven central. So I'll be re-sending with the corrected link. On Mon, May 6, 2019 at 10:14 AM Bryan Bende wrote:

[ANNOUNCE] Apache NiFi NAR Maven Plugin 1.3.0 Release

2019-05-06 Thread Bryan Bende
Hello The Apache NiFi team would like to announce the release of Apache NiFi nifi-nar-maven-plugin 1.3.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data

[RESULT] [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.0

2019-05-03 Thread Bryan Bende
Apache NiFi Community, I am pleased to announce that the 1.3.0 release of the Apache NiFi NAR Maven plugin passes with: 6 +1 (binding) votes 0 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to all who helped make this release possible. Here is the PMC vote thread:

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.0

2019-05-03 Thread Bryan Bende
+1 binding On Fri, May 3, 2019 at 9:42 AM Matt Burgess wrote: > > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.0 > > Ran through release helper, verified artifacts. Thanks for RM'ing Bryan! > > On Tue, Apr 30, 2019 at 12:03 PM Bryan Bende wrote: > >

Re: [EXT] Re: Kicking the Kerberos out of org.apache.nifi.hadoop

2019-04-30 Thread Bryan Bende
sHttpClientConfigurer` to nifi-security-utils? > > Thanks, > Peter > > > -Original Message- > From: Bryan Bende > Sent: Tuesday, April 30, 2019 10:33 AM > To: dev@nifi.apache.org > Subject: [EXT] Re: Kicking the Kerberos out of org.apache.nifi.hadoop >

Re: Kicking the Kerberos out of org.apache.nifi.hadoop

2019-04-30 Thread Bryan Bende
I created a mini-kerberos framework here in nifi-security-utils that does not depend on Hadoop: https://github.com/apache/nifi/tree/master/nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/krb The first use-case was for the Solr processors and I believe this code is also

Apache NiFi NAR Maven Plugin 1.3.0 RC1 Release Helper Guide

2019-04-30 Thread Bryan Bende
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven

[VOTE] Release Apache NiFi NAR Maven Plugin 1.3.0

2019-04-30 Thread Bryan Bende
Hello, I am pleased to be calling this vote for the source release of Apache NiFi NAR Maven Plugin 1.3.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1142 The Git tag is nifi-nar-maven-plugin-1.3.0-RC1 The

Re: [DISCUSS] Release NAR Maven Plugin

2019-04-29 Thread Bryan Bende
I'll start pulling together the release soon. On Fri, Apr 26, 2019 at 11:09 AM Joe Witt wrote: > > I am also happy for you to take the RM duties :) > > Thanks Bryan! > > Joe > > On Fri, Apr 26, 2019 at 10:54 AM Bryan Bende wrote: > > > Hello, > >

[DISCUSS] Release NAR Maven Plugin

2019-04-26 Thread Bryan Bende
Hello, I'd like to kick out a new release of the NAR Maven plugin soon. The primary purpose would be for a new feature that generates an extension descriptor inside each NAR. This descriptor will provide information to NiFi Registry about what extensions exist in the NAR when version controlling

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
ow to get the > timestamp on fetch. That will probably require a change or new scan method. > > Thanks > Shawn > > -Original Message- > From: Bryan Bende > Sent: Thursday, April 25, 2019 4:29 PM > To: dev@nifi.apache.org > Subject: Re: Adding HBase Support for At

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
service could throw unsupported for the new method since it would never be used. On Thu, Apr 25, 2019 at 5:25 PM Bryan Bende wrote: > > Thanks, I'm following now... > > I think adding the new method to the interface and throwing > UnsupportedOperationException for 1_1_2, or usi

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
t); > } > } > > If the atomic guarantee for the original checkAndPut is good enough then > there is no reason I can't implement the atomic map cache for both versions > of HBase. > > Thanks > Shawn > > -Original Message- > From: Bryan Bende > Sen

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
; Thanks > Shawn > > -Original Message- > From: Bryan Bende > Sent: Thursday, April 25, 2019 11:56 AM > To: dev@nifi.apache.org > Subject: Re: Adding HBase Support for AtomicDistributedMapCacheClient > > Can it not be done with the existing checkAndPut method? [1] &

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
While Hbase 1.x supports > checkAndMutate it doesn't provide a way to filter on timestamp which is part > of how I was going to implement the revision requirement for AtomicMapCache. > > Thanks > Shawn > > -Original Message- > From: Bryan Bende > Sent:

Re: Adding HBase Support for AtomicDistributedMapCacheClient

2019-04-25 Thread Bryan Bende
I'm not aware of a JIRA, so I'd say go for it. On Wed, Apr 24, 2019 at 9:27 PM Shawn Weeks wrote: > > Seems like this should be fairly easy for HBase 2.x with the checkAndMutate > functionality and I was wondering if there is already a Jira for this. > Otherwise I might make an attempt at it.

Re: HBase Atomic Cache

2019-04-24 Thread Bryan Bende
I'm not sure if there is a real reason it can't be implemented, I think Hbase has the necessary compare-and-set operations to make it work, may just be that someone needs to do the work to implement it. On Wed, Apr 24, 2019 at 3:18 PM Shawn Weeks wrote: > > With Nifi 1.9 and Hbase 2.x what is

Re: NIFI-1693 Maxmind Database Files and File Expansion with Expression Language Enabled

2019-04-22 Thread Bryan Bende
Hello, 1) In that test, there is a class TestableGeoEnrichIP which is a subclass of the processor that overrides where the database reader comes from, and replaces it with a mock reader. So the property for the file location is not actually used for anything but does need to be set for validation

Re: How to pass a flowfile attribute into the query of a getMongo

2019-04-12 Thread Bryan Bende
Hi, You can use expression language in the Query property in GetMongo, such as ${query} to reference an attribute called 'query'. -Bryan On Fri, Apr 12, 2019 at 6:53 AM Puspak wrote: > > hi > > How to pass a flowfile attribute into the query of a getMongo in nifi to > fetch a particular record

Re: Node Group property usage

2019-04-11 Thread Bryan Bende
ly added to relevant policies (not > necessarily requiring membership in a particular group) > - authorizations.xml and users.xml inherited from the cluster, if different > - add initial admin to component access policies even if flow.xml.gz does > not exist at time of startup > >

Re: Node Group property usage

2019-04-11 Thread Bryan Bende
I believe the Node Group will grant that group the same permissions as the Node Identities would get. So if you have a user-group-provider that already has this group and already knows the nodes are in this group, maybe in LDAP or some other external provider, then you can just specify this group

Re: InvokeHttp - Response - Not readable

2019-04-09 Thread Bryan Bende
Have you tried adding a user-defined property to InvokeHttp with name of "Accept" and value of "application/json" ? Any user defined properties should be sent as headers. On Mon, Apr 8, 2019 at 10:11 PM Puspak wrote: > > Thanks Bryan , > > I sent the floe file to a putfile and read it from

Re: InvokeHttp - Response - Not readable

2019-04-08 Thread Bryan Bende
The response shows that the content type being returned is text/html; charset=utf-8 so that is why you can't see it in the content viewer. You could send the flow file to a PutFile processor to write it out to the filesystem and open it in a text editor to see what is being returned. If you are

Re: InvokeHttp - Response - Not readable

2019-04-08 Thread Bryan Bende
Hello, Can you provide more information about the response? what is the http status code? what is the body of the response? Can you make the same request using curl ? -Bryan On Mon, Apr 8, 2019 at 7:16 AM Puspak wrote: > > Hi team , > My invokehttp is making a get call over http , but the

Re: Code style & backwards compatiblity

2019-04-05 Thread Bryan Bende
I think it really comes down to a case by case basis. Generally code that is in a processor, or in a specific NAR, is not usually meant to be shared or extended from, so the best practice would be to move code like that into utility modules in nifi-nar-bundles/nifi-extension-utils so that it can

Re: Stability of RecordReader et. al.

2019-04-05 Thread Bryan Bende
Hi Lars, I think the issue is that the classes you referenced are not formally part of nifi-api, they are technically part of a controller service API. So the reference to 'unstable' doesn't really mean that record processing itself is unstable, it refers more to the API guarantees that are being

Re: initial component access policies

2019-04-03 Thread Bryan Bende
authorizations (i.e. component policies) for an Initial > Admin User. > > Do you have suggestions on how to access the FileAccessPolicyProvider (or > more generally a ConfigurableAccessPolicyProvider)? > > Thanks again, > Mark > > > > On Tue, Apr 2, 2019 at 10:35

Re: initial component access policies

2019-04-02 Thread Bryan Bende
The initial admin policies are created here: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-file-authorizer/src/main/java/org/apache/nifi/authorization/FileAccessPolicyProvider.java#L595

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

2019-03-28 Thread Bryan Bende
s I > > > > suppose) am looking for is the ability to rebuild the metadata DB from > > > the > > > > Git repo. But to be honest, this is not a blocker at all: I can live > > > with a > > > > custom Docker image containing this feat

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

2019-03-27 Thread Bryan Bende
tart path if I wanted to try to kick-the-tires a little on this? > > Thanks, > Ryan > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende wrote: > > > Mike, > > > > All record-oriented components are extensions and thus can make use of > > versioned components

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

2019-03-26 Thread Bryan Bende
f NiFi. > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende wrote: > > > Hi Pierre, > > > > I think we are definitely close to an 0.4.0 release. A major chunk of > > extension registry work has already landed in master, and I still have one > > other jira th

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

2019-03-25 Thread Bryan Bende
Hi Pierre, I think we are definitely close to an 0.4.0 release. A major chunk of extension registry work has already landed in master, and I still have one other jira that I almost have ready and wanted to include, NIFIREG-233 for generating extensions docs. Plus we probably need to make a few

Re: Getting Duplicate Flowfiles from InvokeHttp and QueryElasticsearchHttp

2019-03-19 Thread Bryan Bende
that helps. -Bryan On Tue, Mar 19, 2019 at 10:58 AM martin.cooley wrote: > > Hey Bryan, > > Indeed it is a 2 node cluster. I would like to say I see where this is > going, but I don't. > > Thanks, > > Martin > > > > Bryan Bende wrote > > Hello, >

Re: Configuration Service

2019-03-14 Thread Bryan Bende
This original thread was from quite a long time ago, and there have been many improvements since then, one of which is a variable registry that is exposed directly through the UI [1] and does not require bouncing the NiFi instance. [1]

Re: [VOTE] Release Apache NiFi 1.9.1 (rc1)

2019-03-14 Thread Bryan Bende
+1 (binding) Ran through release helper and everything looked good, tested a few flows and auto-loading of NARs, worked as expected. On Thu, Mar 14, 2019 at 8:55 AM Mike Thomsen wrote: > > Disregard, I misinterpreted something, but I am having weird local issues > with the maven docker plugin

Re: NiFi Registry feature branch workflow (possibly via Git-Backed MetadataService, FlowPersistenceProvider)

2019-03-05 Thread Bryan Bende
with-apache-nifi > [3] https://github.com/Chaffelson/nipyapi/blob/master/nipyapi/demo/fdlc.py > > Thanks, > Bryan > > On Tue, Mar 5, 2019 at 12:08 PM Bryan Bende wrote: > > > Bryan, > > > > The idea of branching to test changes is definitely interesting.

Re: NiFi Registry feature branch workflow (possibly via Git-Backed MetadataService, FlowPersistenceProvider)

2019-03-05 Thread Bryan Bende
Bryan, The idea of branching to test changes is definitely interesting. A couple of thoughts I had while thinking about it... 1) The FlowPersistenceProvider is an extension point, so how would branching work if someone is not using the GitFlowPersistenceProvider? I guess it could be disable if

Re: SplitRecord behaviour

2019-03-01 Thread Bryan Bende
egmenting is to > > - build your segments as a list while processing the incoming stream > - then after send them all to the relationship > > right? > > > On March 1, 2019 at 09:21:46, Bryan Bende (bbe...@gmail.com) wrote: > > Hello, > > Flow files are not transferred until

Re: SplitRecord behaviour

2019-03-01 Thread Bryan Bende
Hello, Flow files are not transferred until the session they came form is committed. So imagine we periodically commit and some of the splits are transferred, then half way through a failure is encountered, the entire original flow file will be reprocessed, producing some of the same splits that

Re: Opinions on Assembly target location

2019-02-27 Thread Bryan Bende
I think this situation is sort of analogous to some other scenarios we have run into, such as the integrations between NiFi and stream processing. For example, the storm and spark integration lives in NiFi's code base, and NiFi is responsible for determining if a new version of those APIs is

Re: NIFI Unit Test Error with @OnScheduled annotation

2019-02-27 Thread Bryan Bende
re) and synchronize the onTrigger > thread(s) so the "first one in" gets to create the connection object > and the rest use it. If that's not accurate, hopefully someone who > knows more about it will correct me and clarify :) > > Regards, > Matt > > On Wed

Re: NIFI Unit Test Error with @OnScheduled annotation

2019-02-27 Thread Bryan Bende
Lets use one list only. I think there should be more to this error, I believe the invocation target exception means another exception was thrown from OnScheduled while the framework tried to execute OnScheduled. Not sure why the real exception isn't visible, but you could always try temporarily

Re: [DISCUSS] Deprecate processors who have Record oriented counterpart?

2019-02-23 Thread Bryan Bende
One thing I would add is that in the 1.9.0 release there is now schema inference built in so that you can just start using the record processors without having a schema. That being said I am neutral about deprecating the non-record processors for source and destination systems. The processors I

Re: [VOTE] Release Apache NiFi 1.9.0 (rc2)

2019-02-19 Thread Bryan Bende
+1 (binding) - Verified everything in the release helper - Ran a cluster and tested some scenarios for auto-loading NARs Minor note - I think it was mentioned already, but the source NOTICE still has the year as 2018. On Tue, Feb 19, 2019 at 12:33 PM Mike Thomsen wrote: > > +1 (binding) > >

Re: Lowering the barrier of entry

2019-01-28 Thread Bryan Bende
on, 28 Jan 2019 at 16:13, Bryan Bende wrote: > > > > What does everyone think about bumping the "Developer" section of the > > docs ahead of "Processors" so that it's easier to find? > > > > Here is what it would look like - > > https://gist.github

Re: Lowering the barrier of entry

2019-01-28 Thread Bryan Bende
7) I couldn't find (m?)any docs about the Groovy scripting API, but > > >> the great blog posts by Matt Burgess and others were invaluable > > >> 8) In case this all sounds too negative, NiFi is fab! > > >> > > >> On Fri, 25 Jan 2019 at 18:47, Andrew

Re: Future-proofing custom bundles that use the Record API

2019-01-28 Thread Bryan Bende
The record API is part of nifi-record-serialization-services which gets bundled with nifi-standard-services-api. I think when we get the extension registry working we should be able to disconnect the releases of nifi framework from all of the other bundles, instead of the current situation where

Re: Lowering the barrier of entry

2019-01-25 Thread Bryan Bende
very time I need to come back > to this and refresh my brain synapses. If we could make these not require > any of that and make simple thinga dead simple > > Andrew > > On Fri, Jan 25, 2019, 9:47 AM Bryan Bende wrote: > > > Andrew, > > > > I'm not disagree

Re: Lowering the barrier of entry

2019-01-25 Thread Bryan Bende
and straight forward document > > showing them what should be there, best practices for maintainability and > > where to announce it. > > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende wrote: > > > > > I think we have a lot more documentation than most proj

Re: Lowering the barrier of entry

2019-01-25 Thread Bryan Bende
I think we have a lot more documentation than most projects, but I think an issue is that content is scattered in many different locations, and some of the docs are huge reference guides where it can be hard to find all the pieces of what you are trying to do. The first thing a new contributor

Re: [DISCUSS] Nifi Archetypes “refresh”

2019-01-11 Thread Bryan Bende
I think a lot of cases are a specialization of the base cases (processor archetype and service archetype), so when possible I would lean towards some documentation on the Maven Projects wiki page [1] about how to take one of those and slightly modify it to get started with the given scenario. For

Re: Checking for existence of mandatory attributes

2018-12-20 Thread Bryan Bende
RouteOnAttribute? On Thu, Dec 20, 2018 at 11:45 AM Mike Thomsen wrote: > > Do we have any processors that would be particularly good at checking a > flowfile for the existence of certain mandatory attributes and then routing > flowfiles missing them to a failure relationship? > > I have a

Re: ISPEnrichIP test data?

2018-12-19 Thread Bryan Bende
Just wanted to mention that there is also IPLookupService which can be used with LookupRecord. Not sure if that saves you from needing to implement a record-aware version of ISPEnrichIP. On Wed, Dec 19, 2018 at 10:57 AM Mark Payne wrote: > > Mike, > > MaxMind provides 2 datasets. The GeoIP2

Re: Set root level access policies using NiFi REST API

2018-12-18 Thread Bryan Bende
Anything you can do from the UI you can do from the REST API. You can open something like Chrome Dev tools and watch the network tab which performing the desired action in the UI. Then you can see what API calls the UI makes. On Tue, Dec 18, 2018 at 1:53 PM Milan Das wrote: > > I am wondering if

Re: Processor convert between data formats / Community repository

2018-12-14 Thread Bryan Bende
Hi Raymond, Thanks for sharing. Currently we don't really have a great way to keep track of custom NARs that live outside of the main distribution. However, the community is working on adding extension bundles to NiFi Registry, and eventually there would likely be a publicly available NiFi

Re: Include Section for Processor Breaking Changes in Changelog

2018-12-12 Thread Bryan Bende
I added it to the section for 1.4.0 to 1.5.0. On Wed, Dec 12, 2018 at 11:50 AM Bryan Bende wrote: > > Ryan, > > Thanks for mentioning this. Changes like this should really be > captured in the migration guidance for the given release, but it looks > like this was not in

Re: Include Section for Processor Breaking Changes in Changelog

2018-12-12 Thread Bryan Bende
Ryan, Thanks for mentioning this. Changes like this should really be captured in the migration guidance for the given release, but it looks like this was not in the 1.5.0 guidance. https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance -Bryan On Wed, Dec 12, 2018 at 10:53 AM Ryan

Re: Reading avro encoded message key from kafka

2018-11-27 Thread Bryan Bende
Unfortunately I don't think there is a good way to interpret the value of the key when it is Avro because we don't have any expression language functions that understand Avro or record-oriented values. The main option would be to change how the data is being produced in some way... - Put the

Re: Question about performance tuning for Nifi

2018-11-22 Thread Bryan Bende
Have you increased the concurrent tasks on ListenSyslog? If not then going from 1 to 2 would be the first thing to try. Since UDP is connection-less, changing the offer to put or changing the timeout would just move the problem to a different part of the stack. NiFis code would be blocking

Re: Provenance Request Issues

2018-11-20 Thread Bryan Bende
Hello, If you are getting a 500 error there should be an error in the nifi-app.log or nifi-user.log that corresponds to this. Can you provide the error message and stacktrace from the logs? Thanks, Bryan On Tue, Nov 20, 2018 at 8:43 AM Donaldson, Michael < michael.donald...@mclaren.com>

<    1   2   3   4   5   6   7   8   >