New Standard Pattern - Put Exception that caused failure in an attribute
When a FlowFile is routed to failure, frequently there is no clear reason without looking into the actual error message. Some processors work around this by creating many different relationships, but even then frequently the generic Failure relationship also provides little guidance. I've seen a few cases recently where processors are including the exception message as an attribute on the FlowFile when routing to failure (ExecuteStreamCommand, new PR for ExecuteSQL). Should this be a standard pattern so that it's easier for users to route failures? --Peter
Re: [VOTE] Release Apache NiFi 1.8.0 (RC3)
+1 ( binding ) Ran through the helper guide, testing normal flows ( w and w/o secure comms ) Thanks, Marc On Wed, Oct 24, 2018 at 2:05 AM Koji Kawamura wrote: > +1 (binding) > > Ran through the release helper. > No issue was found. > Thanks for RM duties, Jeff! > > On Wed, Oct 24, 2018 at 1:42 PM James Wing wrote: > > > > +1 (binding). Ran though the release helper, tested the resulting > binary. > > Thank you for your persistence, Jeff. > > > > > > On Mon, Oct 22, 2018 at 10:56 PM Jeff wrote: > > > > > Hello, > > > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > > nifi-1.8.0. > > > > > > The source zip, including signatures, digests, etc. can be found at: > > > https://repository.apache.org/content/repositories/orgapachenifi-1135 > > > > > > The Git tag is nifi-1.8.0-RC3 > > > The Git commit ID is 98aabf2c50f857efc72fd6f2bfdd9965b97fa195 > > > > > > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=98aabf2c50f857efc72fd6f2bfdd9965b97fa195 > > > > > > Checksums of nifi-1.8.0-source-release.zip: > > > SHA256: > 6ec21c36ebb232f344493a4aeb5086eed0c462c576e11a79abed8149bc8b65c3 > > > SHA512: > > > > > > > 846aecd4eb497a3b7dee7d1911b02453b8162b6c87e39f3df837744a478212e2e3e3615921079d29c2804671f26ecd05b04ce46a4bb69e8911fc185e27be9c24 > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/jstorck.asc > > > > > > KEYS file available here: > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > 209 issues were closed/resolved for this release: > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343482 > > > > > > Release note highlights can be found here: > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0 > > > > > > The vote will be open for 72 hours. > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build > > > from source, and test. Then please vote: > > > > > > [ ] +1 Release this package as nifi-1.8.0 > > > [ ] +0 no opinion > > > [ ] -1 Do not release this package because... > > > >
Re: [External] Re: Expression language and NiFi registry
Hi there, There is a JIRA [1] about this idea and, as Joe said, we need to carefully think about the implications. Unless we can do something nice and backward compatible, that could be something for NiFi 2.0. [1] https://issues.apache.org/jira/browse/NIFI-5367 Pierre Le mer. 24 oct. 2018 à 20:59, Joe Witt a écrit : > Jon, > > There is very little overhead really. The answer more generally has > to do with history. Expression language was added later in the game > and there can be different scopes to what a given EL statement has > access to. Some EL are evaluated against flowfiles and some arent. > We didn't have a good way to show users this information though this > has recently improved. It also has to do with 'evaluation time' > meaning processors might evaluate some properties frequently/while > executing a task (as would be the case when EL statements have a given > flowfile in scope) or during enabling/scheduling such as pulling a > property value they intend to honor throughout an entire runtime. > > I suspect at this point that we can make most properties (not > enumerations most likely) EL enabled. But, each one requires > review/consideration to ensure the processor is coded to use it > correctly. > > Alternatively, we could explore making all text entry fields allow > users to explicitly state 'this is a variable and here is the variable > name' (but in a cool UI way) and then the actual value used would come > from the variable registry for that process group. This would be > something we can do across the board as it wouldnt' change the > processor's logic/behavior at all. Variables get set at and retained > throughout the life of a processor run. If a used variable is changed > we'd automatically stop/start each impacted component/processor so it > wouldnt' require any additional review/changes in processor > properties. > > ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran > wrote: > > > > Hello there, > > Regarding expression language properties, why aren't all or most > properties enabled by default. > > > > Is there substantial processor overhead on these? > > > > Thanks, > > Jon > > > > On 10/24/18, 1:49 PM, "Joe Witt" wrote: > > > > Mark > > > > There are two scenarios here to discuss: > > 1) What do to with sensitive properties > > 2) What to do with properties that dont allow expression language > statements > > > > For #1 we dont send those properties into the registry and they are > > set (properly) in each environment where the secret belongs. All > good > > as that doesn't change the flow definition. > > > > For #2 we should just look at which properties are causing trouble > and > > see about expression language enabling them. So please share > > precisely which ones you're hitting where that would help you out and > > lets see what can be done. > > > > Thanks > > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton > > wrote: > > > > > > Hi Everyone, > > > > > > I'm currently doing a lot of work with Nifi and recently we have > been trying to come up with a solution to a problem. We installed Nifi > registry backed by our Git repository for versioning our flows. This has > worked out great for us as we can now track the version of our flows > correctly and make sure they are backed up in source control. > > > > > > However when we want to do deployment between our Development Nifi > cluster and our Qa Nifi cluster we have to ofcourse change some values. > These could be amqp queues, directories on the file system etc. > > > > > > So ofcourse we use variables so that we can configure the values > without it being detected as a change to the flow. A problem arises however > when we need to configure an option that does not support expression > language. For example the host name of the amqp processors. > > > > > > This leaves us in a situation where a change to the flow is > detected. The only real option I have as far as I can see is to clone the > flows and have one for each environment which I don't like at all. > > > > > > Is anyone else struggling with similar issues. If so how are you > handling It? > > > > > > Sent from my Sony Xperia™ smartphone > > > > >
Re: [External] Re: Expression language and NiFi registry
Jon, There is very little overhead really. The answer more generally has to do with history. Expression language was added later in the game and there can be different scopes to what a given EL statement has access to. Some EL are evaluated against flowfiles and some arent. We didn't have a good way to show users this information though this has recently improved. It also has to do with 'evaluation time' meaning processors might evaluate some properties frequently/while executing a task (as would be the case when EL statements have a given flowfile in scope) or during enabling/scheduling such as pulling a property value they intend to honor throughout an entire runtime. I suspect at this point that we can make most properties (not enumerations most likely) EL enabled. But, each one requires review/consideration to ensure the processor is coded to use it correctly. Alternatively, we could explore making all text entry fields allow users to explicitly state 'this is a variable and here is the variable name' (but in a cool UI way) and then the actual value used would come from the variable registry for that process group. This would be something we can do across the board as it wouldnt' change the processor's logic/behavior at all. Variables get set at and retained throughout the life of a processor run. If a used variable is changed we'd automatically stop/start each impacted component/processor so it wouldnt' require any additional review/changes in processor properties. ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran wrote: > > Hello there, > Regarding expression language properties, why aren't all or most properties > enabled by default. > > Is there substantial processor overhead on these? > > Thanks, > Jon > > On 10/24/18, 1:49 PM, "Joe Witt" wrote: > > Mark > > There are two scenarios here to discuss: > 1) What do to with sensitive properties > 2) What to do with properties that dont allow expression language > statements > > For #1 we dont send those properties into the registry and they are > set (properly) in each environment where the secret belongs. All good > as that doesn't change the flow definition. > > For #2 we should just look at which properties are causing trouble and > see about expression language enabling them. So please share > precisely which ones you're hitting where that would help you out and > lets see what can be done. > > Thanks > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton > wrote: > > > > Hi Everyone, > > > > I'm currently doing a lot of work with Nifi and recently we have been > trying to come up with a solution to a problem. We installed Nifi registry > backed by our Git repository for versioning our flows. This has worked out > great for us as we can now track the version of our flows correctly and make > sure they are backed up in source control. > > > > However when we want to do deployment between our Development Nifi > cluster and our Qa Nifi cluster we have to ofcourse change some values. These > could be amqp queues, directories on the file system etc. > > > > So ofcourse we use variables so that we can configure the values > without it being detected as a change to the flow. A problem arises however > when we need to configure an option that does not support expression > language. For example the host name of the amqp processors. > > > > This leaves us in a situation where a change to the flow is detected. > The only real option I have as far as I can see is to clone the flows and > have one for each environment which I don't like at all. > > > > Is anyone else struggling with similar issues. If so how are you > handling It? > > > > Sent from my Sony Xperia™ smartphone > >
Re: [External] Re: Expression language and NiFi registry
Hello there, Regarding expression language properties, why aren't all or most properties enabled by default. Is there substantial processor overhead on these? Thanks, Jon On 10/24/18, 1:49 PM, "Joe Witt" wrote: Mark There are two scenarios here to discuss: 1) What do to with sensitive properties 2) What to do with properties that dont allow expression language statements For #1 we dont send those properties into the registry and they are set (properly) in each environment where the secret belongs. All good as that doesn't change the flow definition. For #2 we should just look at which properties are causing trouble and see about expression language enabling them. So please share precisely which ones you're hitting where that would help you out and lets see what can be done. Thanks On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton wrote: > > Hi Everyone, > > I'm currently doing a lot of work with Nifi and recently we have been trying to come up with a solution to a problem. We installed Nifi registry backed by our Git repository for versioning our flows. This has worked out great for us as we can now track the version of our flows correctly and make sure they are backed up in source control. > > However when we want to do deployment between our Development Nifi cluster and our Qa Nifi cluster we have to ofcourse change some values. These could be amqp queues, directories on the file system etc. > > So ofcourse we use variables so that we can configure the values without it being detected as a change to the flow. A problem arises however when we need to configure an option that does not support expression language. For example the host name of the amqp processors. > > This leaves us in a situation where a change to the flow is detected. The only real option I have as far as I can see is to clone the flows and have one for each environment which I don't like at all. > > Is anyone else struggling with similar issues. If so how are you handling It? > > Sent from my Sony Xperia™ smartphone
Re: Local development and testing w/ kerberos
Thanks. Got that working and was able to login as mthom...@nifi.apache.org. On Wed, Oct 24, 2018 at 11:24 AM Bryan Bende wrote: > I think all your kerberos/KDC stuff is fine, you just need to add > mthom...@nifi.apache.org to the user-group-provider. > > My post was old before we had separated authorizer into > user-group-provider and access-policy-provider. > On Wed, Oct 24, 2018 at 11:18 AM Mike Thomsen > wrote: > > > > Alright, I think I'm pretty close here. I followed all of those steps, > > except I changed bbende to mthomsen. > > > > * I can run kinit mthom...@nifi.apache.org and it works. > > * I can run klist and see the expected output. > > > > When I bring up NiFi, I get the following (trimmed for brevity): > > > > Caused by: > > org.apache.nifi.authorization.exception.AuthorizerCreationException: > > org.apache.nifi.authorization.exception.AuthorizerCreationException: > Unable > > to locate initial admin mthom...@nifi.apache.org to seed policies > > at > > > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:263) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54) > > at com.sun.proxy.$Proxy76.onConfigured(Unknown Source) > > at > > > org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:152) > > at > > > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) > > ... 96 common frames omitted > > Caused by: > > org.apache.nifi.authorization.exception.AuthorizerCreationException: > Unable > > to locate initial admin mthom...@nifi.apache.org to seed policies > > at > > > org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:598) > > at > > > org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:541) > > at > > > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:254) > > ... 104 common frames omitted > > > > I double-checked the paths to krb5.conf and the keytab and they're both > > pointing to /tmp/docker-kdc > > > > Any ideas? > > > > Thanks, > > > > Mike > > > > > > On Wed, Oct 24, 2018 at 10:28 AM Mike Thomsen > > wrote: > > > > > Awesome, thanks Bryan! I'm halfway through that (got klist view) and > it's > > > working great so far. > > > > > > On Wed, Oct 24, 2018 at 9:36 AM Bryan Bende wrote: > > > > > >> There is a docker-kdc project that is easy to use: > > >> > > >> > > >> > https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication > > >> > > >> It was made before docker for mac was good/popular and it previously > > >> relied on boot2docker, but I made the following modification to not > > >> use boot2docker > > >> > > >> docker-kdc$ git diff > > >> diff --git a/kdc b/kdc > > >> index 9410fc5..0a887e1 100755 > > >> --- a/kdc > > >> +++ b/kdc > > >> @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' > > >> GET_KDC_HOST="echo $KDC_NATHOST" > > >> > > >> # Adjust container in case of OSX. > > >> -if [[ $OSTYPE =~ darwin.+ ]]; then > > >> - CONTAINER='boot2docker' > > >> - GET_KDC_HOST='boot2docker ip' > > >> -fi > > >> +#if [[ $OSTYPE =~ darwin.+ ]]; then > > >> +# CONTAINER='boot2docker' > > >> +# GET_KDC_HOST='boot2docker ip' > > >> +#fi > > >> > > >> On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen > > >> wrote: > > >> > > > >> > Looking for suggestions on local development and testing with > kerberos. > > >> We > > >> > have a kerberized cluster set up in an AWS instance, but it's more > for > > >> UAT > > >> > than development. Anyone have any suggestions/experience, say, > setting > > >> up a > > >> > Mac or Linux box for developing and testing like this? > > >> > > > >> > Thanks, > > >> > > > >> > Mike > > >> > > > >
Re: Local development and testing w/ kerberos
I think all your kerberos/KDC stuff is fine, you just need to add mthom...@nifi.apache.org to the user-group-provider. My post was old before we had separated authorizer into user-group-provider and access-policy-provider. On Wed, Oct 24, 2018 at 11:18 AM Mike Thomsen wrote: > > Alright, I think I'm pretty close here. I followed all of those steps, > except I changed bbende to mthomsen. > > * I can run kinit mthom...@nifi.apache.org and it works. > * I can run klist and see the expected output. > > When I bring up NiFi, I get the following (trimmed for brevity): > > Caused by: > org.apache.nifi.authorization.exception.AuthorizerCreationException: > org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable > to locate initial admin mthom...@nifi.apache.org to seed policies > at > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:263) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54) > at com.sun.proxy.$Proxy76.onConfigured(Unknown Source) > at > org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:152) > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) > ... 96 common frames omitted > Caused by: > org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable > to locate initial admin mthom...@nifi.apache.org to seed policies > at > org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:598) > at > org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:541) > at > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:254) > ... 104 common frames omitted > > I double-checked the paths to krb5.conf and the keytab and they're both > pointing to /tmp/docker-kdc > > Any ideas? > > Thanks, > > Mike > > > On Wed, Oct 24, 2018 at 10:28 AM Mike Thomsen > wrote: > > > Awesome, thanks Bryan! I'm halfway through that (got klist view) and it's > > working great so far. > > > > On Wed, Oct 24, 2018 at 9:36 AM Bryan Bende wrote: > > > >> There is a docker-kdc project that is easy to use: > >> > >> > >> https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication > >> > >> It was made before docker for mac was good/popular and it previously > >> relied on boot2docker, but I made the following modification to not > >> use boot2docker > >> > >> docker-kdc$ git diff > >> diff --git a/kdc b/kdc > >> index 9410fc5..0a887e1 100755 > >> --- a/kdc > >> +++ b/kdc > >> @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' > >> GET_KDC_HOST="echo $KDC_NATHOST" > >> > >> # Adjust container in case of OSX. > >> -if [[ $OSTYPE =~ darwin.+ ]]; then > >> - CONTAINER='boot2docker' > >> - GET_KDC_HOST='boot2docker ip' > >> -fi > >> +#if [[ $OSTYPE =~ darwin.+ ]]; then > >> +# CONTAINER='boot2docker' > >> +# GET_KDC_HOST='boot2docker ip' > >> +#fi > >> > >> On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen > >> wrote: > >> > > >> > Looking for suggestions on local development and testing with kerberos. > >> We > >> > have a kerberized cluster set up in an AWS instance, but it's more for > >> UAT > >> > than development. Anyone have any suggestions/experience, say, setting > >> up a > >> > Mac or Linux box for developing and testing like this? > >> > > >> > Thanks, > >> > > >> > Mike > >> > >
Re: Local development and testing w/ kerberos
Can you share the authorizers.xml? I guess something wrong with the CN that’s mentioned there. - Sivaprasanna On Wed, 24 Oct 2018 at 8:48 PM, Mike Thomsen wrote: > Alright, I think I'm pretty close here. I followed all of those steps, > except I changed bbende to mthomsen. > > * I can run kinit mthom...@nifi.apache.org and it works. > * I can run klist and see the expected output. > > When I bring up NiFi, I get the following (trimmed for brevity): > > Caused by: > org.apache.nifi.authorization.exception.AuthorizerCreationException: > org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable > to locate initial admin mthom...@nifi.apache.org to seed policies > at > > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:263) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > > org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54) > at com.sun.proxy.$Proxy76.onConfigured(Unknown Source) > at > > org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:152) > at > > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) > ... 96 common frames omitted > Caused by: > org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable > to locate initial admin mthom...@nifi.apache.org to seed policies > at > > org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:598) > at > > org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:541) > at > > org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:254) > ... 104 common frames omitted > > I double-checked the paths to krb5.conf and the keytab and they're both > pointing to /tmp/docker-kdc > > Any ideas? > > Thanks, > > Mike > > > On Wed, Oct 24, 2018 at 10:28 AM Mike Thomsen > wrote: > > > Awesome, thanks Bryan! I'm halfway through that (got klist view) and it's > > working great so far. > > > > On Wed, Oct 24, 2018 at 9:36 AM Bryan Bende wrote: > > > >> There is a docker-kdc project that is easy to use: > >> > >> > >> > https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication > >> > >> It was made before docker for mac was good/popular and it previously > >> relied on boot2docker, but I made the following modification to not > >> use boot2docker > >> > >> docker-kdc$ git diff > >> diff --git a/kdc b/kdc > >> index 9410fc5..0a887e1 100755 > >> --- a/kdc > >> +++ b/kdc > >> @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' > >> GET_KDC_HOST="echo $KDC_NATHOST" > >> > >> # Adjust container in case of OSX. > >> -if [[ $OSTYPE =~ darwin.+ ]]; then > >> - CONTAINER='boot2docker' > >> - GET_KDC_HOST='boot2docker ip' > >> -fi > >> +#if [[ $OSTYPE =~ darwin.+ ]]; then > >> +# CONTAINER='boot2docker' > >> +# GET_KDC_HOST='boot2docker ip' > >> +#fi > >> > >> On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen > >> wrote: > >> > > >> > Looking for suggestions on local development and testing with > kerberos. > >> We > >> > have a kerberized cluster set up in an AWS instance, but it's more for > >> UAT > >> > than development. Anyone have any suggestions/experience, say, setting > >> up a > >> > Mac or Linux box for developing and testing like this? > >> > > >> > Thanks, > >> > > >> > Mike > >> > > >
Re: Local development and testing w/ kerberos
Alright, I think I'm pretty close here. I followed all of those steps, except I changed bbende to mthomsen. * I can run kinit mthom...@nifi.apache.org and it works. * I can run klist and see the expected output. When I bring up NiFi, I get the following (trimmed for brevity): Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin mthom...@nifi.apache.org to seed policies at org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:263) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54) at com.sun.proxy.$Proxy76.onConfigured(Unknown Source) at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:152) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) ... 96 common frames omitted Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin mthom...@nifi.apache.org to seed policies at org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:598) at org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:541) at org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:254) ... 104 common frames omitted I double-checked the paths to krb5.conf and the keytab and they're both pointing to /tmp/docker-kdc Any ideas? Thanks, Mike On Wed, Oct 24, 2018 at 10:28 AM Mike Thomsen wrote: > Awesome, thanks Bryan! I'm halfway through that (got klist view) and it's > working great so far. > > On Wed, Oct 24, 2018 at 9:36 AM Bryan Bende wrote: > >> There is a docker-kdc project that is easy to use: >> >> >> https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication >> >> It was made before docker for mac was good/popular and it previously >> relied on boot2docker, but I made the following modification to not >> use boot2docker >> >> docker-kdc$ git diff >> diff --git a/kdc b/kdc >> index 9410fc5..0a887e1 100755 >> --- a/kdc >> +++ b/kdc >> @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' >> GET_KDC_HOST="echo $KDC_NATHOST" >> >> # Adjust container in case of OSX. >> -if [[ $OSTYPE =~ darwin.+ ]]; then >> - CONTAINER='boot2docker' >> - GET_KDC_HOST='boot2docker ip' >> -fi >> +#if [[ $OSTYPE =~ darwin.+ ]]; then >> +# CONTAINER='boot2docker' >> +# GET_KDC_HOST='boot2docker ip' >> +#fi >> >> On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen >> wrote: >> > >> > Looking for suggestions on local development and testing with kerberos. >> We >> > have a kerberized cluster set up in an AWS instance, but it's more for >> UAT >> > than development. Anyone have any suggestions/experience, say, setting >> up a >> > Mac or Linux box for developing and testing like this? >> > >> > Thanks, >> > >> > Mike >> >
Re: Local development and testing w/ kerberos
Awesome, thanks Bryan! I'm halfway through that (got klist view) and it's working great so far. On Wed, Oct 24, 2018 at 9:36 AM Bryan Bende wrote: > There is a docker-kdc project that is easy to use: > > > https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication > > It was made before docker for mac was good/popular and it previously > relied on boot2docker, but I made the following modification to not > use boot2docker > > docker-kdc$ git diff > diff --git a/kdc b/kdc > index 9410fc5..0a887e1 100755 > --- a/kdc > +++ b/kdc > @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' > GET_KDC_HOST="echo $KDC_NATHOST" > > # Adjust container in case of OSX. > -if [[ $OSTYPE =~ darwin.+ ]]; then > - CONTAINER='boot2docker' > - GET_KDC_HOST='boot2docker ip' > -fi > +#if [[ $OSTYPE =~ darwin.+ ]]; then > +# CONTAINER='boot2docker' > +# GET_KDC_HOST='boot2docker ip' > +#fi > > On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen > wrote: > > > > Looking for suggestions on local development and testing with kerberos. > We > > have a kerberized cluster set up in an AWS instance, but it's more for > UAT > > than development. Anyone have any suggestions/experience, say, setting > up a > > Mac or Linux box for developing and testing like this? > > > > Thanks, > > > > Mike >
Re: Local development and testing w/ kerberos
There is a docker-kdc project that is easy to use: https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication It was made before docker for mac was good/popular and it previously relied on boot2docker, but I made the following modification to not use boot2docker docker-kdc$ git diff diff --git a/kdc b/kdc index 9410fc5..0a887e1 100755 --- a/kdc +++ b/kdc @@ -90,10 +90,10 @@ CONTROL_VM='VBoxManage controlvm boot2docker-vm' GET_KDC_HOST="echo $KDC_NATHOST" # Adjust container in case of OSX. -if [[ $OSTYPE =~ darwin.+ ]]; then - CONTAINER='boot2docker' - GET_KDC_HOST='boot2docker ip' -fi +#if [[ $OSTYPE =~ darwin.+ ]]; then +# CONTAINER='boot2docker' +# GET_KDC_HOST='boot2docker ip' +#fi On Wed, Oct 24, 2018 at 7:35 AM Mike Thomsen wrote: > > Looking for suggestions on local development and testing with kerberos. We > have a kerberized cluster set up in an AWS instance, but it's more for UAT > than development. Anyone have any suggestions/experience, say, setting up a > Mac or Linux box for developing and testing like this? > > Thanks, > > Mike
Re: CSVreader incorrectly parse RFC4180
I think you're running into NIFI-5525 [1], it should be fixed in master as well as the upcoming 1.8.0 release. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-5525 On Wed, Oct 24, 2018 at 7:14 AM Nalinrat Laekawan wrote: > Hi nifi team, > > I found some issue with CSVreader in NIFI. I tried to parse the CSV file > in RFC 4180 format. The data has some rows with a text value of “”””, which > according to RFC 4180 should be parsed to be a single double quote (“). > However, the NIFI job failed with the error of ‘StringIndexOutofBound’. > > > > A flow that I’m implementing and some of mock up data that I’m using for > testing are attached. > > > > Here is the results of the files that I attached, for the id ‘1’ has the > error with a text value of “”””. > [image: image.png] >
Local development and testing w/ kerberos
Looking for suggestions on local development and testing with kerberos. We have a kerberized cluster set up in an AWS instance, but it's more for UAT than development. Anyone have any suggestions/experience, say, setting up a Mac or Linux box for developing and testing like this? Thanks, Mike
CSVreader incorrectly parse RFC4180
Hi nifi team, I found some issue with CSVreader in NIFI. I tried to parse the CSV file in RFC 4180 format. The data has some rows with a text value of “”””, which according to RFC 4180 should be parsed to be a single double quote (“). However, the NIFI job failed with the error of ‘StringIndexOutofBound’. A flow that I’m implementing and some of mock up data that I’m using for testing are attached. Here is the results of the files that I attached, for the id ‘1’ has the error with a text value of “”””. [image: image.png] <>
Re: [VOTE] Release Apache NiFi 1.8.0 (RC3)
+1 (binding) Ran through the release helper. No issue was found. Thanks for RM duties, Jeff! On Wed, Oct 24, 2018 at 1:42 PM James Wing wrote: > > +1 (binding). Ran though the release helper, tested the resulting binary. > Thank you for your persistence, Jeff. > > > On Mon, Oct 22, 2018 at 10:56 PM Jeff wrote: > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache NiFi > > nifi-1.8.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1135 > > > > The Git tag is nifi-1.8.0-RC3 > > The Git commit ID is 98aabf2c50f857efc72fd6f2bfdd9965b97fa195 > > > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=98aabf2c50f857efc72fd6f2bfdd9965b97fa195 > > > > Checksums of nifi-1.8.0-source-release.zip: > > SHA256: 6ec21c36ebb232f344493a4aeb5086eed0c462c576e11a79abed8149bc8b65c3 > > SHA512: > > > > 846aecd4eb497a3b7dee7d1911b02453b8162b6c87e39f3df837744a478212e2e3e3615921079d29c2804671f26ecd05b04ce46a4bb69e8911fc185e27be9c24 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/jstorck.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 209 issues were closed/resolved for this release: > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343482 > > > > Release note highlights can be found here: > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build > > from source, and test. Then please vote: > > > > [ ] +1 Release this package as nifi-1.8.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > >
