Re: Policy storage in Nifi runner and Kubernetes
ok, it appear my nifi node doesn't start an embedded zookeeper server. Can it be related ? Le 30/09/2019 à 09:59, Nicolas Delsaux a écrit : Hi all I'm running my Nifi node in Kubernetes. For that, the /opt/nifi/nifi-current/conf folder is made writable by an init container prior to starting the runner. All other interesting folders (the *_repository ones) are stored on a read-write-once volume. Each time the pod is restarted, the flow id changes, the policies applied to nifi-runner are lost, and the process groups disappear from the flow. I've tried to look at other K8s examples of nifi configuration, but can't find how to have my nifi runner restarting correctly ... (maybe it' because I'm not that good at configuring K8s pods) So what is best way to have my configuration working ? Thanks !
Policy storage in Nifi runner and Kubernetes
Hi all I'm running my Nifi node in Kubernetes. For that, the /opt/nifi/nifi-current/conf folder is made writable by an init container prior to starting the runner. All other interesting folders (the *_repository ones) are stored on a read-write-once volume. Each time the pod is restarted, the flow id changes, the policies applied to nifi-runner are lost, and the process groups disappear from the flow. I've tried to look at other K8s examples of nifi configuration, but can't find how to have my nifi runner restarting correctly ... (maybe it' because I'm not that good at configuring K8s pods) So what is best way to have my configuration working ? Thanks !
Re: can't ush data to bigQuery
Oh well, i've understood my last error : incorrect flow files (with JSOn
arrays) were stuck in the queue.
I removed them and ... to my delight, data seems to come in BigQuery !
Le 26/09/2019 à 14:45, Nicolas Delsaux a écrit :
I didn't knew that command ... i've edited some confidential values in
the result, but here it is
$ bq --project_id={{PROJECT_ID}} --format=prettyjson show -j
9e790299-dc77-46f4-8978-476f284fe5b5
{
"configuration": {
"jobType": "LOAD",
"load": {
"createDisposition": "CREATE_IF_NEEDED",
"destinationTable": {
"datasetId": "Consents",
"projectId": "{{PROJECT_ID}}",
"tableId": "{{TABLE_ID}}"
},
"ignoreUnknownValues": false,
"maxBadRecords": 0,
"schema": {
"fields": [
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "type",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "businessUnit",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "identity",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "finality",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "consentDate",
"type": "TIMESTAMP"
},
{
"mode": "NULLABLE",
"name": "expiryDate",
"type": "TIMESTAMP"
},
{
"mode": "NULLABLE",
"name": "expired",
"type": "BOOLEAN"
},
{
"mode": "NULLABLE",
"name": "createdBy",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "createdDate",
"type": "TIMESTAMP"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "application",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "type",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "sender",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "type",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "relatedEvent",
"type": "RECORD"
}
],
"mode": "NULLABLE",
"name": "Contractual
Re: can't ush data to bigQuery
}
},
"etag": "RqYxd6o2jzl6YiTARI5nxg==",
"id": "{{PROJECT_ID}}:EU.9e790299-dc77-46f4-8978-476f284fe5b5",
"jobReference": {
"jobId": "9e790299-dc77-46f4-8978-476f284fe5b5",
"location": "EU",
"projectId": "{{PROJECT_ID}}"
},
"kind": "bigquery#job",
"selfLink":
"https://bigquery.googleapis.com/bigquery/v2/projects/{{PROJECT_ID}}/jobs/9e790299-dc77-46f4-8978-476f284fe5b5?location=EU;,
"statistics": {
"creationTime": "1569491661818",
"endTime": "1569491662935",
"startTime": "1569491662366"
},
"status": {
"errorResult": {
"message": "Error while reading data, error message: JSON table
encountered too many errors, giving up. Rows: 1; errors: 1. Please look
into the errors[] collection for more details.",
"reason": "invalid"
},
"errors": [
{
"message": "Error while reading data, error message: JSON table
encountered too many errors, giving up. Rows: 1; errors: 1. Please look
into the errors[] collection for more details.",
"reason": "invalid"
},
{
"message": "Error while reading data, error message: JSON
processing encountered too many errors, giving up. Rows: 1; errors: 1;
max bad: 0; error percent: 0",
"reason": "invalid"
},
{
"message": "Error while reading data, error message: JSON
parsing error in row starting at position 0: Start of array encountered
without start of object.",
"reason": "invalid"
}
],
"state": "DONE"
},
"user_email":
"rabbitmq-inges...@psh-analytics-automation.iam.gserviceaccount.com"
}
Error message is interesting.
If I look in data provenance at the data I'm expected to send to
BigQuery, I get
[{"ContractualConsent":{"id":"5d847c5c92913700017692fc","identity":{"id":"511096128","type":"customer","businessUnit":"lmit"},"finality":"commercial_relationship","consentDate":"2019-06-04T15:39:32Z","expiryDate":"2024-06-04T15:39:32Z","expired":false,"createdBy":"DynamoCRM_DC","createdDate":"2019-09-20T07:14:36.576Z","sender":{"id":"511096128","application":"DYNAMO-CRM","type":"CUSTOMER"},"relatedEvent":{"id":"a72c44f1-de86-e911-a827-000d3a2aa91d","type":"customer_request"}}},{"ContractualConsent":{"id":"5d847c5c5fa9420001ebf04e","identity":{"id":"509582521","type":"customer","businessUnit":"lmit"},"finality":"commercial_relationship","consentDate":"2019-06-07T08:09:32Z","expiryDate":"2024-06-07T08:09:32Z","expired":false,"createdBy":"DynamoCRM_DC","createdDate":"2019-09-20T07:14:36.708Z","sender":{"id":"509582521","application":"DYNAMO-CRM","type":"CUSTOMER"},"relatedEvent":{"id":"6c335392-fb88-e911-a827-000d3a2aa91d","type":"customer_request"}}}]
Which is indeed an array, instead of an object.
And maybe it is because my JsonRecordSetWriter has for "Output grouping"
the "Array" value selected ...
Well, strangely, even after having changed configuration of my
JsoNRecordSetwriter, values continue to be json arrays ...
Anyway, I guess i'm on the right path ... (thanks a lot Pierre)
Le 26/09/2019 à 13:18, Pierre Villard a écrit :
What if you run the below command in Cloud Shell:
bq --format=prettyjson show -j
In your case (with your last email):
bq --format=prettyjson show -j 9e790299-dc77-46f4-8978-476f284fe5b5
Does it give you more details?
Le jeu. 26 sept. 2019 à 12:13, Nicolas Delsaux <mailto:nicolas.dels...@gmx.fr>> a écrit :
Sorry for the late reply.
As of today, the issue is still present.
Nifi Web UI just shows the message "Error while reading data,
error message: JSON table encountered too many errors, giving up.
Rows: 1; errors: 1. Please look into the errors[] collection for
more details."
But the log is clearer :
--
Standard FlowFile Attributes
Key: 'entryDate'
Value: '
Re: can't ush data to bigQuery
Sorry for the late reply.
As of today, the issue is still present.
Nifi Web UI just shows the message "Error while reading data, error
message: JSON table encountered too many errors, giving up. Rows: 1;
errors: 1. Please look into the errors[] collection for more details."
But the log is clearer :
--
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Thu Sep 26 09:53:49 UTC 2019'
Key: 'lineageStartDate'
Value: 'Thu Sep 26 09:53:49 UTC 2019'
Key: 'fileSize'
Value: '999'
FlowFile Attribute Map Content
Key: 'avro.schema'
Value:
'{"type":"record","name":"nifiRecord","namespace":"org.apache.nifi","fields":[{"name":"ExplicitConsent","type":["null",{"type":"record","name":"ExplicitConsentType","fields":[{"name":"id","type":["null","string"]},{"name":"identity","type":["null",{"type":"record","name":"identityType","fields":[{"name":"id","type":["null","string"]},{"name":"type","type":["null","string"]},{"name":"businessUnit","type":["null","string"]}]}]},{"name":"finality","type":["null","string"]},{"name":"expired","type":["null","boolean"]},{"name":"createdBy","type":["null","string"]},{"name":"createdDate","type":["null","string"]},{"name":"sender","type":["null",{"type":"record","name":"senderType","fields":[{"name":"id","type":["null","string"]},{"name":"application","type":["null","string"]},{"name":"type","type":["null","string"]}]}]},{"name":"state","type":["null","string"]}]}]}]}'
Key: 'bq.error.message'
Value: 'Error while reading data, error message: JSON table encountered
too many errors, giving up. Rows: 1; errors: 1. Please look into the
errors[] collection for more details.'
Key: 'bq.error.reason'
Value: 'invalid'
Key: 'bq.job.link'
Value:
'https://www.googleapis.com/bigquery/v2/projects/psh-datacompliance/jobs/9e790299-dc77-46f4-8978-476f284fe5b5?location=EU'
Key: 'bq.job.stat.creation_time'
Value: '1569491661818'
Key: 'bq.job.stat.end_time'
Value: '1569491662935'
Key: 'bq.job.stat.start_time'
Value: '1569491662366'
Key: 'filename'
Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
Key: 'kafka.key'
Value: '--'
Key: 'kafka.partition'
Value: '0'
Key: 'kafka.topic'
Value: 'dc.consent-life-cycle.kpi-from-dev-nifi-json'
Key: 'merge.bin.age'
Value: '1'
Key: 'merge.count'
Value: '3'
Key: 'mime.type'
Value: 'application/json'
Key: 'path'
Value: './'
Key: 'record.count'
Value: '3'
Key: 'uuid'
Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
2019-09-26 10:09:39,633 INFO [Timer-Driven Process Thread-4]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=ce9c171f-0c8f-3cab-e0f2-16156faf15b8] logging for flow
file
StandardFlowFileRecord[uuid=e6d604d7-b517-4a87-a398-e4a5df342ce6,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1569490848560-6,
container=default, section=6], offset=569098,
length=999],offset=0,name=e6d604d7-b517-4a87-a398-e4a5df342ce6,size=999]
I don't exactly understand why i would have to set an authentication,
because I've set the service.json content into the GCP Credentials
Provider I use for my PutBigQueryBatch processor ...
Is there anything I'm missing ? or a simple way to make sure verything
work as expected ?
Thanks
Le 24/09/2019 à 16:12, Pierre Villard a écrit :
Hey Nicolas,
Did you manage to solve your issue? Happy to help on this one.
Thanks,
Pierre
Le ven. 20 sept. 2019 à 16:42, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit :
Hello
I'm using PutBigQueryBash and having weird auth issues.
I have set the GCP Credentials Controller Service to use Service
Account JSON which I have copied from the value given in Google
Cloud Console.
But when I run my flow, I get the error message "Error while
reading data, error message: JSON table encountered too many
errors, giving up. Rows: 1; errors: 1. Please look into the
errors[] collection for more details."
What is stranger is that when I log all properties, there is a
bq.job.link which messages indicate "Request is missing required
authentication credential. Expected OAuth 2 access token, login
cookie or other valid authentication credential. See
https://developers.google.com/identity/sign-in/web/devconsole-project.;
...
But nifi can access the bigquery workspace and dataset (I've
checked that by deleting the table schema that I have already
written).
So, is there something I'm doing wrong ?
Thanks !
Re: implementing policies through REST interface
Well, i managed to use the excellent nipyapi Python client which provides nearly all the calls I was needing. Thanks anyway ! Le 24/09/2019 à 16:25, Bryan Bende a écrit : The best way to figure out the REST calls would be to use the UI while you have Chrome Dev Tools open and go through the process of creating the policies you are interested in and then you'll see the requests that are made. In terms of the a REST client, there isn't really an official client, but a few items that might be of interest... 1) All of the Java classes for the entities and DTOs which are the input/output of the REST API are available here: https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto 2) The next version of NiFi CLI will have some new commands for managing users/groups/policies: https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/policies 3) There is a module in the toolkit that runs Swagger code-gen against the swagger spec of the REST API, so theoretically it produces some type of auto-generated client, although I haven't personally used it: https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-api On Tue, Sep 24, 2019 at 3:52 AM Nicolas Delsaux wrote: Hi all i'm deploying my nifi node in containers and, as a consqeunce, i have to periodically rewrite policies to have it working. As it is really painful, i would like to write a script that will write those policies automatically at first startup. Are there any tutorials about that ? I'm particularly thinking about getting the flow policies ids, which are UUID and for which there doesn't seems to exist any "get all" endpoint. Furthermore, is there any kind of "nifi rest client" java api ? Thanks !
implementing policies through REST interface
Hi all i'm deploying my nifi node in containers and, as a consqeunce, i have to periodically rewrite policies to have it working. As it is really painful, i would like to write a script that will write those policies automatically at first startup. Are there any tutorials about that ? I'm particularly thinking about getting the flow policies ids, which are UUID and for which there doesn't seems to exist any "get all" endpoint. Furthermore, is there any kind of "nifi rest client" java api ? Thanks !
can't ush data to bigQuery
Hello I'm using PutBigQueryBash and having weird auth issues. I have set the GCP Credentials Controller Service to use Service Account JSON which I have copied from the value given in Google Cloud Console. But when I run my flow, I get the error message "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details." What is stranger is that when I log all properties, there is a bq.job.link which messages indicate "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.; ... But nifi can access the bigquery workspace and dataset (I've checked that by deleting the table schema that I have already written). So, is there something I'm doing wrong ? Thanks !
Re: In nifi-registry, why can't I edit other users privileges
Well, in fact, I had a number of issues with configuration files. So I took the time to verify all those files, and I took the time to understand Nifi registry UI for permissions (which is as user-friendly as nifi one). And I finally understood what problem I had. In fact, the worst part came when I tried to understand why my nifi runner couldn't connect to nifi registry. Which was simply due to the fact that, on nifi registry side, in authorizers.xml, I used a property called "Nifi identify 1", whereas I should have used "NiFi Identity 1". Can you spot the difference ? For me, it took one phase of reading authorization code, then running the regexp for that property in an online editor. TO my mind, this would deserve a bug, because really, using property names this way is really too much error-prone. I would at least add code to detect nearby texts (through Levenshtein distance, as an example) and show a BIG warning to explain the user what is wrong. But I'm only a user ;-) (a little grumpy, this morning, indeed) Le 04/09/2019 à 18:59, Kevin Doran a écrit : Hi Nicolas, Is it possible you changed the initial admin identity at some point? If so, you will need to delete authorizations.xml and restart NiFi Registry to allow it to be recreated with the new initial admin. Also, nifi registry never allows modifying the permissions for the current user. you would have to login as another admin to change your permissions. Hope this helps, Kevin On Mon, Sep 2, 2019 at 8:56 AM Nicolas Delsaux wrote: Hi all I'm still trying to connect nifi to registry with both of them using authentication. So far, i've understood that, like in Nifi, I have to set identity-providers.xml and authorizers.xml to have connection to ldap configured. And I can connect to the registry using my ldap, so it works (to a certain extend). *However*, it seems like my user is not really an admin, as I can't manage other users. To say things more clearly, nifi-registry UI allow me to view my user privileges, but I can't edit my permissions, and I can edit none of the other users permissions. I can no more add/remove users. Which is weird, cosnidering I'm the initial admin of nifi-registry. Is there smothing I forgot ? Here is my authorizers.xml for nifi-registry file-user-group-provider org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider ./conf/users.xml cn=nifi-runner.mycompany.com, ou=0008 43120727, ou=ssl infra server, o=mycompany, c=fr ldap-user-group-provider org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider LDAPS uid=dont-ask-me,ou=applicationAccounts,o=mycompany.com YOU_KIDDIN___DO_YOU /opt/certs/cacerts.jks pfeblelep JKS TLSv1 FOLLOW 10 secs 10 secs ldaps://ldapserver.my.company.com:636 30 mins OBJECT cn=NIFI-ADMIN,ou=DATAou=applicationRole,ou=role,ou=OU,o=mycompany.com groupofuniquenames SUBTREE cn uniqueMember composite-user-group-provider org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider ldap-user-group-provider file-user-group-provider file-access-policy-provider org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider composite-user-group-provider ./conf/authorizations.xml uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com cn=nifi-psh.adeo.com, ou=0002 421206079, ou=ssl infra server, o=adeo services, c=fr managed-authorizer org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer file-access-policy-provider Thanks for your help
In nifi-registry, why can't I edit other users privileges
Hi all I'm still trying to connect nifi to registry with both of them using authentication. So far, i've understood that, like in Nifi, I have to set identity-providers.xml and authorizers.xml to have connection to ldap configured. And I can connect to the registry using my ldap, so it works (to a certain extend). *However*, it seems like my user is not really an admin, as I can't manage other users. To say things more clearly, nifi-registry UI allow me to view my user privileges, but I can't edit my permissions, and I can edit none of the other users permissions. I can no more add/remove users. Which is weird, cosnidering I'm the initial admin of nifi-registry. Is there smothing I forgot ? Here is my authorizers.xml for nifi-registry file-user-group-provider org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider ./conf/users.xml cn=nifi-runner.mycompany.com, ou=0008 43120727, ou=ssl infra server, o=mycompany, c=fr ldap-user-group-provider org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider LDAPS uid=dont-ask-me,ou=applicationAccounts,o=mycompany.com YOU_KIDDIN___DO_YOU /opt/certs/cacerts.jks pfeblelep JKS TLSv1 FOLLOW 10 secs 10 secs ldaps://ldapserver.my.company.com:636 30 mins OBJECT cn=NIFI-ADMIN,ou=DATAou=applicationRole,ou=role,ou=OU,o=mycompany.com groupofuniquenames SUBTREE cn uniqueMember composite-user-group-provider org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider ldap-user-group-provider file-user-group-provider file-access-policy-provider org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider composite-user-group-provider ./conf/authorizations.xml uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com cn=nifi-psh.adeo.com, ou=0002 421206079, ou=ssl infra server, o=adeo services, c=fr managed-authorizer org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer file-access-policy-provider Thanks for your help
Re: securing nifi-registry
Damn stupid of me ! I had to go in Java SSL code to understand that, due to PKCS12Keystore.java code, it seems like the private key password has to be the same than the keystore password, otherwise I get that funky error. So next time, maybe i will learn this burning lesson :-/ Le 29/08/2019 à 10:30, Nicolas Delsaux a écrit : Hi all I'm trying to secure my nifi registry. So i've created a keystore and a trustore, added to the keystore a private key entry, and configured my nifi-registry docker container to use that keystore/truststore. I can get the key pair in my keystore using keytool, both on my machine and in docker container. But when I start nifi-registry, I always get nifi-registry_1 | java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. nifi-registry_1 | at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:435) ~[na:1.8.0_212] nifi-registry_1 | at java.security.KeyStore.getKey(KeyStore.java:1023) ~[na:1.8.0_212] nifi-registry_1 | at sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:133) ~[na:1.8.0_212] nifi-registry_1 | at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70) ~[na:1.8.0_212] nifi-registry_1 | at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256) ~[na:1.8.0_212] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1113) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:309) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:229) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:72) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:279) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.Server.doStart(Server.java:398) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.apache.nifi.registry.jetty.JettyServer.start(JettyServer.java:423) ~[nifi-registry-jetty-0.4.0.jar:0.4.0] nifi-registry_1 | at org.apache.nifi.registry.NiFiRegistry.(NiFiRegistry.java:117) [nifi-registry-runtime-0.4.0.jar:0.4.0] nifi-registry_1 | at org.apache.nifi.registry.NiFiRegistry.main(NiFiRegistry.java:164) [nifi-registry-runtime-0.4.0.jar:0.4.0] nifi-registry_1 | Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. nifi-registry_1 | at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[sunjce_provider.jar:1.8.0_212
securing nifi-registry
Hi all I'm trying to secure my nifi registry. So i've created a keystore and a trustore, added to the keystore a private key entry, and configured my nifi-registry docker container to use that keystore/truststore. I can get the key pair in my keystore using keytool, both on my machine and in docker container. But when I start nifi-registry, I always get nifi-registry_1 | java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. nifi-registry_1 | at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:435) ~[na:1.8.0_212] nifi-registry_1 | at java.security.KeyStore.getKey(KeyStore.java:1023) ~[na:1.8.0_212] nifi-registry_1 | at sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:133) ~[na:1.8.0_212] nifi-registry_1 | at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70) ~[na:1.8.0_212] nifi-registry_1 | at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256) ~[na:1.8.0_212] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1113) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:309) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:229) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:72) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:279) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.server.Server.doStart(Server.java:398) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] nifi-registry_1 | at org.apache.nifi.registry.jetty.JettyServer.start(JettyServer.java:423) ~[nifi-registry-jetty-0.4.0.jar:0.4.0] nifi-registry_1 | at org.apache.nifi.registry.NiFiRegistry.(NiFiRegistry.java:117) [nifi-registry-runtime-0.4.0.jar:0.4.0] nifi-registry_1 | at org.apache.nifi.registry.NiFiRegistry.main(NiFiRegistry.java:164) [nifi-registry-runtime-0.4.0.jar:0.4.0] nifi-registry_1 | Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. nifi-registry_1 | at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipherCore.java:405) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engineDoFinal(PKCS12PBECipherCore.java:437) ~[sunjce_provider.jar:1.8.0_212] nifi-registry_1 | at
authenticated nifi agent wih unauthenticated registry
Hi all I have correctly setup my nifi runner to use LDAP auth from my company. I'm now trying to understand why registry no more work. As you may guess from message title, my registry is currently not authenticated. Do I need to have auth enabled on registry when it is enabled on nifi runner ? Thanks
Re: ldap authentication and initial admin identity
In that case, i guess the simplest way to improve things is to understand where I got lost. I successfully accessed the controller settings panel to add the nifi registry. But it was when I tried to add a process group that the permission issue byte me. So indeed, a tooltip (beside the disabled state of buttons) in top-level command bar indicating me that I had no permission and an admin should add those permissions for me would be a good solution... Provided that tooltip is clearly able to direct me to the permission dialog :-) Le 22/08/2019 à 12:03, Pierre Villard a écrit : Yeah me know we should try make things easier. On one side we want to have a very fine-grained multi-tenant model for permissions and on the other side we want user to quickly get up and running. If you have ideas to improve the overall experience, any feedback is greatly appreciated. I guess we could have a tool tip message in global Policies (when accessed by the hamburger menu) informing users that they might want to go at process group level to have granular policies. Le jeu. 22 août 2019 à 11:55, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Well, ok, i've understood by clicking everywhere :-) (discoverability of permission in nifi is ... ok ;-) ). So, i've clicked the "manage access policies" item in the right-click menu of the canvas and added my user everywhere, and now I can use the UI. Thanks for your patience :-) Le 22/08/2019 à 11:51, Pierre Villard a écrit : By default the initial admin does not have permissions to do anything on the canvas: the initial admin is usually used to manage users/groups and apply policies to grant permissions to users/groups. If you want to grant permissions to do something on the canvas, this is done at process group level. If you want to define permissions for the whole canvas, you can go in the policies of the root process group and grant your initial admin the corresponding policies. The policies you can grant using the hamburger menu / policies are more global policies (the ones you listed below). https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-config-examples Hope this helps. Le jeu. 22 août 2019 à 11:06, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Well, I sort of sorted it out. I can indeed login with my ldap, which is cool, but the whole UI is ... grayed : I can't create process groups jor import existing ones. So i took a look at the user screen. My permissions are as follows Global policy to access all policies write Global policy to access all policies read Global policy to access the controller write Global policy to access the controller read Global policy to access users/user groups write Global policy to access users/user groups read Global policy to view the user interface read Restricted components regardless of restrictions write I guess I have some invalid policies configured. But which ones ? And how to have them changed considering my user is configured from my ldap account ? Le 20/08/2019 à 11:55, Pierre Villard a écrit : Cool! Glad you got it sorted out! Le mar. 20 août 2019 à 11:30, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Wow, I'm really REALLY puzzled. I'm using Nifi through the docker image, and docker-compose. I was used to do docker-compose up/down, and it failed. But this time, I did a docker-compose down, AND destroyed the folder in which the application is deployed. And this time, it worked ! I'm now logged in as my ldap uid. Thank you very much Pierre ! Le 20/08/2019 à 10:55, Pierre Villard a écrit : Something that I can suggest: the users.xml and authorizations.xml files are generated when NiFi starts for the first time. If you did some modifications (such as the initial admin identity), the files users/authorizations won't be updated with your configuration change... Something you could try: delete authorizations.xml and users.xml files and restart NiFi to be sure it uses the last version of your configuration. Le mar. 20 août 2019 à 10:33, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : When I try to login, UI shows Insufficient Permissions Unable to view the user interface. Contact the system administrator. The log file contains 2019-08-20 08:22:18,808 INFO [main] o.a.n.a.FileA
Re: ldap authentication and initial admin identity
Well, ok, i've understood by clicking everywhere :-) (discoverability of permission in nifi is ... ok ;-) ). So, i've clicked the "manage access policies" item in the right-click menu of the canvas and added my user everywhere, and now I can use the UI. Thanks for your patience :-) Le 22/08/2019 à 11:51, Pierre Villard a écrit : By default the initial admin does not have permissions to do anything on the canvas: the initial admin is usually used to manage users/groups and apply policies to grant permissions to users/groups. If you want to grant permissions to do something on the canvas, this is done at process group level. If you want to define permissions for the whole canvas, you can go in the policies of the root process group and grant your initial admin the corresponding policies. The policies you can grant using the hamburger menu / policies are more global policies (the ones you listed below). https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-config-examples Hope this helps. Le jeu. 22 août 2019 à 11:06, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Well, I sort of sorted it out. I can indeed login with my ldap, which is cool, but the whole UI is ... grayed : I can't create process groups jor import existing ones. So i took a look at the user screen. My permissions are as follows Global policy to access all policies write Global policy to access all policies read Global policy to access the controller write Global policy to access the controller read Global policy to access users/user groups write Global policy to access users/user groups read Global policy to view the user interface read Restricted components regardless of restrictions write I guess I have some invalid policies configured. But which ones ? And how to have them changed considering my user is configured from my ldap account ? Le 20/08/2019 à 11:55, Pierre Villard a écrit : Cool! Glad you got it sorted out! Le mar. 20 août 2019 à 11:30, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Wow, I'm really REALLY puzzled. I'm using Nifi through the docker image, and docker-compose. I was used to do docker-compose up/down, and it failed. But this time, I did a docker-compose down, AND destroyed the folder in which the application is deployed. And this time, it worked ! I'm now logged in as my ldap uid. Thank you very much Pierre ! Le 20/08/2019 à 10:55, Pierre Villard a écrit : Something that I can suggest: the users.xml and authorizations.xml files are generated when NiFi starts for the first time. If you did some modifications (such as the initial admin identity), the files users/authorizations won't be updated with your configuration change... Something you could try: delete authorizations.xml and users.xml files and restart NiFi to be sure it uses the last version of your configuration. Le mar. 20 août 2019 à 10:33, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : When I try to login, UI shows Insufficient Permissions Unable to view the user interface. Contact the system administrator. The log file contains 2019-08-20 08:22:18,808 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Tue Aug 20 08:22:18 UTC 2019 2019-08-20 08:28:24,459 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2019-08-20 08:28:24,521 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. 2019-08-20 08:28:24,678 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does not have permission to access the requested resource. Unknown user with identity 'anonymous'. Returning Unauthorized response. 2019-08-20 08:28:31,702 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET https://nifi-psh.adeo.com:8443/nifi-api/flow/current-user (source ip: 172.20.0.1) 2019-08-20 08:28:31,710 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com <http://corp.leroymer
Re: ldap authentication and initial admin identity
Well, I sort of sorted it out. I can indeed login with my ldap, which is cool, but the whole UI is ... grayed : I can't create process groups jor import existing ones. So i took a look at the user screen. My permissions are as follows Global policy to access all policies write Global policy to access all policies read Global policy to access the controller write Global policy to access the controller read Global policy to access users/user groups write Global policy to access users/user groups read Global policy to view the user interface read Restricted components regardless of restrictions write I guess I have some invalid policies configured. But which ones ? And how to have them changed considering my user is configured from my ldap account ? Le 20/08/2019 à 11:55, Pierre Villard a écrit : Cool! Glad you got it sorted out! Le mar. 20 août 2019 à 11:30, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Wow, I'm really REALLY puzzled. I'm using Nifi through the docker image, and docker-compose. I was used to do docker-compose up/down, and it failed. But this time, I did a docker-compose down, AND destroyed the folder in which the application is deployed. And this time, it worked ! I'm now logged in as my ldap uid. Thank you very much Pierre ! Le 20/08/2019 à 10:55, Pierre Villard a écrit : Something that I can suggest: the users.xml and authorizations.xml files are generated when NiFi starts for the first time. If you did some modifications (such as the initial admin identity), the files users/authorizations won't be updated with your configuration change... Something you could try: delete authorizations.xml and users.xml files and restart NiFi to be sure it uses the last version of your configuration. Le mar. 20 août 2019 à 10:33, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : When I try to login, UI shows Insufficient Permissions Unable to view the user interface. Contact the system administrator. The log file contains 2019-08-20 08:22:18,808 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Tue Aug 20 08:22:18 UTC 2019 2019-08-20 08:28:24,459 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2019-08-20 08:28:24,521 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. 2019-08-20 08:28:24,678 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does not have permission to access the requested resource. Unknown user with identity 'anonymous'. Returning Unauthorized response. 2019-08-20 08:28:31,702 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET https://nifi-psh.adeo.com:8443/nifi-api/flow/current-user (source ip: 172.20.0.1) 2019-08-20 08:28:31,710 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com <http://corp.leroymerlin.com> 2019-08-20 08:28:31,718 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com <http://corp.leroymerlin.com>], groups[GO-LM-ROLE-DATA-INGEST-ADMIN] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. I would love to be able to confirm that my authorizations.xml contains authorization for my initial admin, but the file only contains the opaque identifier ... I have no users.xml generated (which seems normal to me, since I get users from LDAP) I still don't understand what's wrong ... And I really appreciate your help. Le 19/08/2019 à 14:42, Pierre Villard a écrit : Hi Nicolas, Can you share the message you get when accessing the UI? The logs from the nifi-user.log file? As well as having a look at the users.xml and authorizations.xml file generated the first time NiFi is starting based on your configuration? Thanks, Pierre Le lun. 19 août 2019 à 11:35, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Hello all I now have a nifi instance able to connect to LDAP server, with valid certificates and so on. But i'
Re: ldap authentication and initial admin identity
Wow, I'm really REALLY puzzled. I'm using Nifi through the docker image, and docker-compose. I was used to do docker-compose up/down, and it failed. But this time, I did a docker-compose down, AND destroyed the folder in which the application is deployed. And this time, it worked ! I'm now logged in as my ldap uid. Thank you very much Pierre ! Le 20/08/2019 à 10:55, Pierre Villard a écrit : Something that I can suggest: the users.xml and authorizations.xml files are generated when NiFi starts for the first time. If you did some modifications (such as the initial admin identity), the files users/authorizations won't be updated with your configuration change... Something you could try: delete authorizations.xml and users.xml files and restart NiFi to be sure it uses the last version of your configuration. Le mar. 20 août 2019 à 10:33, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : When I try to login, UI shows Insufficient Permissions Unable to view the user interface. Contact the system administrator. The log file contains 2019-08-20 08:22:18,808 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Tue Aug 20 08:22:18 UTC 2019 2019-08-20 08:28:24,459 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2019-08-20 08:28:24,521 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. 2019-08-20 08:28:24,678 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does not have permission to access the requested resource. Unknown user with identity 'anonymous'. Returning Unauthorized response. 2019-08-20 08:28:31,702 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET https://nifi-psh.adeo.com:8443/nifi-api/flow/current-user (source ip: 172.20.0.1) 2019-08-20 08:28:31,710 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com <http://corp.leroymerlin.com> 2019-08-20 08:28:31,718 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com <http://corp.leroymerlin.com>], groups[GO-LM-ROLE-DATA-INGEST-ADMIN] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. I would love to be able to confirm that my authorizations.xml contains authorization for my initial admin, but the file only contains the opaque identifier ... I have no users.xml generated (which seems normal to me, since I get users from LDAP) I still don't understand what's wrong ... And I really appreciate your help. Le 19/08/2019 à 14:42, Pierre Villard a écrit : Hi Nicolas, Can you share the message you get when accessing the UI? The logs from the nifi-user.log file? As well as having a look at the users.xml and authorizations.xml file generated the first time NiFi is starting based on your configuration? Thanks, Pierre Le lun. 19 août 2019 à 11:35, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Hello all I now have a nifi instance able to connect to LDAP server, with valid certificates and so on. But i'm unable to connect to Nifi UI, altough I have set myself as initial admin identity. My ldap full DN is set as initial admin identity file-access-policy-provider org.apache.nifi.authorization.FileAccessPolicyProvider ldap-user-group-provider ./conf/authorizations.xml uid=20008203,ou=people,ou=go-lm,o=corp.company.com <http://corp.company.com> And I'm a member of the group which is used to allow access cn=GO-LM-ROLE-DATA-INGEST-ADMIN,ou=DATA-INGEST,ou=applicationRole,ou=role,ou=GO-LM,o=corp.company.com <http://corp.company.com> groupofuniquenames SUBTREE cn uniqueMember But, when i debug the StandardManagedAuthorizer code it seems the User object created from the authentication attempt has a different identifier than the initial admin. Is it possible ? And if so, how to configure Nifi to make sure the user obtained from a login has the same identifier than an existing one ? Thanks
Re: ldap authentication and initial admin identity
When I try to login, UI shows Insufficient Permissions Unable to view the user interface. Contact the system administrator. The log file contains 2019-08-20 08:22:18,808 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Tue Aug 20 08:22:18 UTC 2019 2019-08-20 08:28:24,459 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2019-08-20 08:28:24,521 INFO [NiFi Web Server-20] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. 2019-08-20 08:28:24,678 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does not have permission to access the requested resource. Unknown user with identity 'anonymous'. Returning Unauthorized response. 2019-08-20 08:28:31,702 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET https://nifi-psh.adeo.com:8443/nifi-api/flow/current-user (source ip: 172.20.0.1) 2019-08-20 08:28:31,710 INFO [NiFi Web Server-26] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com 2019-08-20 08:28:31,718 INFO [NiFi Web Server-26] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com], groups[GO-LM-ROLE-DATA-INGEST-ADMIN] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. I would love to be able to confirm that my authorizations.xml contains authorization for my initial admin, but the file only contains the opaque identifier ... I have no users.xml generated (which seems normal to me, since I get users from LDAP) I still don't understand what's wrong ... And I really appreciate your help. Le 19/08/2019 à 14:42, Pierre Villard a écrit : Hi Nicolas, Can you share the message you get when accessing the UI? The logs from the nifi-user.log file? As well as having a look at the users.xml and authorizations.xml file generated the first time NiFi is starting based on your configuration? Thanks, Pierre Le lun. 19 août 2019 à 11:35, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit : Hello all I now have a nifi instance able to connect to LDAP server, with valid certificates and so on. But i'm unable to connect to Nifi UI, altough I have set myself as initial admin identity. My ldap full DN is set as initial admin identity file-access-policy-provider org.apache.nifi.authorization.FileAccessPolicyProvider ldap-user-group-provider ./conf/authorizations.xml uid=20008203,ou=people,ou=go-lm,o=corp.company.com <http://corp.company.com> And I'm a member of the group which is used to allow access cn=GO-LM-ROLE-DATA-INGEST-ADMIN,ou=DATA-INGEST,ou=applicationRole,ou=role,ou=GO-LM,o=corp.company.com <http://corp.company.com> groupofuniquenames SUBTREE cn uniqueMember But, when i debug the StandardManagedAuthorizer code it seems the User object created from the authentication attempt has a different identifier than the initial admin. Is it possible ? And if so, how to configure Nifi to make sure the user obtained from a login has the same identifier than an existing one ? Thanks
ldap authentication and initial admin identity
Hello all I now have a nifi instance able to connect to LDAP server, with valid certificates and so on. But i'm unable to connect to Nifi UI, altough I have set myself as initial admin identity. My ldap full DN is set as initial admin identity file-access-policy-provider org.apache.nifi.authorization.FileAccessPolicyProvider ldap-user-group-provider ./conf/authorizations.xml uid=20008203,ou=people,ou=go-lm,o=corp.company.com And I'm a member of the group which is used to allow access cn=GO-LM-ROLE-DATA-INGEST-ADMIN,ou=DATA-INGEST,ou=applicationRole,ou=role,ou=GO-LM,o=corp.company.com groupofuniquenames SUBTREE cn uniqueMember But, when i debug the StandardManagedAuthorizer code it seems the User object created from the authentication attempt has a different identifier than the initial admin. Is it possible ? And if so, how to configure Nifi to make sure the user obtained from a login has the same identifier than an existing one ? Thanks
Re: My nifi no more serve admin interface
Oh damn
It appeared (after a long search) that my keystore was incorrectly built.
Indeed, it contained the server certificate as a trusted certificate,
where it should had been a key pair (with both private and public keys
in) as is explained in Jetty documentation
(https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
- see part Layout of keystore and truststore). And this happened because
I'm really bad at certificates.
Sorry to have consumed some of your time, you all.
Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :
oh, sorry, I forgot to mention i use the nifi docker image, with
configuration
services:
nifi-runner:
hostname: nifi-psh.adeo.com
image: apache/nifi:1.9.2
ports:
- "38080:8443"
- "5000:8000"
volumes:
-
${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
-
${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
-
${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
And port 8443 is standard http port, I guess (the port 8000 is the
standard debug one)
Le 13/08/2019 à 16:10, Pierre Villard a écrit :
Might be a dumb question but I'm wondering why you're trying with
port 38080? Did you change the configuration to use that specific
port with a secured instance?
Pierre
Le mar. 13 août 2019 à 16:00, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit :
To go a little further, a test with openssl s_client gives the
following
nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ openssl s_client -host localhost -port 38080
CONNECTED(0164)
416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher :
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1565704262
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Which i weird considering nifi outputs in its startup log the lines
nifi-runner_1 | 2019-08-13 13:37:52,315 INFO [main]
o.e.jetty.server.handler.ContextHandler Started
o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
nifi-runner_1 | 2019-08-13 13:37:52,490 INFO [main]
o.e.jetty.util.ssl.SslContextFactory
x509=X509@3d94d7f3(nifi-psh.adeo.com <http://nifi-psh.adeo.com> (adeo
ca),h=[nifi-psh.adeo.com <http://nifi-psh.adeo.com>],w=[]) for
SslContextFactory@da1abd6[provider=null,keyStore=file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks]
nifi-runner_1 | 2019-08-13 13:37:52,510 INFO [main]
o.eclipse.jetty.server.AbstractConnector Started
ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443
<http://0.0.0.0:8443>}
which seems to indicate Jetty is able to listen for https
connections on
port 8443 using certificates described in SslContextFactory. No ?
Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
> I'm currently trying to implement ldap user group authorization
in nifi.
>
> For that, I've deployed nifi docker image with configuration files
> containing required config elements (a ldap identity provider,
a ldap
> user group provider).
>
> I've also configured https with a keystore/truststore that are
injected
> into docker container through volumes.
>
> Once all is configured, i've taken the time to do some debug
session to
> make sure tue FileAccessPolicyProvider correctly loads my user from
> ldap, and it works ok.
>
> Unfortunatly, now, when i try to load Nifi admin interface, I get a
> strange http response containing only the string " � P".
>
> In other words,
>
>
> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
> $ curl -v -H "Host: nifi-psh.adeo.com
<http://nifi-psh.adeo.com>" http://localhost:38080/ --output -
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 38080 (#0)
> > GET / HTTP/
Re: My nifi no more serve admin interface
oh, sorry, I forgot to mention i use the nifi docker image, with
configuration
services:
nifi-runner:
hostname: nifi-psh.adeo.com
image: apache/nifi:1.9.2
ports:
- "38080:8443"
- "5000:8000"
volumes:
-
${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
-
${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
-
${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
And port 8443 is standard http port, I guess (the port 8000 is the
standard debug one)
Le 13/08/2019 à 16:10, Pierre Villard a écrit :
Might be a dumb question but I'm wondering why you're trying with port
38080? Did you change the configuration to use that specific port with
a secured instance?
Pierre
Le mar. 13 août 2019 à 16:00, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit :
To go a little further, a test with openssl s_client gives the
following
nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ openssl s_client -host localhost -port 38080
CONNECTED(0164)
416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher :
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1565704262
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Which i weird considering nifi outputs in its startup log the lines
nifi-runner_1 | 2019-08-13 13:37:52,315 INFO [main]
o.e.jetty.server.handler.ContextHandler Started
o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
nifi-runner_1 | 2019-08-13 13:37:52,490 INFO [main]
o.e.jetty.util.ssl.SslContextFactory
x509=X509@3d94d7f3(nifi-psh.adeo.com <http://nifi-psh.adeo.com> (adeo
ca),h=[nifi-psh.adeo.com <http://nifi-psh.adeo.com>],w=[]) for
SslContextFactory@da1abd6[provider=null,keyStore=file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks]
nifi-runner_1 | 2019-08-13 13:37:52,510 INFO [main]
o.eclipse.jetty.server.AbstractConnector Started
ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443
<http://0.0.0.0:8443>}
which seems to indicate Jetty is able to listen for https
connections on
port 8443 using certificates described in SslContextFactory. No ?
Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
> I'm currently trying to implement ldap user group authorization
in nifi.
>
> For that, I've deployed nifi docker image with configuration files
> containing required config elements (a ldap identity provider, a
ldap
> user group provider).
>
> I've also configured https with a keystore/truststore that are
injected
> into docker container through volumes.
>
> Once all is configured, i've taken the time to do some debug
session to
> make sure tue FileAccessPolicyProvider correctly loads my user from
> ldap, and it works ok.
>
> Unfortunatly, now, when i try to load Nifi admin interface, I get a
> strange http response containing only the string " � P".
>
> In other words,
>
>
> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
> $ curl -v -H "Host: nifi-psh.adeo.com
<http://nifi-psh.adeo.com>" http://localhost:38080/ --output -
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 38080 (#0)
> > GET / HTTP/1.1
> > Host: nifi-psh.adeo.com <http://nifi-psh.adeo.com>
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> §♥♥ ☻☻P* Connection #0 to host localhost left intact
>
>
> http does not work (which i expects, since I've configured
> authentication/authorization
>
> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
> $ curl -v -H "Host: nifi-psh.adeo.com
<http://nifi-psh.adeo.com>" https://localhost:38080/
> --output -
> * Trying ::1...
> * TCP_NO
Re: My nifi no more serve admin interface
To go a little further, a test with openssl s_client gives the following
nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ openssl s_client -host localhost -port 38080
CONNECTED(0164)
416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher :
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1565704262
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Which i weird considering nifi outputs in its startup log the lines
nifi-runner_1 | 2019-08-13 13:37:52,315 INFO [main]
o.e.jetty.server.handler.ContextHandler Started
o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
nifi-runner_1 | 2019-08-13 13:37:52,490 INFO [main]
o.e.jetty.util.ssl.SslContextFactory
x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
ca),h=[nifi-psh.adeo.com],w=[]) for
SslContextFactory@da1abd6[provider=null,keyStore=file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks]
nifi-runner_1 | 2019-08-13 13:37:52,510 INFO [main]
o.eclipse.jetty.server.AbstractConnector Started
ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
which seems to indicate Jetty is able to listen for https connections on
port 8443 using certificates described in SslContextFactory. No ?
Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
I'm currently trying to implement ldap user group authorization in nifi.
For that, I've deployed nifi docker image with configuration files
containing required config elements (a ldap identity provider, a ldap
user group provider).
I've also configured https with a keystore/truststore that are injected
into docker container through volumes.
Once all is configured, i've taken the time to do some debug session to
make sure tue FileAccessPolicyProvider correctly loads my user from
ldap, and it works ok.
Unfortunatly, now, when i try to load Nifi admin interface, I get a
strange http response containing only the string "�P".
In other words,
nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ curl -v -H "Host: nifi-psh.adeo.com" http://localhost:38080/ --output -
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 38080 (#0)
> GET / HTTP/1.1
> Host: nifi-psh.adeo.com
> User-Agent: curl/7.55.1
> Accept: */*
>
§♥♥ ☻☻P* Connection #0 to host localhost left intact
http does not work (which i expects, since I've configured
authentication/authorization
nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ curl -v -H "Host: nifi-psh.adeo.com" https://localhost:38080/
--output -
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 38080 (#0)
* schannel: SSL/TLS connection with localhost port 38080 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 174 bytes...
* schannel: sent initial handshake data: sent 174 bytes
* schannel: SSL/TLS connection with localhost port 38080 (step 2/3)
* schannel: encrypted data got 7
* schannel: encrypted data buffer: offset 7 length 4096
* schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE
(0x80090326) - This error usually occurs when a fatal SSL/TLS alert is
received (e.g. handshake failed). More detail may be available in the
Windows System event log.
* Closing connection 0
* schannel: shutting down SSL/TLS connection with localhost port 38080
* schannel: clear security context handle
curl: (35) schannel: next InitializeSecurityContext failed:
SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a
fatal SSL/TLS alert is received (e.g. handshake failed). More detail may
be available in the Windows System event log.
But neither is https
I guess there is something wrong with certificate, but the log doesn't
seems to indicate any certificate misconfiguration.
What have i done wrong ?
My nifi no more serve admin interface
I'm currently trying to implement ldap user group authorization in nifi. For that, I've deployed nifi docker image with configuration files containing required config elements (a ldap identity provider, a ldap user group provider). I've also configured https with a keystore/truststore that are injected into docker container through volumes. Once all is configured, i've taken the time to do some debug session to make sure tue FileAccessPolicyProvider correctly loads my user from ldap, and it works ok. Unfortunatly, now, when i try to load Nifi admin interface, I get a strange http response containing only the string "�P". In other words, nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux $ curl -v -H "Host: nifi-psh.adeo.com" http://localhost:38080/ --output - * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 38080 (#0) > GET / HTTP/1.1 > Host: nifi-psh.adeo.com > User-Agent: curl/7.55.1 > Accept: */* > §♥♥ ☻☻P* Connection #0 to host localhost left intact http does not work (which i expects, since I've configured authentication/authorization nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux $ curl -v -H "Host: nifi-psh.adeo.com" https://localhost:38080/ --output - * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 38080 (#0) * schannel: SSL/TLS connection with localhost port 38080 (step 1/3) * schannel: checking server certificate revocation * schannel: sending initial handshake data: sending 174 bytes... * schannel: sent initial handshake data: sent 174 bytes * schannel: SSL/TLS connection with localhost port 38080 (step 2/3) * schannel: encrypted data got 7 * schannel: encrypted data buffer: offset 7 length 4096 * schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log. * Closing connection 0 * schannel: shutting down SSL/TLS connection with localhost port 38080 * schannel: clear security context handle curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log. But neither is https I guess there is something wrong with certificate, but the log doesn't seems to indicate any certificate misconfiguration. What have i done wrong ?
Re: Continuing my LDAP auth adventures
Oh god nifi-runner_1 | Caused by: org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occured during LDAP processing; nested exception is javax.naming.NamingException: LDAP response read timed out, timeout used:1ms. nifi-runner_1 | at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:228) nifi-runner_1 | at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:397) nifi-runner_1 | at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:328) nifi-runner_1 | at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:629) nifi-runner_1 | at org.apache.nifi.ldap.tenants.LdapUserGroupProvider.load(LdapUserGroupProvider.java:493) nifi-runner_1 | at org.apache.nifi.ldap.tenants.LdapUserGroupProvider.onConfigured(LdapUserGroupProvider.java:387) nifi-runner_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) nifi-runner_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) nifi-runner_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) nifi-runner_1 | at java.lang.reflect.Method.invoke(Method.java:498) nifi-runner_1 | at org.apache.nifi.authorization.UserGroupProviderInvocationHandler.invoke(UserGroupProviderInvocationHandler.java:38) nifi-runner_1 | at com.sun.proxy.$Proxy75.onConfigured(Unknown Source) nifi-runner_1 | at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:139) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) nifi-runner_1 | ... 101 common frames omitted nifi-runner_1 | Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:1ms. nifi-runner_1 | at com.sun.jndi.ldap.Connection.readReply(Connection.java:507) nifi-runner_1 | at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:638) nifi-runner_1 | at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:606) nifi-runner_1 | at com.sun.jndi.ldap.LdapCtx.getSearchReply(LdapCtx.java:1918) nifi-runner_1 | at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(AbstractLdapNamingEnumeration.java:130) nifi-runner_1 | at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(AbstractLdapNamingEnumeration.java:217) nifi-runner_1 | at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(AbstractLdapNamingEnumeration.java:189) nifi-runner_1 | at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:365) nifi-runner_1 | ... 113 common frames omitted Seems like I'm trying to get a little too much users from LDAP :-) I guess it's time to use group search Le 19/07/2019 à 16:24, Bryan Bende a écrit : The FileAccessPolicyProvider is making a call to the user group provider using the value you entered for initial admin: final User initialAdmin = userGroupProvider.getUserByIdentity(initialAdminIdentity); It has something to do with the value you entered for the initial admin not lining up with the identities being returned from the LDAP provider. If you entered a full DN, but the LDAP provider returns just the short name, or vice versa, then it doesn't line up. On Fri, Jul 19, 2019 at 9:59 AM Nicolas Delsaux wrote: And indeed, it changed the error nifi-runner_1 | Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin a_dn to seed policies nifi-runner_1 |at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) nifi-runner_1 |at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) nifi-runner_1 |at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640) nifi-runner_1 |at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) nifi-runner_1 |at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) nifi-runner_1 |at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) nifi-runner_1 |... 96 common frames omitted nifi-runner_1 | Caused
Re: Continuing my LDAP auth adventures
Here is the full version (with obvious replacements for manager dn, manager password, ldap server url, and other "sensitive" informations ldap-user-group-provider org.apache.nifi.ldap.tenants.LdapUserGroupProvider LDAPS a_dn a_password /opt/certs/cacerts.jks changeit JKS TLSv1 FOLLOW 10 secs 10 secs ldaps://myserver.mycompany.com:636 30 mins ou=people,o=mycompany.com privPerson SUBTREE uid This attribute doesn't exist to make sure no grouping is done group ONE_LEVEL file-access-policy-provider org.apache.nifi.authorization.FileAccessPolicyProvider ldap-user-group-provider ./conf/authorizations.xml managed-authorizer org.apache.nifi.authorization.StandardManagedAuthorizer file-access-policy-provider Le 19/07/2019 à 12:03, Pierre Villard a écrit : Hi Nicolas, Could you share the full content of your authorizers.xml file? Sometimes it's just a matter of references not being in the right "order". Le ven. 19 juil. 2019 à 11:59, Edward Armes mailto:edward.ar...@gmail.com>> a écrit : I wasn't able to find any single good way, I don't know if switching the logs down to debug or trace might give you a bit more info though . In the end I just went through a worked it out by hand using a combination of manual checking against an alternative tool (i.e. an LDAP browser), file format checkers, or just commenting things out by hand. I did sometimes find that white space character (new line etc...) can occasionally cause a problem with the Spring loading. Edward On Fri, Jul 19, 2019 at 10:45 AM Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> wrote: Is there any way to get a better error ? Le 19/07/2019 à 11:36, Edward Armes a écrit : Hi Nicolas, This one is a bit of a Spring special. The actual cause here is that the Spring Bean that is being created from this file has silently failed, and thus the auto-wiring has failed as well. The result is you get this lovely misleading error. The normal reason for the bean not being created I found was because I made a typo in the configuration file(s). Edward On Fri, Jul 19, 2019 at 10:21 AM Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> wrote: Hi all Now I know how to connect to my LDAP directory, i now have a strange error nifi-runner_1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. [... let me just skip the uninteresting Spring stack ...] nifi-runner_1 | Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactor
Re: Continuing my LDAP auth adventures
Is there any way to get a better error ? Le 19/07/2019 à 11:36, Edward Armes a écrit : Hi Nicolas, This one is a bit of a Spring special. The actual cause here is that the Spring Bean that is being created from this file has silently failed, and thus the auto-wiring has failed as well. The result is you get this lovely misleading error. The normal reason for the bean not being created I found was because I made a typo in the configuration file(s). Edward On Fri, Jul 19, 2019 at 10:21 AM Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> wrote: Hi all Now I know how to connect to my LDAP directory, i now have a strange error nifi-runner_1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. [... let me just skip the uninteresting Spring stack ...] nifi-runner_1 | Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) nifi-runner_1 | at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) nifi-runner_1 | ... 96 common frames omitted nifi-runner_1 | Caused by: java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. nifi-runner_1 | at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:175) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) From what I understand, it seems like the AuthorizerFactoryBean tries to read my user-group-provider from the authorizers.xml file. I have such an user group provider, which is a ldap one : ldap-user-group-provider org.apache.nifi.ldap.tenants.LdapUserGroupProvider LDAPS a_dn a_password /opt/certs/cacerts.jks another JKS TLSv1 FOLLOW 10 secs 10 secs ldaps://myserver.mycompany.com:636 30 mins ou=people,o=mycompany.com <http://mycompany.com> privPerson SUBTREE uid This attribute doesn't exist to make sure no grouping is done group ONE_LEVEL So why can't it be loaded ? Because I don't see any other exception (typically, I would expect a search fail exception, but it seems to work).
Continuing my LDAP auth adventures
Hi all Now I know how to connect to my LDAP directory, i now have a strange error nifi-runner_1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. [... let me just skip the uninteresting Spring stack ...] nifi-runner_1 | Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) nifi-runner_1 | at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) nifi-runner_1 | ... 96 common frames omitted nifi-runner_1 | Caused by: java.lang.Exception: The specified authorizer 'ldap-user-group-provider' could not be found. nifi-runner_1 | at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:175) nifi-runner_1 | at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) From what I understand, it seems like the AuthorizerFactoryBean tries to read my user-group-provider from the authorizers.xml file. I have such an user group provider, which is a ldap one : ldap-user-group-provider org.apache.nifi.ldap.tenants.LdapUserGroupProvider LDAPS a_dn a_password /opt/certs/cacerts.jks another JKS TLSv1 FOLLOW 10 secs 10 secs ldaps://myserver.mycompany.com:636 30 mins ou=people,o=mycompany.com privPerson SUBTREE uid This attribute doesn't exist to make sure no grouping is done group ONE_LEVEL So why can't it be loaded ? Because I don't see any other exception (typically, I would expect a search fail exception, but it seems to work).
Re: ldap auth : error code 12 - Unavailable Critical Extension
Yes Pierre, I have made sure the organization was correct using another
LDAP browser.
Let me make sure by replaying the involved part of code.
From that stack trace, the deeeper nifi code invocation is
nifi-runner_1 | Caused by:
org.springframework.ldap.OperationNotSupportedException: [LDAP: error
code 12 - Unavailable Critical Extension]; nested exception is
javax.naming.OperationNotSupportedException: [LDAP: error code 12 -
Unavailable Critical Extension]; remaining name 'o=corp.mycompany.com'
nifi-runner_1 | at
org.apache.nifi.ldap.tenants.LdapUserGroupProvider.load(LdapUserGroupProvider.java:493)
nifi-runner_1 | at
org.apache.nifi.ldap.tenants.LdapUserGroupProvider.onConfigured(LdapUserGroupProvider.java:387)
which seems to load user from ldap.
More precisely, the error line seems to be
userList.addAll(ldapTemplate.search(userSearchBase, userFilter.encode(),
userControls, new AbstractContextMapper() {
where
- userSearchBase is "o=corp.mycompany.com"
- userFilter is
(&(objectclass=privPerson)(&(objectclass=privPerson)(uid={0}))) yup, a
redundant condition, so I've changed the search filter
So, after having talked with the LDAP team in mycompany, we finally
discovered the LDAP directory didn't support the paging mechanism
implemented in Nifi. I removed the paging attribute, and it worked !
Le 18/07/2019 à 15:54, Pierre Villard a écrit :
Hi Nicolas,
It looks like a LDAP issue: LDAP: error code 12 - Unavailable Critical
Extension.
Are you sure about the LDAP tree structure you have? is the
organization correct 'o=corp.mycompany.com <http://corp.mycompany.com/>'?
Thanks,
Pierre
Le jeu. 18 juil. 2019 à 15:36, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit :
Hello,
I'm trying to use LDAP authentication and am having a weird exception
nifi-runner_1 | 2019-07-18 13:26:03,076 INFO [main]
org.eclipse.jetty.server.Server Started @22069ms
nifi-runner_1 | 2019-07-18 13:26:03,080 WARN [main]
org.apache.nifi.web.server.JettyServer Failed to start web
server... shutting down.
nifi-runner_1 |
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name
'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
Unsatisfied dependency expressed through method
'setFilterChainProxySecurityConfigurer' parameter 1; nested
exception is
org.springframework.beans.factory.BeanExpressionException:
Expression parsing failed; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name
'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
dependency expressed through method 'setJwtAuthenticationProvider'
parameter 0; nested exception is
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'jwtAuthenticationProvider' defined in
class path resource [nifi-web-security-context.xml]: Cannot
resolve reference to bean 'authorizer' while setting constructor
argument; nested exception is
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'authorizer': FactoryBean threw exception
on object creation; nested exception is
org.springframework.ldap.OperationNotSupportedException: [LDAP:
error code 12 - Unavailable Critical Extension]; nested exception
is javax.naming.OperationNotSupportedException: [LDAP: error code
12 - Unavailable Critical Extension]; remaining name
'o=corp.mycompany.com <http://corp.mycompany.com>'
nifi-runner_1 | at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:666)
nifi-runner_1 | at
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
nifi-runner_1 | at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
nifi-runner_1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1269)
nifi-runner_1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
nifi-runner_1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
nifi-runner_1 | at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
nifi-runner_1 | at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBean
ldap auth : error code 12 - Unavailable Critical Extension
Hello, I'm trying to use LDAP authentication and am having a weird exception nifi-runner_1 | 2019-07-18 13:26:03,076 INFO [main] org.eclipse.jetty.server.Server Started @22069ms nifi-runner_1 | 2019-07-18 13:26:03,080 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. nifi-runner_1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.springframework.ldap.OperationNotSupportedException: [LDAP: error code 12 - Unavailable Critical Extension]; nested exception is javax.naming.OperationNotSupportedException: [LDAP: error code 12 - Unavailable Critical Extension]; remaining name 'o=corp.mycompany.com' nifi-runner_1 | at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:666) nifi-runner_1 | at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87) nifi-runner_1 | at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1269) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) nifi-runner_1 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) nifi-runner_1 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) nifi-runner_1 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) nifi-runner_1 | at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) nifi-runner_1 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) nifi-runner_1 | at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443) nifi-runner_1 | at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325) nifi-runner_1 | at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) nifi-runner_1 | at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953) nifi-runner_1 | at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558) nifi-runner_1 | at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918) nifi-runner_1 | at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370) nifi-runner_1 | at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) nifi-runner_1 | at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) nifi-runner_1 | at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848) nifi-runner_1 | at
Nifi and SSL offloading
Hi I'm trying to deploy Nifi in Kubernetes with authentcation. In Kubernetes, it is possible (and recommended in my organization) to have SSL managed by cluster at edge route level. Which means request seen by Nifi are http ones. According to nifi documentation, it seems to imply no authentication is possible in this case. However, in our context, the X-Forwarded-Proto header is set (see https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Common_non-standard_request_fields), which could be used to enable authentication in HTTP. So is it possible to do that ? And if so, how ?
Re: Unable to send JSON to BigQuery
Well, if you take a look at my schema, the error is subtle, but obvious
(once I've added the tests and modified the code).
I've set "Consent" to be of typ "record", not "RECORD". Yes, it was a
case issue.
So I've modified code in BigQueryUtils to use uppercased type in all
cases, AND an exception which is thrown if string corresponds to no type.
Finally, I've set a default value of NULLABLE for mode.
All these changes fix the bug described in
https://issues.apache.org/jira/browse/NIFI-6422
I'm also trying to create the pull request
Le 03/07/2019 à 19:51, Denes Arvay a écrit :
Yes, and please attach the test cases too.
Does this mean that your original issue hasn't been resolved yet by
adding the "mode" fields?
On Wed, Jul 3, 2019, 19:27 Nicolas Delsaux <mailto:nicolas.dels...@gmx.fr>> wrote:
So I have a simple test that replicate the bug. Do I have to open
the issue in Apache JIRA (I already have access to) ?
Le 03/07/2019 à 11:28, Denes Arvay a écrit :
Hi Nicolas,
It seems that NiFi expects to have the "mode" field being
present, even though based on the BigQuery doc [1] it's optional.
I'd suggest trying adding it to every name-type pair with its
default value "NULLABLE". (i.e. { "name": "Consent", "type":
"record", *"mode": "NULLABLE"*, "fields": [ { "name": "id",
"type": "STRING", *"mode": "NULLABLE"* }, ...)
Let me know if it solved the issue. If yes, I'll file a Jira
ticket to fix it.
Best,
Denes
[1]
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
On Wed, Jul 3, 2019 at 11:07 AM Nicolas Delsaux
mailto:nicolas.dels...@gmx.fr>> wrote:
I'm using Apache Nifi 1.9.2 and trying to post JSON content
to a
BigQuery table.
There seems to be something wrong, sicne I get
2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.gcp.bigquery.PutBigQueryBatch
PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at
org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Where can it come from ? And how can i fix it ?
From the stack, I'm understanding there is something wrong
with my
BigQuery schema (which is however recognized as valid by
BigQuery).
My schema is
[
{
"name": "Consent",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "identity",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
Re: Unable to send JSON to BigQuery
So I have a simple test that replicate the bug. Do I have to open the
issue in Apache JIRA (I already have access to) ?
Le 03/07/2019 à 11:28, Denes Arvay a écrit :
Hi Nicolas,
It seems that NiFi expects to have the "mode" field being present,
even though based on the BigQuery doc [1] it's optional.
I'd suggest trying adding it to every name-type pair with its default
value "NULLABLE". (i.e. { "name": "Consent", "type": "record",
*"mode": "NULLABLE"*, "fields": [ { "name": "id", "type": "STRING",
*"mode": "NULLABLE"* }, ...)
Let me know if it solved the issue. If yes, I'll file a Jira ticket to
fix it.
Best,
Denes
[1]
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
On Wed, Jul 3, 2019 at 11:07 AM Nicolas Delsaux
mailto:nicolas.dels...@gmx.fr>> wrote:
I'm using Apache Nifi 1.9.2 and trying to post JSON content to a
BigQuery table.
There seems to be something wrong, sicne I get
2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.gcp.bigquery.PutBigQueryBatch
PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at
org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Where can it come from ? And how can i fix it ?
From the stack, I'm understanding there is something wrong with my
BigQuery schema (which is however recognized as valid by BigQuery).
My schema is
[
{
"name": "Consent",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "identity",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "businessUnit",
"type": "STRING"
}
]
},
{
"name": "finality",
"type": "STRING"
},
{
"name": "source",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "origin",
"type": "STRING"
},
{
"name": "collaborator",
"type": "record",
"fields": [
{
Docker nifi doesn't support OpenID Connect ?
Hi, I've read on Docker hub that nifi docker container doesn't support OpenID Connect. But if I mount the nifi.properties file using a volume, is it possible to have openID Connect working ? or is it replaced by the Docker start.sh script (which invoke secure.sh only for LDAP or two-way SSL) ?
Re: Unable to send JSON to BigQuery
I'm ivnestigating the same way.
I've added the mode field everywhere, but still have the issue.
I'll try to create a minimal reproducing schema for your ticket (by
running unit tests)
Le 03/07/2019 à 11:28, Denes Arvay a écrit :
Hi Nicolas,
It seems that NiFi expects to have the "mode" field being present,
even though based on the BigQuery doc [1] it's optional.
I'd suggest trying adding it to every name-type pair with its default
value "NULLABLE". (i.e. { "name": "Consent", "type": "record",
*"mode": "NULLABLE"*, "fields": [ { "name": "id", "type": "STRING",
*"mode": "NULLABLE"* }, ...)
Let me know if it solved the issue. If yes, I'll file a Jira ticket to
fix it.
Best,
Denes
[1]
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
On Wed, Jul 3, 2019 at 11:07 AM Nicolas Delsaux
mailto:nicolas.dels...@gmx.fr>> wrote:
I'm using Apache Nifi 1.9.2 and trying to post JSON content to a
BigQuery table.
There seems to be something wrong, sicne I get
2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.gcp.bigquery.PutBigQueryBatch
PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at
org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Where can it come from ? And how can i fix it ?
From the stack, I'm understanding there is something wrong with my
BigQuery schema (which is however recognized as valid by BigQuery).
My schema is
[
{
"name": "Consent",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "identity",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "businessUnit",
"type": "STRING"
}
]
},
{
"name": "finality",
"type": "STRING"
},
{
"name": "source",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "origin",
"type": "STRING"
},
{
"name": "collaborator",
"type": "record",
Unable to send JSON to BigQuery
I'm using Apache Nifi 1.9.2 and trying to post JSON content to a
BigQuery table.
There seems to be something wrong, sicne I get
2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.gcp.bigquery.PutBigQueryBatch
PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at
org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Where can it come from ? And how can i fix it ?
From the stack, I'm understanding there is something wrong with my
BigQuery schema (which is however recognized as valid by BigQuery).
My schema is
[
{
"name": "Consent",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "identity",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "businessUnit",
"type": "STRING"
}
]
},
{
"name": "finality",
"type": "STRING"
},
{
"name": "source",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "origin",
"type": "STRING"
},
{
"name": "collaborator",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
}
]
}
]
},
{
"name": "consentDate",
"type": "TIMESTAMP"
},
{
"name": "expiryDate",
"type": "TIMESTAMP"
},
{
"name": "expired",
"type": "BOOLEAN"
},
{
"name": "createdBy",
"type": "STRING"
},
{
"name": "createdDate",
"type": "TIMESTAMP"
}
]
}
]
What can cause the trouble ?
Thanks
