Re: Query syntax error in GetMongo processor?
Hi James,
The query needs to be a valid JSON. You can have a look into the test class
to get some examples:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
In your case, I assume the query parameter should only be:
{'Call':'Marko'}
Regarding the authentication issue, not sure I can help.
Hope this helps,
Pierre.
2017-07-30 18:31 GMT+02:00 James McMahon :
> Hello. I cannot get a simple test query to work against MongoDB from
> GetMongo. Here is what I attempt to do:
>
> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
> Mongo Database Name DataServiceAudit
> Mongo Collection Name Audit
> SSL Context Service No value set
> Client Auth NONE
> Query db.find({'Call':'Marko'})
>
> I leave the collection out (ie, db.Audit.find...) because I have expressly
> stated the collection name in the processor.
> I am trying to get this working in its simplest form before I tackle
> themore complex, and so am not using SSL at this time.
>
> I am using NiFi 1.3.0 in this case.
>
> In the UI the processor indicates this problem:
> 'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
> org.bson.json.JSONParseException
>
> Why does it throw that error?
>
> If I remove the Query entirely, I can get the processor to enter a run
> state. However then in the nifi-app.log it says there is an authentication
> issue:
>
> 2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository
> Successfully checkpointed FlowFile Repository with 2313 records in 122
> milliseconds
> 2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
> o.a.nifi.processors.mongodb.GetMongo
> GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11]
> Failed to execute query null due to com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]: com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]
> com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
> for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}.
> Client view of cluster state is {type=UNKNOWN, servers=[{address=
> 34.227.51.144:27017, type=UNKNOWN, state=CONNECTING,
> exception={com.mongodb.MongoSecurityException: Exception authenticating
> MongoCredential{mechanism=null, userName='nifi', source='admin',
> password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException:
> Command failed with error 18: 'Authentication failed.' on server
> 34.227.51.144:27017. The full response is { "ok" : 0.0, "errmsg" :
> "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed"
> }}}]
> at com.mongodb.connection.BaseCluster.createTimeoutException(
> BaseCluster.java:369)
> at com.mongodb.connection.BaseCluster.selectServer(
> BaseCluster.java:101)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:75)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:71)
> at com.mongodb.binding.ClusterBinding.getReadConnectionSource(
> ClusterBinding.java:63)
> at com.mongodb.operation.OperationHelper.withConnection(
> OperationHelper.java:210)
> at com.mongodb.operation.FindOperation.execute(
> FindOperation.java:482)
> at com.mongodb.operation.FindOperation.execute(
> FindOperation.java:79)
> at com.mongodb.Mongo.execute(Mongo.java:772)
> at
Query syntax error in GetMongo processor?
Hello. I cannot get a simple test query to work against MongoDB from
GetMongo. Here is what I attempt to do:
Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
Mongo Database Name DataServiceAudit
Mongo Collection Name Audit
SSL Context Service No value set
Client Auth NONE
Query db.find({'Call':'Marko'})
I leave the collection out (ie, db.Audit.find...) because I have expressly
stated the collection name in the processor.
I am trying to get this working in its simplest form before I tackle
themore complex, and so am not using SSL at this time.
I am using NiFi 1.3.0 in this case.
In the UI the processor indicates this problem:
'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
org.bson.json.JSONParseException
Why does it throw that error?
If I remove the Query entirely, I can get the processor to enter a run
state. However then in the nifi-app.log it says there is an authentication
issue:
2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 2313 records in 122 milliseconds
2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
o.a.nifi.processors.mongodb.GetMongo
GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11] Failed to execute query
null due to com.mongodb.MongoTimeoutException: Timed out after 3 ms
while waiting for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]:
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]
at
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at
com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:210)
at
com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
at
com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
at
org.apache.nifi.processors.mongodb.GetMongo.onTrigger(GetMongo.java:158)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
NiFi Docker
Hi, We're using Nifi in a containerized environment (using docker-compose for local development and test, and docker swarm in production). The current docker image doesn't have a lot of environment options, meaning if you want to run Nifi with some custom config like the one below : environment: - NIFI_WEB_HTTP_HOST=xxx - NIFI_CLUSTER_HOST_NAME=xxx - NIFI_CLUSTER_HOST_PORT=xxx - NIFI_CLUSTER_PORT=xxx - NIFI_ZOOKEEPER_CONNECT_STRING=xxx - NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_USAGE_PERCENTAGE=xxx - CONFIG_URI_LABEL=xxx You need to create your own "wrapper" docker image (extending the base apache nifi one). Stuff that we typically to in a nifi installation is - change values in nifi.properties (a generic solution could be created for that where nifi property keys could be provided as environment variables to the docker container) - install custom processors (nar files) - custom log configuration (logback.xml) - custom bootstrap values - copy flow templates in nifi so they can be used immediately after startup. Have you ever thought of extending the current docker image, to allow a little bit more customization like this ones above. Some other things I noticed : - the readme on https://hub.docker.com/r/apache/nifi/ doesn't mention docker usage it all. seems to be the default nifi readme. - The current docker image contains 2 layers of 1gb each. I'm guessing this is the result of a) downloading and untarring the nifi distribution and b) executing the chown command. Is there a reason these are spread out over 2 docker run commands ? f5f88f68e088: Downloading [=> ] 25.9MB/978MB 5ed4b763cde4: Downloading [=> ] 34.02MB/978MB -- View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/NiFi-Docker-tp2562.html Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
