Re: Custom NAR Class Loader Issue

[Otto Fowler]

I may have something to test that ;)  I’ll let you know, else my pr will
get the ‘why are you updating the sdk version’ comment.



On March 16, 2018 at 15:57:21, Bryan Bende (bbe...@gmail.com) wrote:

This should be better going forward once the 1.6.0 release is out.

https://issues.apache.org/jira/browse/NIFI-4936

On Fri, Mar 16, 2018 at 3:49 PM, Otto Fowler 
wrote:
> This is also a problem for new AWS processors, say if you use a newer aws
> java sdk core in a dependency but also have to depend on the
nifi-aws-nar.
>
>
> On March 2, 2018 at 13:40:21, Bryan Bende (bbe...@gmail.com) wrote:
>
> Doug,
>
> I think the only solution is what you proposed about fixing the
> nifi-gcp-bundle...
>
> Basically, if a NAR needs a different version of a dependency that is
> already declared in the root pom's dependencyManagement, then the
> bundle's pom needs it own dependencyManagement to force it back to the
> specific version for the bundle.
>
> Ultimately it would be nice to re-evaluate our use of
> dependencyManagement in the root pom, because I don't think we should
> be forcing the version of so many things. Guava is a perfect example
> where we shouldn't be forcing all NARs to use a specific version of
> Guava, that defeats the whole purpose of NARs.
>
> There are other cases though where it does make sense because there is
> code in nifi/nifi-commons that we would likely want to keep in sync
> with the NARs that use the code.
>
> -Bryan
>
> On Fri, Mar 2, 2018 at 1:27 PM, Douglas Willcocks
>  wrote:
>> Hi all,
>>
>> I'm encountering a JAR version dependency issue when developing a custom
>> Processor that uses the GCPCredentialsService in Nifi 1.5.0.
>>
>> 1) The nifi-gcp-bundle distributed with Nifi 1.5.0 (which contains
>> GCPCredentialsService) is packaged with dependencies on:
>>
>> - com.google.cloud:google-cloud:0.8.0
>> - com.google.auth:google-auth-library-oauth2-http:0.6.0
>>
>> Which both transitively depend on com.google.guava:guava:19.0, except
the
>> root-level pom.xml for Nifi (here
>> ) explicitly pins
the
>> guava version to 18.0.
>>
>> 2) The custom Processor I am building also depends on
>> com.google.cloud:google-cloud:0.8.0, but actually uses methods from this
>> library that depend on features added in com.google.guava:guava:19.0.
>>
>> 3) In order to make the GCPCredentialsService available inside my
>> Processor
>> NAR, I have added the nifi-gcp-services-api-nar as a parent.
>>
>> 4) The implementation of NarClassLoader always starts the search for
>> classes in the parent, which means that when my processor looks for
>> com.google.guava class definitions, the com.google.guava:guava:18.0 JAR
>> packaged with the nifi-gcp-services-api-nar is always found first,
>> resulting in a java.lang.NoSuchMethodError exception in my Processor's
>> execution.
>>
>> I'm not sure what the best solution is – I guess I could create my own
>> version of the nifi-gcp-bundle that has the correct pom.xml file to
force
>> the use of guava:19.0, but that seems a bit overkill for a change that
is
>> essentially 5 lines of XML in a POM file.
>>
>> Any suggestions or thoughts?
>>
>> Thanks!
>> Douglas
>> --
>>
>> Douglas Willcocks
>> VP France, Data Science & Engineering
>>
>> 19 Rue Richer, 75009 Paris, France
>> M: +33 6 80 37 60 72
>> E: douglas.willco...@artefact.com
>>
>> W: http://www.artefact.com/
>>
>> --
>> This email and any attachments contains privileged and confidential
>> information intended only for the use of the addressee(s). If you are
not
>> an intended recipient of this email, you are hereby notified that any
>> dissemination, copying or use of information is strictly prohibited. If
>> you
>> received this email in error or without authorization, please delete the
>> email from your system and notify us immediately by sending us an email.
>> If
>> you need any further assistance, please send a message to
>> he...@artefact.com
>> .

Re: Custom NAR Class Loader Issue

[Bryan Bende]

This should be better going forward once the 1.6.0 release is out.

https://issues.apache.org/jira/browse/NIFI-4936

On Fri, Mar 16, 2018 at 3:49 PM, Otto Fowler  wrote:
> This is also a problem for new AWS processors, say if you use a newer aws
> java sdk core in a dependency but also have to depend on the nifi-aws-nar.
>
>
> On March 2, 2018 at 13:40:21, Bryan Bende (bbe...@gmail.com) wrote:
>
> Doug,
>
> I think the only solution is what you proposed about fixing the
> nifi-gcp-bundle...
>
> Basically, if a NAR needs a different version of a dependency that is
> already declared in the root pom's dependencyManagement, then the
> bundle's pom needs it own dependencyManagement to force it back to the
> specific version for the bundle.
>
> Ultimately it would be nice to re-evaluate our use of
> dependencyManagement in the root pom, because I don't think we should
> be forcing the version of so many things. Guava is a perfect example
> where we shouldn't be forcing all NARs to use a specific version of
> Guava, that defeats the whole purpose of NARs.
>
> There are other cases though where it does make sense because there is
> code in nifi/nifi-commons that we would likely want to keep in sync
> with the NARs that use the code.
>
> -Bryan
>
> On Fri, Mar 2, 2018 at 1:27 PM, Douglas Willcocks
>  wrote:
>> Hi all,
>>
>> I'm encountering a JAR version dependency issue when developing a custom
>> Processor that uses the GCPCredentialsService in Nifi 1.5.0.
>>
>> 1) The nifi-gcp-bundle distributed with Nifi 1.5.0 (which contains
>> GCPCredentialsService) is packaged with dependencies on:
>>
>> - com.google.cloud:google-cloud:0.8.0
>> - com.google.auth:google-auth-library-oauth2-http:0.6.0
>>
>> Which both transitively depend on com.google.guava:guava:19.0, except the
>> root-level pom.xml for Nifi (here
>> ) explicitly pins the
>> guava version to 18.0.
>>
>> 2) The custom Processor I am building also depends on
>> com.google.cloud:google-cloud:0.8.0, but actually uses methods from this
>> library that depend on features added in com.google.guava:guava:19.0.
>>
>> 3) In order to make the GCPCredentialsService available inside my
>> Processor
>> NAR, I have added the nifi-gcp-services-api-nar as a parent.
>>
>> 4) The implementation of NarClassLoader always starts the search for
>> classes in the parent, which means that when my processor looks for
>> com.google.guava class definitions, the com.google.guava:guava:18.0 JAR
>> packaged with the nifi-gcp-services-api-nar is always found first,
>> resulting in a java.lang.NoSuchMethodError exception in my Processor's
>> execution.
>>
>> I'm not sure what the best solution is – I guess I could create my own
>> version of the nifi-gcp-bundle that has the correct pom.xml file to force
>> the use of guava:19.0, but that seems a bit overkill for a change that is
>> essentially 5 lines of XML in a POM file.
>>
>> Any suggestions or thoughts?
>>
>> Thanks!
>> Douglas
>> --
>>
>> Douglas Willcocks
>> VP France, Data Science & Engineering
>>
>> 19 Rue Richer, 75009 Paris, France
>> M: +33 6 80 37 60 72
>> E: douglas.willco...@artefact.com
>>
>> W: http://www.artefact.com/
>>
>> --
>> This email and any attachments contains privileged and confidential
>> information intended only for the use of the addressee(s). If you are not
>> an intended recipient of this email, you are hereby notified that any
>> dissemination, copying or use of information is strictly prohibited. If
>> you
>> received this email in error or without authorization, please delete the
>> email from your system and notify us immediately by sending us an email.
>> If
>> you need any further assistance, please send a message to
>> he...@artefact.com
>> .

Re: Custom NAR Class Loader Issue

[Otto Fowler]

This is also a problem for new AWS processors, say if you use a newer aws
java sdk core in a dependency but also have to depend on the nifi-aws-nar.


On March 2, 2018 at 13:40:21, Bryan Bende (bbe...@gmail.com) wrote:

Doug,

I think the only solution is what you proposed about fixing the
nifi-gcp-bundle...

Basically, if a NAR needs a different version of a dependency that is
already declared in the root pom's dependencyManagement, then the
bundle's pom needs it own dependencyManagement to force it back to the
specific version for the bundle.

Ultimately it would be nice to re-evaluate our use of
dependencyManagement in the root pom, because I don't think we should
be forcing the version of so many things. Guava is a perfect example
where we shouldn't be forcing all NARs to use a specific version of
Guava, that defeats the whole purpose of NARs.

There are other cases though where it does make sense because there is
code in nifi/nifi-commons that we would likely want to keep in sync
with the NARs that use the code.

-Bryan

On Fri, Mar 2, 2018 at 1:27 PM, Douglas Willcocks
 wrote:
> Hi all,
>
> I'm encountering a JAR version dependency issue when developing a custom
> Processor that uses the GCPCredentialsService in Nifi 1.5.0.
>
> 1) The nifi-gcp-bundle distributed with Nifi 1.5.0 (which contains
> GCPCredentialsService) is packaged with dependencies on:
>
> - com.google.cloud:google-cloud:0.8.0
> - com.google.auth:google-auth-library-oauth2-http:0.6.0
>
> Which both transitively depend on com.google.guava:guava:19.0, except the
> root-level pom.xml for Nifi (here
> ) explicitly pins the
> guava version to 18.0.
>
> 2) The custom Processor I am building also depends on
> com.google.cloud:google-cloud:0.8.0, but actually uses methods from this
> library that depend on features added in com.google.guava:guava:19.0.
>
> 3) In order to make the GCPCredentialsService available inside my
Processor
> NAR, I have added the nifi-gcp-services-api-nar as a parent.
>
> 4) The implementation of NarClassLoader always starts the search for
> classes in the parent, which means that when my processor looks for
> com.google.guava class definitions, the com.google.guava:guava:18.0 JAR
> packaged with the nifi-gcp-services-api-nar is always found first,
> resulting in a java.lang.NoSuchMethodError exception in my Processor's
> execution.
>
> I'm not sure what the best solution is – I guess I could create my own
> version of the nifi-gcp-bundle that has the correct pom.xml file to force
> the use of guava:19.0, but that seems a bit overkill for a change that is
> essentially 5 lines of XML in a POM file.
>
> Any suggestions or thoughts?
>
> Thanks!
> Douglas
> --
>
> Douglas Willcocks
> VP France, Data Science & Engineering
>
> 19 Rue Richer, 75009 Paris, France
> M: +33 6 80 37 60 72
> E: douglas.willco...@artefact.com
>
> W: http://www.artefact.com/
>
> --
> This email and any attachments contains privileged and confidential
> information intended only for the use of the addressee(s). If you are not
> an intended recipient of this email, you are hereby notified that any
> dissemination, copying or use of information is strictly prohibited. If
you
> received this email in error or without authorization, please delete the
> email from your system and notify us immediately by sending us an email.
If
> you need any further assistance, please send a message to
he...@artefact.com
> .

Re: Will you accept contributions in Scala?

[Otto Fowler]

Sorry for the late response, but the point about the Extension Repo making
this a moot point is an important one.
The idea that you need to get your processor into the main codebase to have
it be known and useful is something
that is easier to overcome than a language war.


On February 13, 2018 at 12:15:23, Russell Bateman (r...@windofkeltia.com)
wrote:

I completely second Mike's sentiment here as well as cautionary
statements by other contributors over this thread's history (Matt
Burgess' post in particular). Besides fueling flame wars and religious
inquisition, you cannot open the door to Scala without opening it to
other (JVM languages at least) and there's no umbrella that will shield
you from the chaos doing so will unleash rendering NiFi difficult to
develop to (and maybe even to deploy). As it is presently, all
JVM-language developers are able to consume NiFi easily for development
and to deal with their own deployment issues.

Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:
> Milan,
>
> I don't think you can do that without creating a lot of fuel for a flame
> war. I have personally never met a Scala developer who was incapable of
> writing decent Java. The same is true of Groovy. I think anyone who finds
> the requirement to use Java over their language of choice to be a deal
> breaker on contributing is probably someone unlikely to be more help than
> trouble anyway.
>
> Mike
>
> On Tue, Feb 13, 2018 at 11:35 AM, Milan Das  wrote:
>
>> I think we should not add blindly any language but should be open to add
>> couple of language like Scala.
>> In Bigdata world Scala/Python/Java are widely accepted.
>>
>> Thanks,
>> Milan Das
>> Interset
>>
>> On 2/13/18, 10:20 AM, "Weiss, Adam"  wrote:
>>
>> I think it makes the most sense to me for us to publish a separate
>> repo with a module and nar build for now and post when it's available in
>> the users group.
>>
>> Thanks for the discussion everyone, hopefully we can start making some
>> helpful contributions soon.
>>
>> -Adam
>>
>>
>> On 2018/02/10 23:43:31, Tony Kurc > gmail.com>> wrote:
>> > It is like Matt read my mind.>
>> >
>> > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess > > wrote:>
>> >
>> > > I'm fine with a vote, but I'll be voting to keep Java as the
>> single>
>> > > language for the (non-test) code. I share the same concerns as
>> many of>
>> > > the other folks as far as accepting other languages, it's mainly
>> the>
>> > > "slippery slope" argument that I don't want to turn into a>
>> > > JVM-language flame war. If Scala, why not Groovy? Certainly the>
>> > > syntax is closer to Java, and the community has accepted it as a
>> valid>
>> > > language for writing unit tests, although we stopped short for>
>> > > allowing it for the deployable NiFi codebase, for the same reasons>
>> > > IIRC. If Scala and/or Groovy, why not Kotlin? The same argument>
>> > > (albeit more tenuous) goes for Clojure and just about every other
>> JVM>
>> > > language (although I don't expect a call for LuaJ processors lol).>
>> > >>
>> > > Whether we decide to support various languages ad-hoc or not, I
>> would>
>> > > strenuously object to multiple/hybrid build systems for the
>> deployed>
>> > > artifacts. If I could switch NiFi completely to Gradle I would,
>> but I>
>> > > realize there are good reasons for not doing so (yet?) in the
>> Apache>
>> > > NiFi community, and I would never want any hybrid Maven/Gradle
>> build>
>> > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>> > > custom Mojo for Maven NAR builds, and the complexity for hybrid
>> builds>
>> > > in general, I think this would create a maintenance nightmare.>
>> > >>
>> > > The language thing is a tough decision though, it's not awesome
>> that>
>> > > specifying a single language can be a barrier to a more diverse>
>> > > community, certainly Scala-based bundles would be more than
>> welcome in>
>> > > the overall NiFi ecosystem, I just think the cons outweigh the
>> pros>
>> > > for the baseline code. I've written Groovy processors/NARs using>
>> > > Gradle as the build system, and I'm good with keeping them in my
>> own>
>> > > repo, especially when the Extension Registry becomes a thing. I
>> can>
>> > > see the Extension Registry perhaps making this a moot point, but>
>> > > clearly we need to have this discussion in the meantime.>
>> > >>
>> > > Regards,>
>> > > Matt>
>> > >>
>> > >>
>> > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande > > wrote:>
>> > > > Wasn't there a warning trigger about the NiFi distro size from
>> Apache>
>> > > > recently? IMO, before talking alternative languages, solve the
>> modularity>
>> > > > and NAR distribution problem. I think the implementation of a
>> module>
>> > > won't>
>> > > > matter much then, the point being not everything has to go in
>> the core,>
>> > > > 

Re: AWS Processors and Proxy User/Password

[Otto Fowler]

Can you please explain how you do that?  I have now seen
https://github.com/apache/nifi/pull/2016 which adds pretty much
what I think is required.


On March 16, 2018 at 15:05:10, Charlie Meyer (char...@charliemeyer.net)
wrote:

For these, I have used the AWS Credentials Provider controller service

-Charlie
ᐧ


Charlie Meyer

On Fri, Mar 16, 2018 at 1:25 PM, Otto Fowler 
wrote:

> Does anyone know why the AWS Processors support proxy host and port but
not
> user and password as InvokeHttp does?
>
> ottO
>

Re: AWS Processors and Proxy User/Password

[Charlie Meyer]

For these, I have used the AWS Credentials Provider controller service

-Charlie
ᐧ


Charlie Meyer

On Fri, Mar 16, 2018 at 1:25 PM, Otto Fowler 
wrote:

> Does anyone know why the AWS Processors support proxy host and port but not
> user and password as InvokeHttp does?
>
> ottO
>

AWS Processors and Proxy User/Password

[Otto Fowler]

Does anyone know why the AWS Processors support proxy host and port but not
user and password as InvokeHttp does?

ottO

Re: Finding Performance bottleneck issue

[Mike Thomsen]

That seems like a very reasonable use case. You said:

> I see that my processor is actually queuing up  records at source.

Are you saying that the processor isn't able to process them that quickly
such that you're seeing a big backlog in the input queue?

On Fri, Mar 16, 2018 at 11:56 AM, Milan Das  wrote:

> Hi Mike,
> My processor is processing windows Text event as below and creating a JSON
> out of it.
> Also I am applying simple JoltTransformer (Simple just Shift and Default)
> to convert to different  JSON (no hierarchy) .
>
> Output have the following:
> 1. Original text
> 2. Converted JSON
> 3. JOLT transformed JSON
> 4. Failure
>
>
> Steps in program:
> 1. Converting the event to Java Map (using regex: "([^:=]*)[:=]([^:=]*)")
> 2. Map to Json (using Gson)
> 3. Jolt transfeormation
>
>
>
> Example event:
>
> Examples of 4626
> User / Device claims information.
>
> Subject:
> Security ID: %1
> Account Name:%2
> Account Domain:  %3
> Logon ID:%4
>
> Logon Type:  %9
>
> New Logon:
> Security ID: %5
> Account Name:%6
> Account Domain:  %7
> Logon ID:%8
>
> Event in sequence:   %10 of %11
>
> User Claims: %12
>
> Device Claims:   %13
>
> The subject fields indicate the account on the local system which
> requested the logon. This is most commonly a service such as the Server
> service, or a local process such as Winlogon.exe or Services.exe.
>
> The logon type field indicates the kind of logon that occurred. The most
> common types are 2 (interactive) and 3 (network).
>
> The New Logon fields indicate the account for whom the new logon was
> created, i.e. the account that was logged on.
>
> This event is generated when the Audit User/Device claims subcategory is
> configured and the user’s logon token contains user/device claims
> information. The Logon ID field can be used to correlate this event with
> the corresponding user logon event as well as to any other security audit
> events generated during this logon session.
>
>
>
> Regards,
> Milan Das
>
>
> On 3/16/18, 10:56 AM, "Mike Thomsen"  wrote:
>
> Milan,
>
> Can you share some details about where you are running into problems?
> Like
> a basic description of what it's trying to do?
>
> On Fri, Mar 16, 2018 at 10:39 AM, Milan Das  wrote:
>
> > I have a custom processor, it works as expected. But I feel there is
> some
> > performance measure need to be done. I see that my processor is
> actually
> > queuing up  records at source.
> >
> > Is there a run a load  test and do performance measure for Custom
> > Processor?
> >
> >
> >
> > Regards,
> >
> > Milan Das
> >
> >
>
>
>
>

Re: Finding Performance bottleneck issue

[Milan Das]

Hi Mike,
My processor is processing windows Text event as below and creating a JSON out 
of it.
Also I am applying simple JoltTransformer (Simple just Shift and Default) to 
convert to different  JSON (no hierarchy) .

Output have the following:
1. Original text
2. Converted JSON
3. JOLT transformed JSON
4. Failure


Steps in program:
1. Converting the event to Java Map (using regex: "([^:=]*)[:=]([^:=]*)")
2. Map to Json (using Gson)
3. Jolt transfeormation



Example event:

Examples of 4626
User / Device claims information.

Subject:
Security ID: %1
Account Name:%2
Account Domain:  %3
Logon ID:%4

Logon Type:  %9

New Logon:
Security ID: %5
Account Name:%6
Account Domain:  %7
Logon ID:%8

Event in sequence:   %10 of %11

User Claims: %12

Device Claims:   %13

The subject fields indicate the account on the local system which requested the 
logon. This is most commonly a service such as the Server service, or a local 
process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common 
types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, 
i.e. the account that was logged on.

This event is generated when the Audit User/Device claims subcategory is 
configured and the user’s logon token contains user/device claims information. 
The Logon ID field can be used to correlate this event with the corresponding 
user logon event as well as to any other security audit events generated during 
this logon session.



Regards,
Milan Das


On 3/16/18, 10:56 AM, "Mike Thomsen"  wrote:

Milan,

Can you share some details about where you are running into problems? Like
a basic description of what it's trying to do?

On Fri, Mar 16, 2018 at 10:39 AM, Milan Das  wrote:

> I have a custom processor, it works as expected. But I feel there is some
> performance measure need to be done. I see that my processor is actually
> queuing up  records at source.
>
> Is there a run a load  test and do performance measure for Custom
> Processor?
>
>
>
> Regards,
>
> Milan Das
>
>

Re: Finding Performance bottleneck issue

[Mike Thomsen]

Milan,

Can you share some details about where you are running into problems? Like
a basic description of what it's trying to do?

On Fri, Mar 16, 2018 at 10:39 AM, Milan Das  wrote:

> I have a custom processor, it works as expected. But I feel there is some
> performance measure need to be done. I see that my processor is actually
> queuing up  records at source.
>
> Is there a run a load  test and do performance measure for Custom
> Processor?
>
>
>
> Regards,
>
> Milan Das
>
>

Re: Finding Performance bottleneck issue

[Joe Witt]

Milan,

We dont offer any load testing specific tooling.  You can though write
integration tests which setup a more complete system.  The most common
model i see people do though for true performance/bottleneck analysis
though is setup a running nifi instance with a flow through your
custom processor and attach a profiler at runtime.

This is easy to do by editing conf/bootstrap.conf

#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

change that line to

#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

Then with your running nifi you can attach something like JVisualVM
from the JDK which is a really great tool.  You could look at more
powerful options like Yourkit as well.

These let you really learn where the bottlenecks are.

You might also want to look at the existing library of 200+ processors
code and see if any have similar characteristics to yours.

Design things:
1) Does how your processor works lend to idempotent (safe to do over
and over) operation?  If yes you can enable session batching.  This
means instead of one flowfile at a time we can do hundreds or
thousands and merge them into a single flowfile repo transaction
2) Are you using a lot of heap/memory?  This can be if you're reading
content fully into memory or worse doing array copies, etc..

If you need more help and pointers please share more details on what
the processor is doing/the design of it as there are many folks on
this list which can give general pointers.

Thanks

On Fri, Mar 16, 2018 at 2:39 PM, Milan Das  wrote:
> I have a custom processor, it works as expected. But I feel there is some 
> performance measure need to be done. I see that my processor is actually 
> queuing up  records at source.
>
> Is there a run a load  test and do performance measure for Custom Processor?
>
>
>
> Regards,
>
> Milan Das
>

Finding Performance bottleneck issue

[Milan Das]

I have a custom processor, it works as expected. But I feel there is some 
performance measure need to be done. I see that my processor is actually 
queuing up  records at source.

Is there a run a load  test and do performance measure for Custom Processor?

 

Regards,

Milan Das

Re: PutMongo 1.3.0 - Upsert issue

[Sivaprasanna]

It need be _id. In case of it being generated automatically, it will be
tough to keep track. It can be any valid MongoDB filter query, if I’m
right. So the query should be:
{
,
“$set” : {
“FieldA” : “New FieldA value”
}
}


On Fri, 16 Mar 2018 at 5:07 PM, Mike Thomsen  wrote:

> I noticed that your update document didn't have an update key in it. The
> default lookup key is _id so you'd need something like this:
> {
> "_id": "ID_HERE",
> "$set": {
> "new_field": 1
> }
> }
>
> On Wed, Jan 10, 2018 at 6:05 AM, fabe_bdx  wrote:
>
> > Dear Pierre,
> >
> > First all many thnaks for your help, we were able to use the copy you
> > provided us and restart Nifi with 1.5.0 module.
> >
> > But it's appear that don't realy work like expected.
> >
> > If we try to update some attributes with the $set commant it don't work.
> >
> > *Original MongoDb record :*
> >
> > {
> > "_id" : ObjectId("5a535bcea33116f026ef65f0"),
> > "type" : "Cc",
> > "nmess" : 3172,
> > "idgtr" : "C-0--00061--0-CMF-RMF",
> > "zone" : 1.0,
> > "idzone" : "NULL",
> > "cmf" : 24,
> > "rmf" : 24,
> > "carrefour" : 61,
> > "idcarrefour" : 2,
> > "last_update" : "2018-01-09T13:49:58",
> > "libelle" : " Henrique#Cebolas",
> > "geojson" :
> > "{\"features\":[{\"geometry\":{\"coordinates\":[\"-9.
> > 132289080874671\",\"38.708011\"],\"type\":\"
> > Point\"},\"type\":\"Feature\"}],\"type\":\"FeatureCollection\"}",
> > "positionOk" : true,
> > "orientation" : 0
> > }
> >
> > *Update request throw putmongo processor :*
> >
> > {
> > "$set": {
> > "type":"Cc",
> > "nmess":14488,
> > "idgtr":"C-0--00088--0-CMF-RMF",
> > "last_update":"2018-01-10T09:32:35",
> > "libelle":"Bonifacio_Estefan. "
> > }
> > }
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
>

Re: PutMongo 1.3.0 - Upsert issue

[Mike Thomsen]

I noticed that your update document didn't have an update key in it. The
default lookup key is _id so you'd need something like this:
{
"_id": "ID_HERE",
"$set": {
"new_field": 1
}
}

On Wed, Jan 10, 2018 at 6:05 AM, fabe_bdx  wrote:

> Dear Pierre,
>
> First all many thnaks for your help, we were able to use the copy you
> provided us and restart Nifi with 1.5.0 module.
>
> But it's appear that don't realy work like expected.
>
> If we try to update some attributes with the $set commant it don't work.
>
> *Original MongoDb record :*
>
> {
> "_id" : ObjectId("5a535bcea33116f026ef65f0"),
> "type" : "Cc",
> "nmess" : 3172,
> "idgtr" : "C-0--00061--0-CMF-RMF",
> "zone" : 1.0,
> "idzone" : "NULL",
> "cmf" : 24,
> "rmf" : 24,
> "carrefour" : 61,
> "idcarrefour" : 2,
> "last_update" : "2018-01-09T13:49:58",
> "libelle" : " Henrique#Cebolas",
> "geojson" :
> "{\"features\":[{\"geometry\":{\"coordinates\":[\"-9.
> 132289080874671\",\"38.708011\"],\"type\":\"
> Point\"},\"type\":\"Feature\"}],\"type\":\"FeatureCollection\"}",
> "positionOk" : true,
> "orientation" : 0
> }
>
> *Update request throw putmongo processor :*
>
> {
> "$set": {
> "type":"Cc",
> "nmess":14488,
> "idgtr":"C-0--00088--0-CMF-RMF",
> "last_update":"2018-01-10T09:32:35",
> "libelle":"Bonifacio_Estefan. "
> }
> }
>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>