[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/280#issuecomment-211726456
  
@jskora do you think this PR can be closed now given the updates made to 
fix the underlying defects found?  A new PR could be submitted which adds the 
proposed features or goes into ReplaceText or a new processor.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/354#issuecomment-211726075
  
I am less concerned about thread safety of the transforms themselves now.  
Jolt's docs do refer to this at the readme level 
https://github.com/bazaarvoice/jolt/tree/1271e1919693e63ce1efdf534dfee0e0579d0b2f#-performance.

So that leaves the update to NOTICE(s) concern :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Joe Witt]

It would be an unnecessary party foul to kill off the old tags right?

On Tue, Apr 19, 2016 at 12:21 AM, Sean Busbey  wrote:
> No magic that I know of. Each project that I've seen do this just
> pushed new tags. Some make the tag point to the prior release tag,
> some have it point to the commit that the prior release tag points to.
> Those who want to be fancy use git cli to make the date match.
>
> On Mon, Apr 18, 2016 at 10:58 PM, Joe Witt  wrote:
>> Is there some good git magic to follow to do this Sean?  I could push
>> new signed tags?
>>
>> On Mon, Apr 18, 2016 at 11:54 PM, Sean Busbey  wrote:
>>> The "rel/" prefix is part of the implementation of an ASF policy that 
>>> requires a
>>> protected immutable tag for releases. It will be present on all
>>> releases going forward and we should probably add them for our prior
>>> releases.
>>>
>>>
>>> On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri  wrote:
 The "rel" prefix is a bit of a deviation from what we have been doing.  Up
 to this point, it had been x.y.z-RC# up until a final release occurred
 resulting in just an x.y.z tag.

 On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:

> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>
> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:
>
> > Yep!  Thanks for catching that. Had made but not pushed.  Should be 
> > there
> > now.
> >
> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
> release?
> > >
> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
> > >
> > >> Hello
> > >>
> > >> The Apache NiFi team would like to announce the release of Apache 
> > >> NiFi
> > >> 0.6.1.
> > >>
> > >> Apache NiFi is an easy to use, powerful, and reliable system to
> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
> > >> supports highly configurable directed graphs of data routing,
> > >> transformation, and system mediation logic.
> > >>
> > >> More details on Apache NiFi can be found here:
> > >>   http://nifi.apache.org/
> > >>
> > >> The release artifacts can be downloaded from here:
> > >>   http://nifi.apache.org/download.html
> > >>
> > >> Maven artifacts have been made available here:
> > >>
> > >>
> >
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
> > >>
> > >> Release note highlights can be found here:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
> > >>
> > >> Thank you
> > >> The Apache NiFi team
> > >>
> >
>

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/354#issuecomment-211724253
  
Oleg off-list pointed out the finer concerns with the member variable.  
While it seems pretty far out it also seems like it is accurate.  In reading 
https://docs.oracle.com/javase/tutorial/essential/concurrency/atomic.html and 
https://docs.oracle.com/javase/tutorial/essential/concurrency/memconsist.html

And indeed I cannot think of anything the framework is doing with thread 
management or how those lifecycle calls work that would enforce/guarantee a 
happens-before relationship for these processor member variables.  @markap14 
can you? 

That said the licensing and thread safety issue of the transform itself are 
still needing to be addressed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Sean Busbey]

No magic that I know of. Each project that I've seen do this just
pushed new tags. Some make the tag point to the prior release tag,
some have it point to the commit that the prior release tag points to.
Those who want to be fancy use git cli to make the date match.

On Mon, Apr 18, 2016 at 10:58 PM, Joe Witt  wrote:
> Is there some good git magic to follow to do this Sean?  I could push
> new signed tags?
>
> On Mon, Apr 18, 2016 at 11:54 PM, Sean Busbey  wrote:
>> The "rel/" prefix is part of the implementation of an ASF policy that 
>> requires a
>> protected immutable tag for releases. It will be present on all
>> releases going forward and we should probably add them for our prior
>> releases.
>>
>>
>> On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri  wrote:
>>> The "rel" prefix is a bit of a deviation from what we have been doing.  Up
>>> to this point, it had been x.y.z-RC# up until a final release occurred
>>> resulting in just an x.y.z tag.
>>>
>>> On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:
>>>
 Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?

 On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:

 > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
 > now.
 >
 > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
 > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
 release?
 > >
 > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
 > >
 > >> Hello
 > >>
 > >> The Apache NiFi team would like to announce the release of Apache NiFi
 > >> 0.6.1.
 > >>
 > >> Apache NiFi is an easy to use, powerful, and reliable system to
 > >> process and distribute data.  Apache NiFi was made for dataflow.  It
 > >> supports highly configurable directed graphs of data routing,
 > >> transformation, and system mediation logic.
 > >>
 > >> More details on Apache NiFi can be found here:
 > >>   http://nifi.apache.org/
 > >>
 > >> The release artifacts can be downloaded from here:
 > >>   http://nifi.apache.org/download.html
 > >>
 > >> Maven artifacts have been made available here:
 > >>
 > >>
 >
 https://repository.apache.org/content/repositories/releases/org/apache/nifi/
 > >>
 > >> Release note highlights can be found here:
 > >>
 > >>
 >
 https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
 > >>
 > >> Thank you
 > >> The Apache NiFi team
 > >>
 >

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Joe Witt]

Is there some good git magic to follow to do this Sean?  I could push
new signed tags?

On Mon, Apr 18, 2016 at 11:54 PM, Sean Busbey  wrote:
> The "rel/" prefix is part of the implementation of an ASF policy that 
> requires a
> protected immutable tag for releases. It will be present on all
> releases going forward and we should probably add them for our prior
> releases.
>
>
> On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri  wrote:
>> The "rel" prefix is a bit of a deviation from what we have been doing.  Up
>> to this point, it had been x.y.z-RC# up until a final release occurred
>> resulting in just an x.y.z tag.
>>
>> On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:
>>
>>> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>>>
>>> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:
>>>
>>> > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
>>> > now.
>>> >
>>> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
>>> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
>>> release?
>>> > >
>>> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
>>> > >
>>> > >> Hello
>>> > >>
>>> > >> The Apache NiFi team would like to announce the release of Apache NiFi
>>> > >> 0.6.1.
>>> > >>
>>> > >> Apache NiFi is an easy to use, powerful, and reliable system to
>>> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
>>> > >> supports highly configurable directed graphs of data routing,
>>> > >> transformation, and system mediation logic.
>>> > >>
>>> > >> More details on Apache NiFi can be found here:
>>> > >>   http://nifi.apache.org/
>>> > >>
>>> > >> The release artifacts can be downloaded from here:
>>> > >>   http://nifi.apache.org/download.html
>>> > >>
>>> > >> Maven artifacts have been made available here:
>>> > >>
>>> > >>
>>> >
>>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>>> > >>
>>> > >> Release note highlights can be found here:
>>> > >>
>>> > >>
>>> >
>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>>> > >>
>>> > >> Thank you
>>> > >> The Apache NiFi team
>>> > >>
>>> >
>>>

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/354#issuecomment-211718256
  
@joewitt As I am thinking more about it, you have a point for @OnStopped, 
so I am retracting my comment. As for volatile, what's at question is the 
visibility of the _transformer_ by another thread. Basically if T1 executed 
@OnSchedule where it is set to a value, T2 may come in and it will still be 
null, not because of some race condition, but simply because the value set by 
T1 may not be visible by T2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Sean Busbey]

The "rel/" prefix is part of the implementation of an ASF policy that requires a
protected immutable tag for releases. It will be present on all
releases going forward and we should probably add them for our prior
releases.


On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri  wrote:
> The "rel" prefix is a bit of a deviation from what we have been doing.  Up
> to this point, it had been x.y.z-RC# up until a final release occurred
> resulting in just an x.y.z tag.
>
> On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:
>
>> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>>
>> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:
>>
>> > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
>> > now.
>> >
>> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
>> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
>> release?
>> > >
>> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
>> > >
>> > >> Hello
>> > >>
>> > >> The Apache NiFi team would like to announce the release of Apache NiFi
>> > >> 0.6.1.
>> > >>
>> > >> Apache NiFi is an easy to use, powerful, and reliable system to
>> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
>> > >> supports highly configurable directed graphs of data routing,
>> > >> transformation, and system mediation logic.
>> > >>
>> > >> More details on Apache NiFi can be found here:
>> > >>   http://nifi.apache.org/
>> > >>
>> > >> The release artifacts can be downloaded from here:
>> > >>   http://nifi.apache.org/download.html
>> > >>
>> > >> Maven artifacts have been made available here:
>> > >>
>> > >>
>> >
>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>> > >>
>> > >> Release note highlights can be found here:
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>> > >>
>> > >> Thank you
>> > >> The Apache NiFi team
>> > >>
>> >
>>

[GitHub] nifi pull request: NIFI-1614 File Identity Provider implementation

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/267#issuecomment-211716593
  
@jvwing @alopresto Have you had a chance to re-engage on this?  It seems 
like a reasonable easy-path option for folks just wanting to use some 
simple/local username and password based setup.  My responses to the questions 
James posed:

What is required to make this viable?
- This discussion appears on track

Is there a better medium than bcrypt that combines widespread tool support 
with decent encryption.
- Sounds like you and Andy both see it as a good option.

Are we open to including a command-line user admin tool?
- In my opinion we should be consistent that administrative actions occur 
by editing files on the command line in the less optimal case and interacting 
through a designed/intentional UX in the best case.  We should strive to move 
away from config file based options and move fully towards service/REST API 
driven approaches.  These will serve us better in clustered/cloud type 
environments as well.

Are we open to including a sample credentials file? Where would you 
recommend it go?
- Absolutely.  In conf directory like the others of its type.  I think an 
argument could be made to have this username/password driven mode be the 
default.

Are we open to documenting this identity provider on the front-page of the 
Admin Guide alongside X.509 and LDAP? Where else should I do so?
- We must do so.  We should fully embrace this as an option and document 
what it is good for and not good for.  Our current default of having no 
authentication at all is what we should be working to eliminate.  I think this 
offers us a good first step to do that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Slack for 'apachenifi'?

[Joe Witt]

Team,

In attempting to test/review NIFI-1578 I went to slack to create a
team for 'apachenifi'.  I found though that it wasn't available.
Anyone here already make that and if so can you transfer it to the
Apache NiFi PMC?

Thanks
Joe

[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/252#issuecomment-211712568
  
@jskora I recall there was some discussion where you were considering 
supporting this as part of something a bit broader.  Is that right or is this 
one you think good to go as is? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-840: Create ListS3 processor

[jskora]

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/238#issuecomment-211711893
  
I can, but it might not be until later this week.

On Mon, Apr 18, 2016 at 11:26 PM, Joe Witt  wrote:

> Anyone in a good position to test/validate this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> 
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1540 - AWS Kinesis Get and Put Processors

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/239#issuecomment-211711720
  
Hello @mans2singh .  Do you think you'll have a chance to incorporate these 
review feedback items?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-840: Create ListS3 processor

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/238#issuecomment-211711025
  
Anyone in a good position to test/validate this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/354#issuecomment-211708711
  
@YolandaMDavis @olegz some feedback.

1. The com.bazaar.jolt... dependency does not appear to be accounted for in 
the LICENSE/NOTICE updates.  Their LICENSE does indicate they would like 
reference made to 'Copyright 2013-2014 Bazaarvoice, Inc.'  This is found in 
https://github.com/bazaarvoice/jolt/blob/1271e1919693e63ce1efdf534dfee0e0579d0b2f/LICENSE
  So we should propogate that into the NOTICE file as well.  Also we need to 
reference JOLT in our top level nifi-assembly/NOTICE as well if it is new and 
I'd assume at least for Bazaar it would be.

2. I don't believe I follow the recommendation to change the member 
variable 'transform' to volatile or to set it to null on stopped.  The variable 
appears to be set at the proper lifecycle point in NiFi and thus would not be 
reassigned in some non-thread safe way.  The only question then is could its 
use itself be non Thread safe and that does appear questionable to me.  In that 
case marking the variable as volatile won't help at all anyway.  If indeed it 
is not thread safe you will want to consider marking the class as executing 
serially or you could have a pool of transforms or you could transform batches 
at a time.  Lots of ways to tackle it.  Of course if that is thread safe 
already then you are fine and in any event the volatile should be unnecessary.  
Finally, I am not aware of any cause for setting it to null during onStopped or 
on unScheduled as if the processor was being removed from the flow then it 
would get cleaned up anyway.  If there was a need to call a close meth
 od or something then yes that is good.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60167981
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
--- End diff --

Looking at the tests I am now wondering if 

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60167411
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
--- End diff --

Something @alopresto pointed out on another unrelated PR, yet a valid point 
and we need to start making it as a rule and that is using _name_ and 
_displayName_. In fact I'll quote Andy's comment:
".name should be a machine-safe string (i.e. ssl-context-service) which 
will remain constant over the life of the processor because it is 

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/354#issuecomment-211703487
  
@YolandaMDavis this looks very good. One general comment; Just to finish 
the life-cycle loop I'd suggest adding a stop operation (operation annotated 
with @OnStopped) where based on what I see you would simply set _transformer_ 
to null, unless there is some stop/cleanup procedure required (e.g., close 
something etc.). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1337: Add Riemann Reporting Task

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/188#issuecomment-211702845
  
@rickysaltzer Looks like there might be a lack of time to focus on this at 
this time.  How do you feel about closing this PR down for now and revisiting 
later?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NiFi-924: Camel integration

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/219#issuecomment-211701587
  
Commented on NIFI-924 to reference 
https://issues.apache.org/jira/browse/NIFI-1571 and consider this as resolved 
by that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[YolandaMDavis]

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60164832
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+

[GitHub] nifi pull request: NIFI-1197 Added SSL support for MongoDB process...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/360#discussion_r60159711
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -52,6 +62,32 @@
 .required(true)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+.name("Client Auth")
--- End diff --

Yes, the documentation does not sufficiently encourage this, and changing 
existing `PropertyDescriptors` will break backward-compatibility, but moving 
forward, we should explicitly conform to this standard. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1778 Adding NiFiBolt to write back to NiFi...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/361#discussion_r60155833
  
--- Diff: 
nifi-external/nifi-storm-spout/src/main/java/org/apache/nifi/storm/NiFiBolt.java
 ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.storm;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.utils.TupleUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.remote.Transaction;
+import org.apache.nifi.remote.TransferDirection;
+import org.apache.nifi.remote.client.SiteToSiteClient;
+import org.apache.nifi.remote.client.SiteToSiteClientConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+/**
+ * A Storm bolt that can send tuples back to NiFi. This bolt provides a 
micro-batching approach for higher
+ * through put scenarios. The bolt will queue tuples until the number of 
tuples reaches the provided batch size, or
+ * until the provided batch interval in seconds has been exceeded. Setting 
the batch size to 1 will send each tuple
+ * immediately in a single transaction.
+ */
+public class NiFiBolt extends BaseRichBolt {
+
+private static final long serialVersionUID = 3067274587595578836L;
+public static final Logger LOGGER = 
LoggerFactory.getLogger(NiFiBolt.class);
+
+private final SiteToSiteClientConfig clientConfig;
+private final NiFiDataPacketBuilder builder;
+private final int tickFrequencySeconds;
+
+private SiteToSiteClient client;
+private OutputCollector collector;
+private BlockingQueue queue = new LinkedBlockingQueue<>();
+
+private int batchSize = 10;
+private int batchIntervalInSec = 10;
+private long lastBatchProcessTimeSeconds = 0;
+
+public NiFiBolt(final SiteToSiteClientConfig clientConfig, final 
NiFiDataPacketBuilder builder, final int tickFrequencySeconds) {
+this.clientConfig = clientConfig;
+this.builder = builder;
+this.tickFrequencySeconds = tickFrequencySeconds;
+Validate.notNull(this.clientConfig);
+Validate.notNull(this.builder);
+Validate.isTrue(this.tickFrequencySeconds > 0);
+}
+
+public NiFiBolt withBatchSize(int batchSize) {
+this.batchSize = batchSize;
+Validate.isTrue(this.batchSize > 0);
+return this;
+}
+
+public NiFiBolt withBatchInterval(int batchIntervalInSec) {
+this.batchIntervalInSec = batchIntervalInSec;
+Validate.isTrue(this.batchIntervalInSec > 0);
--- End diff --

Should the validation be moved before assigning it to 
_this.batchIntervalInSec_? Same for the other two operations above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1778 Adding NiFiBolt to write back to NiFi...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/361#discussion_r60155646
  
--- Diff: 
nifi-external/nifi-storm-spout/src/main/java/org/apache/nifi/storm/NiFiBolt.java
 ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.storm;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.utils.TupleUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.remote.Transaction;
+import org.apache.nifi.remote.TransferDirection;
+import org.apache.nifi.remote.client.SiteToSiteClient;
+import org.apache.nifi.remote.client.SiteToSiteClientConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+/**
+ * A Storm bolt that can send tuples back to NiFi. This bolt provides a 
micro-batching approach for higher
+ * through put scenarios. The bolt will queue tuples until the number of 
tuples reaches the provided batch size, or
+ * until the provided batch interval in seconds has been exceeded. Setting 
the batch size to 1 will send each tuple
+ * immediately in a single transaction.
+ */
+public class NiFiBolt extends BaseRichBolt {
+
+private static final long serialVersionUID = 3067274587595578836L;
+public static final Logger LOGGER = 
LoggerFactory.getLogger(NiFiBolt.class);
+
+private final SiteToSiteClientConfig clientConfig;
+private final NiFiDataPacketBuilder builder;
+private final int tickFrequencySeconds;
+
+private SiteToSiteClient client;
+private OutputCollector collector;
+private BlockingQueue queue = new LinkedBlockingQueue<>();
+
+private int batchSize = 10;
+private int batchIntervalInSec = 10;
+private long lastBatchProcessTimeSeconds = 0;
--- End diff --

Any chance these attributes will be modified/accessed by multiple threads? 
Just wondering if we should make them volatile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1197 Added SSL support for MongoDB process...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/360#discussion_r60144583
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -52,6 +62,32 @@
 .required(true)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+.name("Client Auth")
--- End diff --

I agree and actually started using it myself on new components, so even 
though its not a common pattern in NiFi, I am certainly +1 into making it one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1197 Added SSL support for MongoDB process...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/360#discussion_r60144254
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -52,6 +62,32 @@
 .required(true)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+public static final PropertyDescriptor CLIENT_AUTH = new 
PropertyDescriptor.Builder()
+.name("Client Auth")
--- End diff --

`.name` should be a machine-safe string (i.e. `ssl-client-auth`) which will 
remain constant over the life of the processor because it is used for object 
resolution when loading from the `flow.tar.gz` file. For the human-readable 
value to display in the UI, please use `.displayName`. This can be changed 
without affecting object resolution (for future renaming, internationalization, 
etc.). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1197 Added SSL support for MongoDB process...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/360#discussion_r60144137
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -52,6 +62,32 @@
 .required(true)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("SSL Context Service")
--- End diff --

`.name` should be a machine-safe string (i.e. `ssl-context-service`) which 
will remain constant over the life of the processor because it is used for 
object resolution when loading from the `flow.tar.gz` file. For the 
human-readable value to display in the UI, please use `.displayName`. This can 
be changed without affecting object resolution (for future renaming, 
internationalization, etc.). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request: MINIFI-18 Moving initialization of confi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/11


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1778 Adding NiFiBolt to write back to NiFi...

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/361

NIFI-1778 Adding NiFiBolt to write back to NiFi from Storm

- Adding example topology that creates a full loop between NiFi and Storm.
- Bumping Storm to 0.10.0

There is an example topology in src/test/resources that can be used with a 
NiFi flow that has output port "Data for Storm" and input port "Data from 
Storm", and of course enables site-to-site in nifi.properties.

An example template is here:
https://gist.github.com/bbende/279824e65d07f63e0002727159b5d78b

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-1778

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #361


commit 9b46a22f617d8e058d6b97654bf8c2d84a2b4415
Author: Bryan Bende 
Date:   2016-04-18T18:00:40Z

NIFI-1778 Adding NiFiBolt to write back to NiFi from Storm
- Adding example topology that creates a full loop between NiFi and Storm.
- Bumping Storm to 0.10.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request: MINIFI-18 Moving initialization of confi...

[apiri]

GitHub user apiri opened a pull request:

https://github.com/apache/nifi-minifi/pull/11

MINIFI-18 Moving initialization of configuration notifier/listener from 
constructor

MINIFI-18 Moving initialization of configuration notifier/listener from 
constructor such that it is only performed when starting an instance.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apiri/nifi-minifi MINIFI-18

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/11.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11


commit 2cd26322e5c580a03be6b812c51b05352bcd2ffd
Author: Aldrin Piri 
Date:   2016-04-18T20:57:37Z

MINIFI-18 Moving initialization of configuration notifier/listener from
constructor such that it is only performed when starting  an instance.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60118214
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
   

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60118046
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
   

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60117078
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
   

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[YolandaMDavis]

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60115891
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+

[GitHub] nifi-minifi pull request: MINIFI-14 Incorporating Listener/Notifie...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/10


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1678

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/323#discussion_r60107339
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/CuratorHeartbeatMonitor.java
 ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.heartbeat;
+
+import java.io.ByteArrayInputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.Unmarshaller;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.RetryForever;
+import org.apache.nifi.cluster.coordination.ClusterCoordinator;
+import org.apache.nifi.cluster.coordination.node.DisconnectionCode;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionState;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionStatus;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.cluster.protocol.message.HeartbeatMessage;
+import org.apache.nifi.controller.cluster.ZooKeeperClientConfig;
+import org.apache.nifi.engine.FlowEngine;
+import org.apache.nifi.reporting.Severity;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.util.StopWatch;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+import org.apache.zookeeper.data.Stat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Uses Apache Curator to monitor heartbeats from nodes
+ */
+public class CuratorHeartbeatMonitor implements HeartbeatMonitor {
+private static final Logger logger = 
LoggerFactory.getLogger(CuratorHeartbeatMonitor.class);
+private static final Unmarshaller unmarshaller;
+
+private final ClusterCoordinator clusterCoordinator;
+private final ZooKeeperClientConfig zkClientConfig;
+private final String heartbeatPath;
+private final int heartbeatIntervalMillis;
+
+private volatile CuratorFramework curatorClient;
+private volatile ScheduledFuture future;
+private volatile Map 
latestHeartbeatMessages;
+private volatile long latestHeartbeatTime;
+
+private final FlowEngine flowEngine = new FlowEngine(1, "Heartbeat 
Monitor", true);
+
+static {
+try {
+final JAXBContext jaxbContext = 
JAXBContext.newInstance(HeartbeatMessage.class);
+unmarshaller = jaxbContext.createUnmarshaller();
+} catch (final Exception e) {
+throw new RuntimeException("Failed to create an Unmarshaller 
for unmarshalling Heartbeat Messages", e);
+}
+}
+
+public CuratorHeartbeatMonitor(final ClusterCoordinator 
clusterCoordinator, final Properties properties) {
+this.clusterCoordinator = clusterCoordinator;
+this.zkClientConfig = 
ZooKeeperClientConfig.createConfig(properties);
+this.heartbeatPath = 
zkClientConfig.resolvePath("cluster/heartbeats");
+
+final String heartbeatInterval = 
properties.getProperty(NiFiProperties.CLUSTER_PROTOCOL_HEARTBEAT_INTERVAL,
+NiFiProperties.DEFAULT_CLUSTER_PROTOCOL_HEARTBEAT_INTERVAL);
+
+this.heartbeatIntervalMillis = (int) 
FormatUtils.getTimeDuration(heartbeatInterval, TimeUnit.MILLISECONDS);
+}
+
+@Override
+public void start() {
+final RetryPolicy retryPolicy = new RetryForever(5000);
+curatorClient = 
CuratorFrameworkFactory.newClient(zkClientConfig.getConnectString(),
+

[GitHub] nifi pull request: NIFI-1028 initial commit of NiFi In Depth docum...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/339#discussion_r60105112
  
--- Diff: nifi-docs/src/main/asciidoc/nifi-in-depth.adoc ---
@@ -0,0 +1,209 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+Apache NiFi In Depth
+
+Apache NiFi Team 
+:homepage: http://nifi.apache.org
+
+Intro
+-
+This advanced level document is aimed at providing an in-depth look at the 
implementation and design decisions of NiFi. It assumes the reader has read 
enough of the other documentation to know the basics of NiFi.
+
+FlowFiles are at the heart of NiFi and its flow-based design. A FlowFile 
is just a collection of attributes and a pointer to content, which is 
associated with one or more provenance events. The attributes are key/value 
pairs that act as the metadata for the FlowFile, such as the FlowFile filename. 
The content is the actual data or the payload of the file. Provenance is a 
record of what’s happened to the FlowFile. Each one of these parts has its 
own repository (repo) for storage.
+
+One key aspect of the repositories is immutability. The content in the 
Content Repository and data within the FlowFile Repository are immutable. When 
a change occurs to the attributes of a FlowFile new copies of the attributes 
are created in memory and then persisted on disk. When content is being changed 
for a given FlowFile its original content is read, streamed through the 
transform, and written to a new stream. Then the FlowFile's content pointer is 
updated to the new location on disk. As a result, the default approach for 
FlowFile content storage can be said to be an immutable versioned content 
store.  The benefits of which are many including substantial reduction in 
storage space required for the typical complex graphs of processing, natural 
replay capability, takes advantage of OS caching, reduces random read/write 
performance hits, and is easy to reason over. The previous revisions are kept 
according to the archiving properties set in nifi.properties file and outlined 
in
  the NiFi System Administrator’s Guide.
+
+== Repositories
+There are three repositories that are utilized by NiFi. Each exists within 
the OS/Host's file system and provides specific functionality. In order to 
fully understand FlowFiles and how they are used by the underlying system it's 
important to know about these repositories. All three repositories are 
directories on local storage that NiFi uses to persist data.
+
+- The FlowFile Repository contains metadata for all the current FlowFiles 
in the flow.
+- The Content Repository holds the content for current and past FlowFiles.
+- The Provenance Repository holds the history of FlowFiles.
+
+image::NiFiArchitecture.png["NiFi Architecture Diagram"]
+
+=== FlowFile Repository
+FlowFiles that are actively being processed by the system is held in a 
hash map in the JVM memory (more about that in "Deeper View: FlowFiles in 
Memory and on Disk"). This makes it very efficient to process them, but 
requires a secondary mechanism to provide durability of data across process 
restarts due to any number of reasons. Reasons such as power loss, kernel 
panics, system upgrades, and maintenance cycles. The FlowFile Repository is a 
"Write-Ahead Log" (or data record) of the metadata of each of the FlowFiles 
that currently exist in the system. This FlowFile metadata includes all the 
attributes associated with the FlowFile, a pointer to the actual content of the 
FlowFile (which exists in the Content Repo) and the state of the FlowFile, such 
as which Connection/Queue the FlowFile belongs in. This Write-Ahead Log 
provides NiFi the resiliency it needs to handle restarts and unexpected system 
failures.
+
+The FlowFile Repository acts as NiFi's Write-Ahead Log, so as the 
FlowFiles are flowing through the system each change is logged in the FlowFile 
Repository before it happens as a transactional unit of work. This allows the 
system to know exactly what step the 

[GitHub] nifi pull request: NIFI-1771 deprecated ObjectHolder

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/351


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1028 initial commit of NiFi In Depth docum...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/339#discussion_r60103526
  
--- Diff: nifi-docs/src/main/asciidoc/nifi-in-depth.adoc ---
@@ -0,0 +1,209 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+Apache NiFi In Depth
+
+Apache NiFi Team 
+:homepage: http://nifi.apache.org
+
+Intro
+-
+This advanced level document is aimed at providing an in-depth look at the 
implementation and design decisions of NiFi. It assumes the reader has read 
enough of the other documentation to know the basics of NiFi.
+
+FlowFiles are at the heart of NiFi and its flow-based design. A FlowFile 
is just a collection of attributes and a pointer to content, which is 
associated with one or more provenance events. The attributes are key/value 
pairs that act as the metadata for the FlowFile, such as the FlowFile filename. 
The content is the actual data or the payload of the file. Provenance is a 
record of what’s happened to the FlowFile. Each one of these parts has its 
own repository (repo) for storage.
--- End diff --

May be instead of "FlowFile is just a collection of attributes and a 
pointer to content" use "FlowFile is a data record which consists of pointer to 
the content (payload) and attributes to support such content". The ordering of 
_content_ then _attributes_ is also important, otherwise what I am reading is 
that "FlowFile is just attributes. . ."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1668 modified TestProcessorLifecycle to en...

[apiri]

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/324#issuecomment-211491167
  
Actually ran into this error myself on both Travis and the efforts of 
https://issues.apache.org/jira/browse/NIFI-1654 for Appveyor.  I found that 
using JUnit temp directories as per a4ef5a1badf0ae475acb1ba73867702aa5f89c80 
seemed to remedy this, but then ran into other, similar problems later on.  Not 
sure exactly what the trigger is, but it seems to be fairly consistent across 
Windows.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1771 deprecated ObjectHolder

[alopresto]

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/351#issuecomment-211491138
  
+1. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1214

[ToivoAdams]

Github user ToivoAdams commented on a diff in the pull request:

https://github.com/apache/nifi/pull/321#discussion_r60099883
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/verifier/ConditionsBuilder.java ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.verifier;
+
+import java.util.ArrayList;
+
+public class ConditionsBuilder {
+
+protected final ArrayList conditions = new ArrayList<>();
+
+public ConditionsBuilder(Condition condition) {
+conditions.add(condition);
+}
+
+static public ConditionsBuilder attributeEqual(String name, String 
value) {
+return new ConditionsBuilder(new AttributeEqual(name,value));
+}
+
+static public ConditionsBuilder contentEqual(String content) {
+return new ConditionsBuilder(new ContentEqual(content));
+}
+
+public ConditionsBuilder andAttributeEqual(String name, String value) {
+conditions.add(new AttributeEqual(name,value));
+return this;
+}
+
+public ConditionsBuilder andContentEqual(String content) {
+conditions.add(new ContentEqual(content));
+return this;
+}
+
+public ArrayList getConditions() {
+return conditions;
+}
+
--- End diff --

You are right.
But we are moving to Java 8 and Predicate will be used instead of 
ConditionsBuilder.
So this code is not relevant anymore.

Thanks
toivo


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1668 modified TestProcessorLifecycle to en...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/324#issuecomment-211487325
  
@pvillard31 any idea why? I mean 'target' directory should be deletable. 
Just wonder if you have any idea/pointers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Aldrin Piri]

The "rel" prefix is a bit of a deviation from what we have been doing.  Up
to this point, it had been x.y.z-RC# up until a final release occurred
resulting in just an x.y.z tag.

On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:

> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>
> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:
>
> > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
> > now.
> >
> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
> release?
> > >
> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
> > >
> > >> Hello
> > >>
> > >> The Apache NiFi team would like to announce the release of Apache NiFi
> > >> 0.6.1.
> > >>
> > >> Apache NiFi is an easy to use, powerful, and reliable system to
> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
> > >> supports highly configurable directed graphs of data routing,
> > >> transformation, and system mediation logic.
> > >>
> > >> More details on Apache NiFi can be found here:
> > >>   http://nifi.apache.org/
> > >>
> > >> The release artifacts can be downloaded from here:
> > >>   http://nifi.apache.org/download.html
> > >>
> > >> Maven artifacts have been made available here:
> > >>
> > >>
> >
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
> > >>
> > >> Release note highlights can be found here:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
> > >>
> > >> Thank you
> > >> The Apache NiFi team
> > >>
> >
>

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Joe Witt]

Yep!  Thanks for catching that. Had made but not pushed.  Should be there now.

On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
> Thanks, Joe.  Will the 0.6.1 git tag be published as part of the release?
>
> On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
>
>> Hello
>>
>> The Apache NiFi team would like to announce the release of Apache NiFi
>> 0.6.1.
>>
>> Apache NiFi is an easy to use, powerful, and reliable system to
>> process and distribute data.  Apache NiFi was made for dataflow.  It
>> supports highly configurable directed graphs of data routing,
>> transformation, and system mediation logic.
>>
>> More details on Apache NiFi can be found here:
>>   http://nifi.apache.org/
>>
>> The release artifacts can be downloaded from here:
>>   http://nifi.apache.org/download.html
>>
>> Maven artifacts have been made available here:
>>
>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>>
>> Release note highlights can be found here:
>>
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>>
>> Thank you
>> The Apache NiFi team
>>

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60093499
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/FlowSnippetEndpointMerger.java
 ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import java.net.URI;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.cluster.coordination.http.EndpointResponseMerger;
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.dto.FlowSnippetDTO;
+import org.apache.nifi.web.api.dto.ProcessorDTO;
+import org.apache.nifi.web.api.dto.RemoteProcessGroupDTO;
+import org.apache.nifi.web.api.entity.FlowSnippetEntity;
+
+public class FlowSnippetEndpointMerger implements EndpointResponseMerger {
+public static final Pattern TEMPLATE_INSTANCE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller/process-groups/(?:(?:root)|(?:[a-f0-9\\-]{36}))/template-instance");
+public static final Pattern FLOW_SNIPPET_INSTANCE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller/process-groups/(?:(?:root)|(?:[a-f0-9\\-]{36}))/snippet-instance");
+
+@Override
+public boolean canHandle(final URI uri, final String method) {
+return "POST".equalsIgnoreCase(method) && 
(TEMPLATE_INSTANCE_URI_PATTERN.matcher(uri.getPath()).matches() ||
+
FLOW_SNIPPET_INSTANCE_URI_PATTERN.matcher(uri.getPath()).matches());
+}
+
+@Override
+public NodeResponse merge(final URI uri, final String method, 
Set successfulResponses, final Set 
problematicResponses, final NodeResponse clientResponse) {
+final FlowSnippetEntity responseEntity = 
clientResponse.getClientResponse().getEntity(FlowSnippetEntity.class);
+final FlowSnippetDTO contents = responseEntity.getContents();
+
+if (contents == null) {
+return clientResponse;
+} else {
+final Map> 
processorMap = new HashMap<>();
+final Map> 
remoteProcessGroupMap = new HashMap<>();
+
+for (final NodeResponse nodeResponse : successfulResponses) {
+final FlowSnippetEntity nodeResponseEntity = nodeResponse 
== clientResponse ? responseEntity : 
nodeResponse.getClientResponse().getEntity(FlowSnippetEntity.class);
+final FlowSnippetDTO nodeContents = 
nodeResponseEntity.getContents();
+
+for (final ProcessorDTO nodeProcessor : 
nodeContents.getProcessors()) {
+Map innerMap = 
processorMap.get(nodeProcessor.getId());
+if (innerMap == null) {
+innerMap = new HashMap<>();
+processorMap.put(nodeProcessor.getId(), innerMap);
+}
+
+innerMap.put(nodeResponse.getNodeId(), nodeProcessor);
+}
+
+for (final RemoteProcessGroupDTO nodeRemoteProcessGroup : 
nodeContents.getRemoteProcessGroups()) {
+Map innerMap = 
remoteProcessGroupMap.get(nodeRemoteProcessGroup.getId());
+if (innerMap == null) {
--- End diff --

This pattern appears 5 times in this PR. May be we should consider some 
utility helper class
```
public static Map<> getInnerMap(Map 
parentMap, Sting id) {
 Map innerMap = parentMap.get(id);
 if (innerMap == null) {
  innerMap = new HashMap<>();
  parentMap.put(id, 

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60092307
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ListFlowFilesEndpointMerger.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.controller.queue.ListFlowFileState;
+import org.apache.nifi.web.api.dto.FlowFileSummaryDTO;
+import org.apache.nifi.web.api.dto.ListingRequestDTO;
+import org.apache.nifi.web.api.dto.QueueSizeDTO;
+import org.apache.nifi.web.api.entity.ListingRequestEntity;
+
+public class ListFlowFilesEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern LISTING_REQUESTS_URI = Pattern
+
.compile("/nifi-api/controller/process-groups/(?:(?:root)|(?:[a-f0-9\\-]{36}))/connections/[a-f0-9\\-]{36}/listing-requests");
+public static final Pattern LISTING_REQUEST_URI = Pattern
+
.compile("/nifi-api/controller/process-groups/(?:(?:root)|(?:[a-f0-9\\-]{36}))/connections/[a-f0-9\\-]{36}/listing-requests/[a-f0-9\\-]{36}");
+
+@Override
+public boolean canHandle(URI uri, String method) {
+if (("GET".equalsIgnoreCase(method) || 
"DELETE".equalsIgnoreCase(method)) && 
LISTING_REQUEST_URI.matcher(uri.getPath()).matches()) {
+return true;
+} else if ("POST".equalsIgnoreCase(method) && 
LISTING_REQUESTS_URI.matcher(uri.getPath()).matches()) {
+return true;
+}
+
+return false;
+}
+
+@Override
+protected Class getEntityClass() {
+return ListingRequestEntity.class;
+}
+
+@Override
+protected ListingRequestDTO getDto(ListingRequestEntity entity) {
+return entity.getListingRequest();
+}
+
+@Override
+protected void mergeResponses(ListingRequestDTO clientDto, 
Map dtoMap, Set 
successfulResponses, Set problematicResponses) {
+final Comparator comparator = new 
Comparator() {
+@Override
+public int compare(final FlowFileSummaryDTO dto1, final 
FlowFileSummaryDTO dto2) {
+int positionCompare = 
dto1.getPosition().compareTo(dto2.getPosition());
+if (positionCompare != 0) {
+return positionCompare;
+}
+
+final String address1 = dto1.getClusterNodeAddress();
+final String address2 = dto2.getClusterNodeAddress();
+if (address1 == null && address2 == null) {
+return 0;
+}
+if (address1 == null) {
+return 1;
+}
+if (address2 == null) {
+return -1;
+}
+return address1.compareTo(address2);
+}
+};
+
+final NavigableSet flowFileSummaries = new 
TreeSet<>(comparator);
+
+ListFlowFileState state = null;
+int numStepsCompleted = 0;
+int numStepsTotal = 0;
+int objectCount = 0;
+long byteCount = 0;
+boolean finished = true;
+for (final Map.Entry entry : 
dtoMap.entrySet()) {
+final NodeIdentifier nodeIdentifier = 

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60090174
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ControllerStatusEndpointMerger.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.manager.StatusMerger;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.dto.BulletinDTO;
+import org.apache.nifi.web.api.dto.status.ControllerStatusDTO;
+import org.apache.nifi.web.api.entity.ControllerStatusEntity;
+
+public class ControllerStatusEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern CONTROLLER_STATUS_URI_PATTERN = 
Pattern.compile("/nifi-api/controller/status");
+
+@Override
+public boolean canHandle(URI uri, String method) {
+return "GET".equalsIgnoreCase(method) && 
CONTROLLER_STATUS_URI_PATTERN.matcher(uri.getPath()).matches();
+}
+
+@Override
+protected Class getEntityClass() {
+return ControllerStatusEntity.class;
+}
+
+@Override
+protected ControllerStatusDTO getDto(ControllerStatusEntity entity) {
+return entity.getControllerStatus();
+}
+
+@Override
+protected void mergeResponses(ControllerStatusDTO clientDto, 
Map dtoMap, Set 
successfulResponses, Set problematicResponses) {
+ControllerStatusDTO mergedStatus = clientDto;
+for (final Map.Entry entry : 
dtoMap.entrySet()) {
+final NodeIdentifier nodeId = entry.getKey();
+final ControllerStatusDTO nodeStatus = entry.getValue();
+
+final String nodeAddress = nodeId.getApiAddress() + ":" + 
nodeId.getApiPort();
+for (final BulletinDTO bulletin : nodeStatus.getBulletins()) {
+bulletin.setNodeAddress(nodeAddress);
+}
+for (final BulletinDTO bulletin : 
nodeStatus.getControllerServiceBulletins()) {
+bulletin.setNodeAddress(nodeAddress);
+}
+for (final BulletinDTO bulletin : 
nodeStatus.getReportingTaskBulletins()) {
+bulletin.setNodeAddress(nodeAddress);
+}
--- End diff --

Also, since we are using Java 8 this could be further simplified into a 
single line
```
Stream.concat(nodeStatus.getBulletins(), 
nodeStatus.getControllerServiceBulletins(), 
nodeStatus.getReportingTaskBulletins()).forEach(bulletin:: setNodeAddress)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60088663
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ControllerServicesEndpointMerger.java
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.dto.ControllerServiceDTO;
+import org.apache.nifi.web.api.entity.ControllerServicesEntity;
+
+public class ControllerServicesEndpointMerger extends 
AbstractMultiEntityEndpoint {
+public static final String CONTROLLER_SERVICES_URI = 
"/nifi-api/controller/controller-services/node";
+
+@Override
+public boolean canHandle(URI uri, String method) {
+return "GET".equalsIgnoreCase(method) && 
CONTROLLER_SERVICES_URI.equals(uri.getPath());
+}
+
+@Override
+protected Class getEntityClass() {
+return ControllerServicesEntity.class;
+}
+
+@Override
+protected Set getDtos(ControllerServicesEntity 
entity) {
+return entity.getControllerServices();
+}
+
+@Override
+protected String getComponentId(ControllerServiceDTO dto) {
+return dto.getId();
+}
--- End diff --

Not sure about the design considerations, must be missing something. . ., 
but the two methods above seem to be simply delegating to public accessors of 
the instance passed as an argument (e.g., dto.getId(), 
entity.getControllerServices() etc.). Since the calling class obviously has a 
reference to such instance, it could invoke those operations directly. Correct?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60087458
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ControllerServiceEndpointMerger.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import java.net.URI;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.controller.service.ControllerServiceState;
+import org.apache.nifi.web.api.dto.ControllerServiceDTO;
+import 
org.apache.nifi.web.api.dto.ControllerServiceReferencingComponentDTO;
+import org.apache.nifi.web.api.entity.ControllerServiceEntity;
+
+public class ControllerServiceEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final String CONTROLLER_SERVICES_URI = 
"/nifi-api/controller/controller-services/node";
+public static final Pattern CONTROLLER_SERVICE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller/controller-services/node/[a-f0-9\\-]{36}");
+
+@Override
+public boolean canHandle(URI uri, String method) {
+if (("GET".equalsIgnoreCase(method) || 
"PUT".equalsIgnoreCase(method)) && 
CONTROLLER_SERVICE_URI_PATTERN.matcher(uri.getPath()).matches()) {
+return true;
+} else if ("POST".equalsIgnoreCase(method) && 
CONTROLLER_SERVICES_URI.equals(uri.getPath())) {
+return true;
+}
+
+return false;
+}
--- End diff --

Hmm, IF -> true and ELSE -> true. I mean I see what you're doing, but i 
think it would be more readable if it was:
```
return (("GET".equalsIgnoreCase(method) || "PUT".equalsIgnoreCase(method)) 
&& CONTROLLER_SERVICE_URI_PATTERN.matcher(uri.getPath()).matches()) 
|| ("POST".equalsIgnoreCase(method) && 
CONTROLLER_SERVICES_URI.equals(uri.getPath()));
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[James Wing]

Thanks, Joe.  Will the 0.6.1 git tag be published as part of the release?

On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:

> Hello
>
> The Apache NiFi team would like to announce the release of Apache NiFi
> 0.6.1.
>
> Apache NiFi is an easy to use, powerful, and reliable system to
> process and distribute data.  Apache NiFi was made for dataflow.  It
> supports highly configurable directed graphs of data routing,
> transformation, and system mediation logic.
>
> More details on Apache NiFi can be found here:
>   http://nifi.apache.org/
>
> The release artifacts can be downloaded from here:
>   http://nifi.apache.org/download.html
>
> Maven artifacts have been made available here:
>
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>
> Thank you
> The Apache NiFi team
>

[GitHub] nifi pull request: NIFI-1727: Refactored logic for merging HTTP Re...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/338#discussion_r60086063
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMerger.java
 ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import javax.ws.rs.core.StreamingOutput;
+
+import 
org.apache.nifi.cluster.coordination.http.endpoints.BulletinBoardEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ComponentStateEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ConnectionStatusEndpiontMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ControllerServiceEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ControllerServiceReferenceEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ControllerServicesEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ControllerStatusEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.CountersEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.DropRequestEndpiontMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.FlowSnippetEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.GroupStatusEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ListFlowFilesEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.PortStatusEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProcessGroupEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProcessorEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProcessorStatusEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProcessorsEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProvenanceEventEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ProvenanceQueryEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.RemoteProcessGroupEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.RemoteProcessGroupStatusEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.RemoteProcessGroupsEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ReportingTaskEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.ReportingTasksEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.StatusHistoryEndpointMerger;
+import 
org.apache.nifi.cluster.coordination.http.endpoints.SystemDiagnosticsEndpointMerger;
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.stream.io.NullOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StandardHttpResponseMerger implements HttpResponseMerger {
+private Logger logger = 
LoggerFactory.getLogger(StandardHttpResponseMerger.class);
+private static final List endpointMergers = 
new ArrayList<>();
+
+static {
+endpointMergers.add(new ControllerStatusEndpointMerger());
+endpointMergers.add(new GroupStatusEndpointMerger());
+endpointMergers.add(new ProcessorStatusEndpointMerger());
+endpointMergers.add(new ConnectionStatusEndpiontMerger());
+endpointMergers.add(new PortStatusEndpointMerger());
+endpointMergers.add(new RemoteProcessGroupStatusEndpointMerger());
+endpointMergers.add(new ProcessorEndpointMerger());
+endpointMergers.add(new 

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60084831
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
   

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60084343
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
   

[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/319#discussion_r60083778
  
--- Diff: 
nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java
 ---
@@ -187,24 +167,83 @@ public UpdateAttribute() {
 protected List getSupportedPropertyDescriptors() {
 List descriptors = new ArrayList<>();
 descriptors.add(DELETE_ATTRIBUTES);
+descriptors.add(STORE_STATE);
+descriptors.add(STATEFUL_VARIABLES_INIT_VALUE);
 return Collections.unmodifiableList(descriptors);
 }
 
 @Override
 protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) {
-return new PropertyDescriptor.Builder()
-.name(propertyDescriptorName)
-.required(false)
-
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
 true))
-
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
-.expressionLanguageSupported(true)
-.dynamic(true)
-.build();
+if(!stateful){
+return new PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.required(false)
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.dynamic(true)
+.build();
+} else {
+return new PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.required(false)
+
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
 true))
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
+.expressionLanguageSupported(true)
+.dynamic(true)
+.build();
+}
--- End diff --

That would be a cleaner approach. I will change. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[YolandaMDavis]

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60080948
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+

Re: Is my custom processor doing too many things? OnTrigger question

[idioma]

Oleg, thanks for your help and patience, much appreciated. No, I would
actually like to stop the flow if I am catching or at least I thought it was
what I was doing with session.transfer(flowFile, FAILURE). Is that
incorrect? 



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9241.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request: NIFI-1661 add random() function to expression l...

[ckmcd]

Github user ckmcd commented on the pull request:

https://github.com/apache/nifi/pull/322#issuecomment-211429563
  
I've update the PR to address the issues (1 & 2) from above.  

1. Updated the EL Guide to use plus() instead of add()
2. Use Math.abs() so as only to generate non-negative numbers.  Updated EL 
guide to reflect this change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is my custom processor doing too many things? OnTrigger question

[Oleg Zhurakousky]

Well, you have session transfer to success after postHttp call; You don’t need 
to do that since you only wan to transfer the new file to success.
Also, I am not sure I understand the exception handling. You seem to be 
catching exception, handling it and attempt to create flow file regardless if 
exception happened. Is that the intention?

Cheers
Oleg

> On Apr 18, 2016, at 10:03 AM, idioma  wrote:
> 
> Thanks Oleg,
> would this be what I want? 
> 
> public void onTrigger(final ProcessContext context, final ProcessSession
> session) throws ProcessException {
> 
>FlowFile flowFile = session.get();
>if (flowFile == null) return;
> 
>final String userId = context.getProperty(USER_ID).getValue();
>final String password = context.getProperty(PASSWORD).getValue();
>final String http_post_url =
> context.getProperty(HTTP_POST_URL).getValue();
> 
>final AtomicReference httpPostRequestHolder = new
> AtomicReference<>();
>session.read(flowFile, new InputStreamCallback() {
>@Override
>public void process(InputStream inputStream) throws IOException
> {
>StringWriter strWriter = new StringWriter();
>IOUtils.copy(inputStream, strWriter, "UTF-8");
>httpPostRequestHolder.set(userId);
>httpPostRequestHolder.set(password);
>httpPostRequestHolder.set(http_post_url);
>}
>});
> 
>try {
>postHttpRequest(userId, password, source, message,
> http_post_url, resource_ids_file_path);
>session.transfer(flowFile, SUCCESS);
>} catch (IllegalArgumentException ex) {
>session.transfer(flowFile, FAILURE);
>ex.printStackTrace();
>}
> 
>flowFile = session.create();
>flowFile = session.write(flowFile, new OutputStreamCallback() {
>@Override
>public void process(OutputStream out) throws IOException {
>out.write(httpPostRequestHolder.get().getBytes());
>}
>});
> 
>session.transfer(flowFile, SUCCESS);
> 
> is this the correct way to write out the JSON response?
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9238.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/347#issuecomment-211417082
  
@pvillard31 I was gonna play with it a bit more and possibly create the 
test, but getting tied up with other things. Yet I was able to dig up one of my 
experiments that I used to play with RPGs in unit test environment, and since 
you already familiar with FlowController, you may make sense of it. Anyway, let 
me know. 
```java
System.setProperty("nifi.properties.file.path",

"/Users/ozhurakousky/dev/nifi-integration/src/main/resources/nifi.properties");
NiFiProperties properties = NiFiProperties.getInstance();

RingBufferEventRepository repository = new 
RingBufferEventRepository(1);
UserService us = mock(UserService.class);
AuditService as = mock(AuditService.class);
FlowController controller = 
FlowController.createStandaloneInstance(repository, properties, us, as, null);

ProcessGroup senderGroup = controller.createProcessGroup("SENDER");
senderGroup.setName("SENDER");
RemoteProcessGroup remoteProcessGroup = 
controller.createRemoteProcessGroup("SENDER_REMOTE", 
"http://localhost:8080/nifi;);
senderGroup.addRemoteProcessGroup(remoteProcessGroup);

// INPUT PORT (DESTINATIOIN)
StandardRemoteProcessGroupPortDescriptor inputPortDescriptor = new 
StandardRemoteProcessGroupPortDescriptor();
inputPortDescriptor.setId("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPortDescriptor.setName("in");

remoteProcessGroup.setInputPorts(Collections. 
singleton(inputPortDescriptor));
RemoteGroupPort inputPort = 
remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPort.setProcessGroup(senderGroup);
System.out.println(inputPort.isRunning());
System.out.println(inputPort.isTargetRunning());
System.out.println(inputPort.isValid());


// SOURCE
ProcessorNode source = 
controller.createProcessor(GenerateFlowFile.class.getName(), "A");
source.setProperty("File Size", "10 b");
source.setScheduldingPeriod("2 sec");
source.setProcessGroup(senderGroup);
senderGroup.addProcessor(source);

// CONNECTION
Connection connection = 
controller.createConnection("MyConnectionId", "MyConnectionName", source,

remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c"),
Collections.singletonList("success"));
connection.setProcessGroup(senderGroup);
senderGroup.addConnection(connection);

inputPort.addConnection(connection);
remoteProcessGroup.startTransmitting();
// inputPort.verifyCanStart();
// senderGroup.addInputPort(inputPort);
// senderGroup.startInputPort(inputPort);

//controller.startTransmitting(remoteProcessGroup.getInputPort("in"));
senderGroup.startProcessor(source);
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is my custom processor doing too many things? OnTrigger question

[idioma]

Thanks Oleg,
would this be what I want? 

public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {

FlowFile flowFile = session.get();
if (flowFile == null) return;

final String userId = context.getProperty(USER_ID).getValue();
final String password = context.getProperty(PASSWORD).getValue();
final String http_post_url =
context.getProperty(HTTP_POST_URL).getValue();

final AtomicReference httpPostRequestHolder = new
AtomicReference<>();
session.read(flowFile, new InputStreamCallback() {
@Override
public void process(InputStream inputStream) throws IOException
{
StringWriter strWriter = new StringWriter();
IOUtils.copy(inputStream, strWriter, "UTF-8");
httpPostRequestHolder.set(userId);
httpPostRequestHolder.set(password);
httpPostRequestHolder.set(http_post_url);
}
});

try {
postHttpRequest(userId, password, source, message,
http_post_url, resource_ids_file_path);
session.transfer(flowFile, SUCCESS);
} catch (IllegalArgumentException ex) {
session.transfer(flowFile, FAILURE);
ex.printStackTrace();
}

flowFile = session.create();
flowFile = session.write(flowFile, new OutputStreamCallback() {
@Override
public void process(OutputStream out) throws IOException {
out.write(httpPostRequestHolder.get().getBytes());
}
});

session.transfer(flowFile, SUCCESS);

is this the correct way to write out the JSON response?





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9238.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request: NIFI-1568: Add Filter Capability to UnpackConte...

[rickysaltzer]

Github user rickysaltzer commented on the pull request:

https://github.com/apache/nifi/pull/248#issuecomment-211408619
  
sure thing, @olegz. I just got back from vacation today, so I will try and 
take a look this week. Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is my custom processor doing too many things? OnTrigger question

[Oleg Zhurakousky]

I don’t see any response from postHttp. You just invoke a method and that’s it. 
I thought you said you need to send the results of HTTP Post (json) downstream. 
If so you should receive response form HTTP and create a new FlowFile 
(session.create(..) or session.clone(..)) and write the contents of the HTTP 
response to it (session.write(..)).

Cheers
Oleg

> On Apr 18, 2016, at 8:58 AM, idioma  wrote:
> 
> Oleg,
> this is actually a great idea, I will follow it for certain (a good
> compromise, too). I have an additional question on the onTrigger method for
> my Post Http. Among the main NiFi components, my custom processor contains a
> number of separate methods, in particular one that send a Post HTTP request
> to an endpoint and return a Json response. The method takes a number of
> parameter such as userId, password, etc and this is where I am rather
> confused when it comes to the operations I should be perform in my on
> Trigger. I have the following: 
> 
>   @Override
>public void onTrigger(final ProcessContext context, final ProcessSession
> session) throws ProcessException {
> 
>FlowFile flowFile = session.get();
>if (flowFile == null) return;
> 
>final String userId = context.getProperty(USER_ID).getValue();
>final String password = context.getProperty(PASSWORD).getValue();
>final String http_post_url =
> context.getProperty(HTTP_POST_URL).getValue();
> 
>final AtomicReference httpPostRequestHolder = new
> AtomicReference<>();
>session.read(flowFile, new InputStreamCallback() {
>@Override
>public void process(InputStream inputStream) throws IOException
> {
>StringWriter strWriter = new StringWriter();
>IOUtils.copy(inputStream, strWriter, "UTF-8");
>httpPostRequestHolder.set(userId);
>httpPostRequestHolder.set(password);
>httpPostRequestHolder.set(http_post_url);
>}
>});
> 
> try {
>postHttpRequest(userId, password, http_post_url);
>session.transfer(flowFile, SUCCESS);
>} catch (IllegalArgumentException ex) {
>session.transfer(flowFile, FAILURE);
>ex.printStackTrace();
>}
> 
> Is that what I am supposed to do? Is that correct? I am trying to process
> the inputstream and convert it into an AtomicReference object with the same
> arguments as the one passed in the postHttpRequest method. Does it make
> sense? 
> 
> Thank you again for all your help, so much appreciated!
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9232.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

[GitHub] nifi-minifi pull request: MINIFI-14 Incorporating Listener/Notifie...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/10#discussion_r60062578
  
--- Diff: 
minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/conf/bootstrap.conf
 ---
@@ -28,6 +28,16 @@ conf.dir=./conf
 # How long to wait after telling MiNiFi to shutdown before explicitly 
killing the Process
 graceful.shutdown.seconds=20
 
+# The location for the configuration file
+nifi.minifi.config=./conf/config.yml
+
+# Notifiers to use for the associated agent, comma separated list of class 
names

+#nifi.minifi.notifier.components=org.apache.nifi.minifi.bootstrap.configuration.FileChangeNotifier
+
+# File change notifier configuration for path of the file to monitor and 
the associated polling period
+#nifi.minifi.notifier.file.config.path=
--- End diff --

Sounds good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is my custom processor doing too many things? OnTrigger question

[idioma]

Oleg,
this is actually a great idea, I will follow it for certain (a good
compromise, too). I have an additional question on the onTrigger method for
my Post Http. Among the main NiFi components, my custom processor contains a
number of separate methods, in particular one that send a Post HTTP request
to an endpoint and return a Json response. The method takes a number of
parameter such as userId, password, etc and this is where I am rather
confused when it comes to the operations I should be perform in my on
Trigger. I have the following: 

   @Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {

FlowFile flowFile = session.get();
if (flowFile == null) return;

final String userId = context.getProperty(USER_ID).getValue();
final String password = context.getProperty(PASSWORD).getValue();
final String http_post_url =
context.getProperty(HTTP_POST_URL).getValue();
   
final AtomicReference httpPostRequestHolder = new
AtomicReference<>();
session.read(flowFile, new InputStreamCallback() {
@Override
public void process(InputStream inputStream) throws IOException
{
StringWriter strWriter = new StringWriter();
IOUtils.copy(inputStream, strWriter, "UTF-8");
httpPostRequestHolder.set(userId);
httpPostRequestHolder.set(password);
httpPostRequestHolder.set(http_post_url);
}
});

 try {
postHttpRequest(userId, password, http_post_url);
session.transfer(flowFile, SUCCESS);
} catch (IllegalArgumentException ex) {
session.transfer(flowFile, FAILURE);
ex.printStackTrace();
}

Is that what I am supposed to do? Is that correct? I am trying to process
the inputstream and convert it into an AtomicReference object with the same
arguments as the one passed in the postHttpRequest method. Does it make
sense? 

Thank you again for all your help, so much appreciated!





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9232.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi-minifi pull request: MINIFI-14 Incorporating Listener/Notifie...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/10#discussion_r60062452
  
--- Diff: 
minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/conf/bootstrap.conf
 ---
@@ -28,6 +28,16 @@ conf.dir=./conf
 # How long to wait after telling MiNiFi to shutdown before explicitly 
killing the Process
 graceful.shutdown.seconds=20
 
+# The location for the configuration file
+nifi.minifi.config=./conf/config.yml
+
+# Notifiers to use for the associated agent, comma separated list of class 
names

+#nifi.minifi.notifier.components=org.apache.nifi.minifi.bootstrap.configuration.FileChangeNotifier
+
+# File change notifier configuration for path of the file to monitor and 
the associated polling period
+#nifi.minifi.notifier.file.config.path=
--- End diff --

This has turned out to be not quite as straightforward as hoped and think 
for now will punt until we see what needs there are.  I did create a new issue 
to track this, however: https://issues.apache.org/jira/browse/MINIFI-16


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/319#discussion_r60060311
  
--- Diff: 
nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java
 ---
@@ -187,24 +167,83 @@ public UpdateAttribute() {
 protected List getSupportedPropertyDescriptors() {
 List descriptors = new ArrayList<>();
 descriptors.add(DELETE_ATTRIBUTES);
+descriptors.add(STORE_STATE);
+descriptors.add(STATEFUL_VARIABLES_INIT_VALUE);
 return Collections.unmodifiableList(descriptors);
 }
 
 @Override
 protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) {
-return new PropertyDescriptor.Builder()
-.name(propertyDescriptorName)
-.required(false)
-
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
 true))
-
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
-.expressionLanguageSupported(true)
-.dynamic(true)
-.build();
+if(!stateful){
+return new PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.required(false)
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.dynamic(true)
+.build();
+} else {
+return new PropertyDescriptor.Builder()
+.name(propertyDescriptorName)
+.required(false)
+
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
 true))
+
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
+.expressionLanguageSupported(true)
+.dynamic(true)
+.build();
+}
+}
+
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor, 
final String oldValue, final String newValue) {
+super.onPropertyModified(descriptor, oldValue, newValue);
+
+if (descriptor.equals(STORE_STATE)) {
+if ("true".equalsIgnoreCase(newValue)) {
+stateful = true;
+} else {
+stateful = false;
+}
+}
--- End diff --

Yup, will fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/319#discussion_r60060257
  
--- Diff: 
nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java
 ---
@@ -489,17 +546,32 @@ private FlowFile executeActions(final ProcessSession 
session, final ProcessConte
 final Map attributesToUpdate = new 
HashMap<>(actions.size());
 final Set attributesToDelete = new 
HashSet<>(actions.size());
 
+final Map statefulAttributesToSet;
+
+if (statefulAttributes != null){
+statefulAttributesToSet = new HashMap<>();
+} else {
+statefulAttributesToSet = null;
+}
+
+
 // go through each action
 for (final Action action : actions.values()) {
 if 
(!action.getAttribute().equals(DELETE_ATTRIBUTES.getName())) {
--- End diff --

I didn't write this piece of code so I'm not entirely sure but I highly 
doubt it since its been on master for a while and hasn't been a problem so far.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is my custom processor doing too many things? OnTrigger question

[Oleg Zhurakousky]

Idioma

I would suggest  for your learning experience to have custom processor to do 
your HTTP stuff and if successful transfer to ‘success’ relationship and 
connect it to PutKafka processor that comes with NiFi.
This way you’ll not only be able to learn on how to develop custom processor 
but also see it being integrated with another processor that was not developed 
by you.

Cheers
Oleg

> On Apr 18, 2016, at 8:27 AM, idioma  wrote:
> 
> Thank you Joe and thank you for understanding the struggle of a newbie, not
> many communities are so welcoming and inclusive like Apache NiFi Developer
> List! :) Yes, you are right posting to a URL endpoint to get a response and
> put that on Kafka is all I want, do you have any existing out of the box
> processors you can point me to? I actually wanted to create my own custom
> processor for my personal benefit and for understanding how to build them,
> so I probably will try to go ahead with a custom one, but bearing in mind
> that I can fully exploit out of the box bundles. Am I heading towards the
> right direction with the code I posted? 
> 
> Thank you
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9228.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Re: Is my custom processor doing too many things? OnTrigger question

[idioma]

Thank you Joe and thank you for understanding the struggle of a newbie, not
many communities are so welcoming and inclusive like Apache NiFi Developer
List! :) Yes, you are right posting to a URL endpoint to get a response and
put that on Kafka is all I want, do you have any existing out of the box
processors you can point me to? I actually wanted to create my own custom
processor for my personal benefit and for understanding how to build them,
so I probably will try to go ahead with a custom one, but bearing in mind
that I can fully exploit out of the box bundles. Am I heading towards the
right direction with the code I posted? 

Thank you



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225p9228.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Is my custom processor doing too many things? OnTrigger question

[Joe Witt]

Idioma,

No worries about questions being newbie questions.  We're here to help
and grow the community so you have to start somewhere!

You will get lots of opinions on whether something is doing too many
things.  For this example you've provided I'd say yes it is probably
doing too many things.  There are already processors to interact with
web services and put to kafka.  In the case of this particular
processor do you want do to anything other than post to an HTTP
endpoint to get a response and put that on Kafka?  If not then the out
of the box processors may be all you need.

Thanks
Joe

On Mon, Apr 18, 2016 at 7:25 AM, idioma  wrote:
> Hi,
> I have some fundamental questions on a custom Processor I am working on.
> First of all, I think I am trying to do too many things in one custom
> processor (correct me if I am wrong :) as 1) I am sending a Post HTTP
> request 2) Posting the JSON response to Kafka. In my custom processor I have
> all the required properties and two methods one that prepares and send the
> HTTP request and another that post the result to Kafka. My OnScheduled
> checks if the post Url connection and the kafka one can be established. It
> looks something like this:
>
> String http_post_url = context.getProperty(HTTP_POST_URL).getValue();
> String kafka_url = context.getProperty(KAFKA_URL).getValue();
>
> try {
> HttpURLConnection httpPostUrlConnection = (HttpURLConnection)
> httpPostUrl.openConnection();
> } catch (IOException ex) {
> ex.printStackTrace();
> }
>
> URL kafkaUrl = new URL(kafka_url);
>
> try {
> HttpURLConnection kafkaUrlConnection = (HttpURLConnection)
> kafkaUrl.openConnection();
> } catch (IOException ex) {
> ex.printStackTrace();
> }
>
> Now, all my doubts are in the onTrigger method as I am still struggling to
> understand what really should be happening there. I guess that what I want
> is to call the method that constructs the HTTP post request and then do the
> same thing with the posting to kafka. Am I doing too many things in my
> onTrigger? My postToKafka will look something like this (I am using
> AtomicReference now):
>
> final AtomicReference postToKafkaHolder = new AtomicReference<>();
>
> session.read(flowFile, new InputStreamCallback() {
>
> @Override
> public void process(InputStream in) throws IOException {
>
> StringWriter strWriter = new StringWriter();
> IOUtils.copy(in, strWriter, "UTF-8");
> String contents = strWriter.toString();
> postToKafkaHolder.set(contents);
> }
>
> });
>
> try {
> postToKafka(postToKafkaHolder.get());
> session.transfer(flowFile, SUCCESS);
> } catch (IOException e) {
> session.transfer(flowFile, FAILURE);
> }
>
> Similarly, this should be done the postHTTPRequest, would that be correct in
> the same onTrigger? I apologise for such a newbie question, but I am still
> confused of how work with the concept of flowfile.
>
> Thank you so much for your help!
>
>
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request: NIFI-1724 Added properties to configure log lev...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Is my custom processor doing too many things? OnTrigger question

[idioma]

Hi,
I have some fundamental questions on a custom Processor I am working on.
First of all, I think I am trying to do too many things in one custom
processor (correct me if I am wrong :) as 1) I am sending a Post HTTP
request 2) Posting the JSON response to Kafka. In my custom processor I have
all the required properties and two methods one that prepares and send the
HTTP request and another that post the result to Kafka. My OnScheduled
checks if the post Url connection and the kafka one can be established. It
looks something like this: 

String http_post_url = context.getProperty(HTTP_POST_URL).getValue();
String kafka_url = context.getProperty(KAFKA_URL).getValue();

try {
HttpURLConnection httpPostUrlConnection = (HttpURLConnection)
httpPostUrl.openConnection();
} catch (IOException ex) {
ex.printStackTrace();
}

URL kafkaUrl = new URL(kafka_url);

try {
HttpURLConnection kafkaUrlConnection = (HttpURLConnection)
kafkaUrl.openConnection();
} catch (IOException ex) {
ex.printStackTrace();
}

Now, all my doubts are in the onTrigger method as I am still struggling to
understand what really should be happening there. I guess that what I want
is to call the method that constructs the HTTP post request and then do the
same thing with the posting to kafka. Am I doing too many things in my
onTrigger? My postToKafka will look something like this (I am using
AtomicReference now):

final AtomicReference postToKafkaHolder = new AtomicReference<>();

session.read(flowFile, new InputStreamCallback() {

@Override
public void process(InputStream in) throws IOException {

StringWriter strWriter = new StringWriter();
IOUtils.copy(in, strWriter, "UTF-8");
String contents = strWriter.toString();
postToKafkaHolder.set(contents);
}

});

try {
postToKafka(postToKafkaHolder.get());
session.transfer(flowFile, SUCCESS);
} catch (IOException e) {
session.transfer(flowFile, FAILURE);
}

Similarly, this should be done the postHTTPRequest, would that be correct in
the same onTrigger? I apologise for such a newbie question, but I am still
confused of how work with the concept of flowfile.

Thank you so much for your help!





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Is-my-custom-processor-doing-too-many-things-OnTrigger-question-tp9225.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[ANNOUNCE] Apache NiFi 0.6.1 release

[Joe Witt]

Hello

The Apache NiFi team would like to announce the release of Apache NiFi 0.6.1.

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute data.  Apache NiFi was made for dataflow.  It
supports highly configurable directed graphs of data routing,
transformation, and system mediation logic.

More details on Apache NiFi can be found here:
  http://nifi.apache.org/

The release artifacts can be downloaded from here:
  http://nifi.apache.org/download.html

Maven artifacts have been made available here:
  https://repository.apache.org/content/repositories/releases/org/apache/nifi/

Release note highlights can be found here:
  
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1

Thank you
The Apache NiFi team

Re: Multiple nar/custom processors: advisable directory structure

[idioma]

Yes, but eventually I have resorted to copy and pasted the existing nar and
processor folders and changing references in the pom.xml and in the
resources folder.

Thank you!



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Multiple-nar-custom-processors-advisable-directory-structure-tp9089p9223.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request: NIFI-1724 Added properties to configure log lev...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/348#issuecomment-211349732
  
@pvillard31 Looks good, will test and merge. Could you please reassign this 
issue in JIRA to yourself (since you did all the work)? Even if its resolved it 
would be nice for tracking purposes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1777 Prevent deleting a connection going t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/357


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1777 Prevent deleting a connection going t...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/357#issuecomment-211322321
  
Was able to validate that when/if someone attempts to delete a connected 
processor a window pops up with the error message "Destination of Connection 
(LogAttribute[id=7a37a5bb-1017-4732-8c31-a6c1f44eea18]) is running". So all 
good, adequate tests provided, +1.

@mcgilman @scottyaslan  perhaps in the future we can further address it (as 
an additional feature) in the UI and disable "Delete" option all together. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1777 Prevent deleting a connection going t...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/357#issuecomment-211319503
  
@pvillard31 also, when you submit PR click on "Submit Patch" in JIRA 
(without writing anything in description). I know it may be a bit awkward but 
this is how we signal that JIRA is ready for review/resolution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1777 Prevent deleting a connection going t...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/357#issuecomment-211318338
  
Perfect, reviewing now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-211295813
  
@olegz I just added a unit test even if I don't think it is a vital one! 
Otherwise it's ready for final review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/347#issuecomment-211283811
  
@olegz I agree a unit test would be nice, but I must admit that I am not 
sure to see how to do it. It involves RPG, remote ports, remote processors and 
I don't see how I can mock the whole remote flow to get the expected testing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1777 Prevent deleting a connection going t...

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/357#issuecomment-211276565
  
@olegz You're right, just added a unit test. (Note: I believe that travis 
build failure is not related to this PR)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1724 Added properties to configure log lev...

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/348#issuecomment-211276895
  
@olegz You're right, just added a unit test. (Note: I believe that travis 
build failure is not related to this PR)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---