date:20160319

Bryan,

I believe the archetype is already updated if this what you're referring to.

https://github.com/apache/nifi/blob/master/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-nar/pom.xml

Thanks
Joe

On Wed, Mar 16, 2016 at 9:19 AM, Bryan Bende  wrote:
> Also sounds like we need to update the archetype based on whatever approach
> we come up with, either adding those properties  to the NAR Pom in the
> archetype, or having it use a specific parent.
>
> On Wednesday, March 16, 2016, Joe Witt  wrote:
>
>> Would certainly like to better understand what you have in mind.
>>
>> thanks
>>
>> On Wed, Mar 16, 2016 at 12:02 AM, Sean Busbey > > wrote:
>> > we could make a parent pom for all the nar modules.
>> >
>> > wanna see what that looks like?
>> >
>> > On Tue, Mar 15, 2016 at 8:46 PM, Joe Witt > > wrote:
>> >> Team,
>> >>
>> >> During the previous build/release cycle it was found that
>> >> javadocs/sources were being made for the Nar bundles themselves and
>> >> was causing invalid licensing/notice information to be present.  All
>> >> the existing bundles and the archetypes were fixed for this.  Just be
>> >> sure on new nars to include these as well if you aren't copying from
>> >> something existing or using the archetype.  I just fixed a couple of
>> >> them for new things in the 0.6.0 release.
>> >>
>> >> The nar pom itself should have a properties section such as
>> >>
>> >> 
>> >> true
>> >> true
>> >> 
>> >>
>> >> Perhaps there is a nicer maven way of ensuring this doesn't happen for
>> Nars.
>> >>
>> >> Thanks
>> >> Joe
>>
>
>
> --
> Sent from Gmail Mobile

[GitHub] nifi pull request: NIFI-1488 Refactoring HBase Kerberos support

Github user markap14 commented on the pull request:

https://github.com/apache/nifi/pull/281#issuecomment-197888150
  
@bbende code looks good. Just a few minor comments in-line (stylistic for 
the most part). There were a lot of files that were changed due to just white 
space and Java imports being reorganized, which made the review a bit harder 
but otherwise, the code all looks good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1488 Refactoring HBase Kerberos support

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/281#discussion_r56506078
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/test/resources/hbase-site-security.xml
 ---
@@ -0,0 +1,30 @@
+
+
--- End diff --

I think we should keep the License header above the xml-stylesheet 
directive, no?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1488 Refactoring HBase Kerberos support

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/281#discussion_r56410644
  
--- Diff: 
nifi-commons/nifi-hadoop-utils/src/main/java/org/apache/nifi/hadoop/KerberosProperties.java
 ---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.hadoop;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * All processors and controller services that need properties for 
Kerberos Principal and Keytab
+ * should obtain them through this class by calling:
+ *
+ * KerberosProperties props = 
KerberosProperties.create(NiFiProperties.getInstance())
+ *
+ * The properties can be accessed from the resulting KerberosProperties 
instance.
+ */
+public class KerberosProperties {
+
+private final File kerberosConfigFile;
+private final Validator kerberosConfigValidator;
+private final PropertyDescriptor kerberosPrincipal;
+private final PropertyDescriptor kerberosKeytab;
+
+private KerberosProperties(final File kerberosConfigFile) {
+this.kerberosConfigFile = kerberosConfigFile;
+
+if (this.kerberosConfigFile != null) {
+System.setProperty("java.security.krb5.conf", 
kerberosConfigFile.getAbsolutePath());
+}
+
+this.kerberosConfigValidator = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+// Check that the Kerberos configuration is set
+if (kerberosConfigFile == null) {
+return new ValidationResult.Builder()
+.subject(subject).input(input).valid(false)
+.explanation("you are missing the 
nifi.kerberos.krb5.file property which "
++ "must be set in order to use 
Kerberos")
+.build();
+}
+
+// Check that the Kerberos configuration is readable
+if (!kerberosConfigFile.canRead()) {
+return new 
ValidationResult.Builder().subject(subject).input(input).valid(false)
+.explanation(String.format("unable to read 
Kerberos config [%s], please make sure the path is valid "
++ "and nifi has adequate permissions", 
kerberosConfigFile.getAbsoluteFile()))
+.build();
+}
+
+return new 
ValidationResult.Builder().subject(subject).input(input).valid(true).build();
+}
+};
+
+this.kerberosPrincipal = new PropertyDescriptor.Builder()
+.name("Kerberos Principal")
+.required(false)
+.description("Kerberos principal to authenticate as. 
Requires nifi.kerberos.krb5.file to be set in your nifi.properties")
+.addValidator(kerberosConfigValidator)
+.build();
+
+this.kerberosKeytab = new PropertyDescriptor.Builder()
+.name("Kerberos Keytab").required(false)
+.description("Kerberos keytab associated with the 
principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties")
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.addValidator(kerberosConfigValidator)

Processor additional documentation

Hello,

I am writing my first processor.

As described in the documentation, I have added an HTML file to be used when 
the user selects "Usage":

docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html

This is located in the root or the Processors nar file.

The processor class is this:

com/datamelt/nifi/test/TemplateProcessor.class

The processor works, but selecting "Usage" won't show my HTML file.

I understood that I write the HTML file and Nifi will picks it up when it 
starts. Or is this not true?

Thanks for feedback,

Uwe

[GitHub] nifi pull request: Nifi 1274

Github user mcgilman commented on the pull request:

https://github.com/apache/nifi/pull/284#issuecomment-197950387
  
+1 from me as well. Verified functionality for both standalone and 
clustered instances. Looking good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: patch and pull request reviewing question

2016-03-19 Thread Oleg Zhurakousky

Going back to T's original point, I'd go as far as asking the following:
Once the PR is issued, do NOT squash until asked to do so.
The reason for it is that I just had a great experience with Percival on new 
JMS support. There were 6+ follow ups. Joe only needed to review what we 
agreed/argued, nothing else. Once he was satisfied he asked me to squash and 
merged. Done deal.
The Spring PR (assuming this's thread origin) was a bit different since I 
somehow thought that fresh look at the whole thing would be better. My bad!

That is why I love Git, but went right against its wisdom;)

Cheers 
Oleg

Sent from my iPhone

> On Mar 17, 2016, at 22:21, Aldrin Piri  wrote:
> 
> Sounds good to here as well.
> 
> Given the increasing number of contributions, which predominantly seem to
> arrive via GitHub PR, I also created NIFI-1615 [1] a little while ago to
> make a GitHub PR template [1].  Would like to distill the contributor guide
> down into a few easy steps for folks to check out and make apprised of
> before they commit the PR that I think can help our reviewers and
> infrastructure.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1615
> [2] https://github.com/blog/2111-issue-and-pull-request-templates
> 
> On Thu, Mar 17, 2016 at 9:55 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> +1 here as well
>> My apologies Tony
>> 
>> Sent from my iPhone
>> 
>>> On Mar 17, 2016, at 21:42, Joe Witt  wrote:
>>> 
>>> +1
>>> 
 On Thu, Mar 17, 2016 at 9:37 PM, Tony Kurc  wrote:
 As I was reviewing a pull request, i realized that it actually was a bit
 more mental effort in this instance to review a squashed and rebased
>> set of
 commits than if it was if I could just review the changes in a commit.
>> Does
 anyone object to me adding in the contributors guide something to the
 extent of "Although you may be asked to rebase or squash your
>> contribution
 as part of the review process, don't feel the need to do so
>> speculatively.
 The committer working on merging the contribution may prefer to do these
 types of operations as part of the merge process, and the history of
>> your
 patch or pull request may aid in the review process"
 
 Tony
>>

Re: Rollbacks

2016-03-19 Thread Devin Fisher

Cool, thanks. I'll take a look at that Processor.  I've learned a lot from
looking at the processor in the main project.

Devin

On Thu, Mar 17, 2016 at 10:46 AM, Joe Witt  wrote:

> Devin,
>
> Good stuff.  Unless the api call to change the content completes
> normally the transformation effectively did not happen.  Take a look
> at CompressContent [1] for an example.
>
> It reads in the original content and streams out new content.  If that
> process of transformation fails it routes the flow file to failure.
> What gets routed is the original representation of the data.
>
> Go ahead and give a really simple form of your case a try.  Do
> something like always throw an exception in your callback to test it
> and then you can validate the behavior meets your needs.  In fact,
> that could be a great test case.
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/CompressContent.java
>
> Thanks
> Joe
>
> On Thu, Mar 17, 2016 at 12:36 PM, Devin Fisher
>  wrote:
> > Thanks for the detailed response. The details about the batching were
> very
> > helpful. I've changed my processors to take advantage of the batching
> that
> > framework provides since my use case fits what you described to a tee.
> >
> > But I have a question about the failure relationship. Maybe a concrete
> > example might help. So let's say I have an XML document that I'm
> processing
> > with a Processor that converts XML to JSON. This Processor makes use an
> SAX
> > parser so it doesn't have to read the whole XML document into memory. The
> > SAX parser read from an input stream provided by Nifi and calls SAX
> > callbacks base on the XML it encounters. These callbacks write the JSON
> to
> > an output stream provided by Nifi. So as soon as the Processor starts it
> > has written to the current Session. If towards the end of the XML
> document
> > it hits a syntax error, let say there is a text element with an '&' that
> > has not been encoded as an entity. The SAX parser will fail and will not
> be
> > able to complete successfully. (the parser is very strict) But the
> > processor will have already written most of the resulting JSON but not
> all.
> > At this point if I transfer it the failed relationship without rolling
> back
> > what will I get? Will I get the incomplete JSON or will I get the invalid
> > but complete XML? I'm guessing that it will be half written JSON. What I
> > would think you would want is the complete XML so that a processor that
> > fixes the '&' entity issue could handle the issue and route it back. Or a
> > different processor could log it or something else.
> >
> > Anyway, I hope that example explains the confusion that I still have
> about
> > handling error cases. Rolling back here would not be helpful from what I
> > understand because it will just get processed again by the same Processor
> > with the SAX parser that can't handle the '&' over and over again. The
> > penalty will make sure that other documents get process before it is
> tried
> > again but that one malformed XML document will never progress because it
> > will always fail in the same way each time and be rollback again.
> >
> > Again thanks for the reply. The info was really useful and helped my
> > understanding.
> >
> > Devin
> >
> > On Tue, Mar 15, 2016 at 7:27 PM, Joe Witt  wrote:
> >
> >> Devin,
> >>
> >> So the session follows the unit of work pattern that Martin Fowler
> >> describes [1].  The idea here is that whether you access one flow file
> >> or many flow files or anything in between you are doing some session
> >> of work.  You can commit all the things you do on that session or
> >> rollback all the things you do on that session.
> >>
> >> This pattern/concept provides a nice and safe environment in which the
> >> contract between the developer and framework is well understood.
> >> Generally, rollback is only necessary when some unplanned or
> >> unexpected exception occurs and it is largely there so that the
> >> framework can ensure all things are returned to a safe state.  It can
> >> also do things like penalize that processor/extension so that if it is
> >> some programming error that it will reduce its impact on the system
> >> overall.
> >>
> >> So, with that said there are two ways to think about your case.  It
> >> appears you are doing your own batching and probably this is for
> >> higher throughput and also it appears you'd really like to treat each
> >> flow file independently in terms of logic/handling.
> >>
> >> This is precisely why in addition to this nice clean unit of work
> >> pattern we also support automated session batching (this is what
> >> Andrew was referring to).  In this mode you can add an annotation to
> >> your processor called @SupportsBatching which signals to the framework
> >> that it may attempt to

[GitHub] nifi pull request: NIFI-1647 fixed validators and config resolutio...

GitHub user olegz opened a pull request:

https://github.com/apache/nifi/pull/288

NIFI-1647 fixed validators and config resolution



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/olegz/nifi NIFI-1647

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #288


commit ebc2cf68aaaf7df4d5350c0dbfaa9b77a4c3a327
Author: Oleg Zhurakousky 
Date:   2016-03-18T13:26:54Z

NIFI-1647 fixed validators and config resolution




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464 removed e.printStackTrace() from onTr...

GitHub user olegz opened a pull request:

https://github.com/apache/nifi/pull/287

NIFI-1464 removed e.printStackTrace() from onTrigger in StandarProcesâ¦

â¦sScheduler

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/olegz/nifi NIFI-1464F

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #287


commit 2e6482a9ad8f98e149e0ad40002dfd4bed999bb9
Author: Oleg Zhurakousky 
Date:   2016-03-17T20:52:07Z

NIFI-1464 removed e.printStackTrace() from onTrigger in 
StandarProcessScheduler




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Re: Processor additional documentation

2016-03-19 Thread dan bress

Uwe,
   No its not a problem to have both index.html and additionalDetails.html
 The NiFi framework generates nearly all of the documentation for your
processor for you.  It will generate information about the properties and
relationships your processor exposes to its users.  If you need to express
more about your processor, then that is where additionalDetails.html comes
into play.  For example, if your processor uses a custom query language.

Generated index.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html

additionalDetails.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html

On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken  wrote:

> Bryan,
>
> all looks ok. I looked into the nifi-home/work/docs folder. There is
> nothing but a components folder. Inside there is a folder for my processor:
> com.datamelt.nifi.test.TemplateProcessor and inside the folder there is a
> file index.html and it contains the code of my additionalDetails.html file.
>
> when I open the file in the web browser it looks good. I looked at other
> index.html files and they look similar.
>
> but I noted that some folders have an inde.html file AND an
> additionalDetails.html file. maybe that is the problem?
>
> greetings,
>
> Uwe
>
>
>
> Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> Von: "Bryan Bende" 
> An: dev@nifi.apache.org
> Betreff: Re: Processor additional documentation
> Hi Uwe,
>
> Do you have the additionalDetails.html file in your processors jar project,
> under src/main/resources?
>
> Similar to this:
>
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
>
> The expected project structure is described here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> 
>
> If you think that part is setup correctly, can you check under
> nifi_home/work/docs and see if com.datamelt.nifi.test.TemplateProcessor is
> there?
>
> -Bryan
>
> On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken 
> wrote:
>
> >
> > Hello,
> >
> > I am writing my first processor. As described in the documentation, I
> have
> > added an HTML file to be used when the user selects "Usage":
> >
> > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> >
> > This is located in the root or the Processors nar file.
> >
> > The processor class is this:
> >
> > com/datamelt/nifi/test/TemplateProcessor.class
> >
> > The processor works, but selecting "Usage" won't show my HTML file.
> >
> > I understood that I write the HTML file and Nifi will picks it up when it
> > starts. Or is this not true?
> >
> > Thanks for feedback,
> >
> > Uwe
> >
>

[GitHub] nifi pull request: nifi-1562 ExecuteStreamCommand and ExecuteProce...

Github user mattyb149 commented on the pull request:

https://github.com/apache/nifi/pull/247#issuecomment-198162336
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: quick reminder on new nars

2016-03-19 Thread Bryan Bende

Ah perfect, didn't know that was already done.

On Wed, Mar 16, 2016 at 9:24 AM, Joe Witt  wrote:

> Bryan,
>
> I believe the archetype is already updated if this what you're referring
> to.
>
>
> https://github.com/apache/nifi/blob/master/nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-nar/pom.xml
>
> Thanks
> Joe
>
> On Wed, Mar 16, 2016 at 9:19 AM, Bryan Bende  wrote:
> > Also sounds like we need to update the archetype based on whatever
> approach
> > we come up with, either adding those properties  to the NAR Pom in the
> > archetype, or having it use a specific parent.
> >
> > On Wednesday, March 16, 2016, Joe Witt  wrote:
> >
> >> Would certainly like to better understand what you have in mind.
> >>
> >> thanks
> >>
> >> On Wed, Mar 16, 2016 at 12:02 AM, Sean Busbey  >> > wrote:
> >> > we could make a parent pom for all the nar modules.
> >> >
> >> > wanna see what that looks like?
> >> >
> >> > On Tue, Mar 15, 2016 at 8:46 PM, Joe Witt  >> > wrote:
> >> >> Team,
> >> >>
> >> >> During the previous build/release cycle it was found that
> >> >> javadocs/sources were being made for the Nar bundles themselves and
> >> >> was causing invalid licensing/notice information to be present.  All
> >> >> the existing bundles and the archetypes were fixed for this.  Just be
> >> >> sure on new nars to include these as well if you aren't copying from
> >> >> something existing or using the archetype.  I just fixed a couple of
> >> >> them for new things in the 0.6.0 release.
> >> >>
> >> >> The nar pom itself should have a properties section such as
> >> >>
> >> >> 
> >> >> true
> >> >> true
> >> >> 
> >> >>
> >> >> Perhaps there is a nicer maven way of ensuring this doesn't happen
> for
> >> Nars.
> >> >>
> >> >> Thanks
> >> >> Joe
> >>
> >
> >
> > --
> > Sent from Gmail Mobile
>

[GitHub] nifi pull request: Nifi 1274

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/284#issuecomment-197978496
  
All of @mcgilman 's changes are great. The test resources are files I added 
that make it easier to set up a Kerberized instance of NiFi for testing, but 
were not related to any code tests. We should remove them and I will try to 
follow up with integration tests for a future release. Thanks. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Rollbacks

2016-03-19 Thread Devin Fisher

Thanks for the detailed response. The details about the batching were very
helpful. I've changed my processors to take advantage of the batching that
framework provides since my use case fits what you described to a tee.

But I have a question about the failure relationship. Maybe a concrete
example might help. So let's say I have an XML document that I'm processing
with a Processor that converts XML to JSON. This Processor makes use an SAX
parser so it doesn't have to read the whole XML document into memory. The
SAX parser read from an input stream provided by Nifi and calls SAX
callbacks base on the XML it encounters. These callbacks write the JSON to
an output stream provided by Nifi. So as soon as the Processor starts it
has written to the current Session. If towards the end of the XML document
it hits a syntax error, let say there is a text element with an '&' that
has not been encoded as an entity. The SAX parser will fail and will not be
able to complete successfully. (the parser is very strict) But the
processor will have already written most of the resulting JSON but not all.
At this point if I transfer it the failed relationship without rolling back
what will I get? Will I get the incomplete JSON or will I get the invalid
but complete XML? I'm guessing that it will be half written JSON. What I
would think you would want is the complete XML so that a processor that
fixes the '&' entity issue could handle the issue and route it back. Or a
different processor could log it or something else.

Anyway, I hope that example explains the confusion that I still have about
handling error cases. Rolling back here would not be helpful from what I
understand because it will just get processed again by the same Processor
with the SAX parser that can't handle the '&' over and over again. The
penalty will make sure that other documents get process before it is tried
again but that one malformed XML document will never progress because it
will always fail in the same way each time and be rollback again.

Again thanks for the reply. The info was really useful and helped my
understanding.

Devin

On Tue, Mar 15, 2016 at 7:27 PM, Joe Witt  wrote:

> Devin,
>
> So the session follows the unit of work pattern that Martin Fowler
> describes [1].  The idea here is that whether you access one flow file
> or many flow files or anything in between you are doing some session
> of work.  You can commit all the things you do on that session or
> rollback all the things you do on that session.
>
> This pattern/concept provides a nice and safe environment in which the
> contract between the developer and framework is well understood.
> Generally, rollback is only necessary when some unplanned or
> unexpected exception occurs and it is largely there so that the
> framework can ensure all things are returned to a safe state.  It can
> also do things like penalize that processor/extension so that if it is
> some programming error that it will reduce its impact on the system
> overall.
>
> So, with that said there are two ways to think about your case.  It
> appears you are doing your own batching and probably this is for
> higher throughput and also it appears you'd really like to treat each
> flow file independently in terms of logic/handling.
>
> This is precisely why in addition to this nice clean unit of work
> pattern we also support automated session batching (this is what
> Andrew was referring to).  In this mode you can add an annotation to
> your processor called @SupportsBatching which signals to the framework
> that it may attempt to automatically combine subsequent calls to
> commit into small batches and commit them in a single batch.  In this
> way you can build your processor in a very simple single flow file
> sort of manner and call commit.  But the framework will combine a
> series of commits in a very small time window together to get higher
> throughput.  In the UI a user can signal their willingness to let the
> framework to do this and acknowledge that they may be trading off some
> small latency in favor of higher throughput.  Now there are some
> additional things to think about when using this.  For instance, it is
> best used when the processor and its function is side effect free
> meaning that there are no external system state changes or things like
> that.  In this sense you can think of the processor you're building as
> idempotent (as the REST folks like to say).  If your processor fits
> that then SupportsBatching can have really powerful results.
>
> Now, you also mention that some flow files you'd consider failures and
> others you'd consider something else, presumably success.  This is
> perfect and very common and does not require a rollback.  Keep
> rollback in mind for bad stuff that can happen that you don't plan
> for.  For the scenario of failures that you can predict such as
> invalid data or invalid state of something you actually want to have a
> failure relationship

[GitHub] nifi pull request: nifi-1562 ExecuteStreamCommand and ExecuteProce...

Github user mattyb149 commented on the pull request:

https://github.com/apache/nifi/pull/247#issuecomment-198182467
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/271#discussion_r56446243
  
--- Diff: 
nifi-nar-bundles/nifi-spring-bundle/nifi-spring-processors/src/main/java/org/apache/nifi/spring/SpringContextProcessor.java
 ---
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.spring;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.spring.SpringDataExchanger.SpringResponse;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.MessageChannel;
+import org.springframework.messaging.PollableChannel;
+
+/**
+ * Implementation of {@link Processor} capable of sending and receiving 
data
+ * from application defined in Spring Application context. It does so via
+ * predefined in/out {@link MessageChannel}s (see spring-messaging module 
of
+ * Spring). Once such channels are defined user is free to implement the 
rest of
+ * the application any way they wish (e.g., custom code and/or using 
frameworks
+ * such as Spring Integration or Camel).
+ * 
+ * The requirement and expectations for channel types are:
+ * 
+ * Input channel must be of type {@link MessageChannel} and named 
"fromNiFi"
+ * (see {@link SpringNiFiConstants#FROM_NIFI})
+ * Output channel must be of type {@link PollableChannel} and named 
"toNiFi"
+ * (see {@link SpringNiFiConstants#TO_NIFI})
+ * 
+ * 
+ * Below is the example of sample configuration:
+ *
+ * 
+ * ?xml version="1.0" encoding="UTF-8"?
+ * beans xmlns="http://www.springframework.org/schema/beans;
+ *   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ *   xmlns:int="http://www.springframework.org/schema/integration;
+ *  xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
+ *  http://www.springframework.org/schema/integration 
http://www.springframework.org/schema/integration/spring-integration-4.2.xsd";
+ *
+ *  int:channel id="fromNiFi"/
+ *
+ *  . . . . .
+ *
+ *  int:channel id="toNiFi"
+ *  int:queue/
+ *  /int:channel
+ *
+ * /beans
+ * 
+ * 
+ * Defining {@link MessageChannel} is optional. That's why this processor
+ * supports 3 modes of interaction with Spring Application Context:
+ * 
+ * Headless â no channels

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user rickysaltzer commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56422806
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

Sure thing, my battery just died so I will as soon as able. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56507643
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/jwt/JwtService.java
 ---
@@ -70,8 +70,6 @@ public String getAuthenticationFromToken(final String 
base64EncodedToken) throws
 
 // TODO: Validate issuer against active registry?
--- End diff --

NVM, it's not something that was modified in this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Closing in on the Apache NiFi 0.6.0 release

2016-03-19 Thread Tony Kurc

Recommend https://issues.apache.org/jira/browse/NIFI-1651 be included in
0.6.0

On Wed, Mar 16, 2016 at 4:08 PM, Joe Witt  wrote:

> Team,
>
> Ok sooo close.  We have 5 tickets remaining.
>
> - Additional functionality/cleanup for SplitText [1]
> [status] Still in discussions. Recommend we move this change to 0.7.0.
> Solid effort on both code contributor and reviewer side but this is a
> tricky one.
>
> - Support Kerberos based authentication to REST API [2]
> [status] PR is in. Reviewing and PR tweaking appears active.  Looks
> quite close and comments indicate great results.
>
> - Add Kerberos support to HBase processors [3]
> [status] Patch in. Under review.  Running on live test system with
> great results.
>
> - Add support for Spring Context loaded processors (Spring
> Integrations, Camel, ...) [4]
> [status] Appears ready. Getting review feedback.
>
> - Zookeeper interaction for NiFI state management should limit state to
> 1MB [6]
> [status] Patch is in and review under way.  Looks close.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1118
> [2] https://issues.apache.org/jira/browse/NIFI-1274
> [3] https://issues.apache.org/jira/browse/NIFI-1488
> [4] https://issues.apache.org/jira/browse/NIFI-1571
> [5] https://issues.apache.org/jira/browse/NIFI-1626
>
> Thanks
>
> On Wed, Mar 16, 2016 at 4:04 PM, Joe Witt  wrote:
> > Team,
> >
> > Ok sooo close.  We have 6 tickets remaining.
> >
> > - Additional functionality/cleanup for SplitText [1]
> > [status] Still in discussions. Recommend we move this change to 0.7.0.
> > Solid effort on both code contributor and reviewer side but this is a
> > tricky one.
> >
> > - Support Kerberos based authentication to REST API [2]
> > [status] PR is in. Reviewing and PR tweaking appears active.  Looks
> > quite close and comments indicate great results.
> >
> > - Add Kerberos support to HBase processors [3]
> > [status] Patch in. Under review.  Running on live test system with
> > great results.
> >
> > - Add support for Spring Context loaded processors (Spring
> > Integrations, Camel, ...) [4]
> > [status] Appears ready. Getting review feedback.
> >
> > - Support based database snapshot/query/change capture [5]
> > [status] Appears ready. Needs review.
> >
> > - Zookeeper interaction for NiFI state management should limit state to
> 1MB [6]
> > [status] Patch is in and review under way.  Looks close.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-1118
> > [2] https://issues.apache.org/jira/browse/NIFI-1274
> > [3] https://issues.apache.org/jira/browse/NIFI-1488
> > [4] https://issues.apache.org/jira/browse/NIFI-1571
> > [5] https://issues.apache.org/jira/browse/NIFI-1575
> > [6] https://issues.apache.org/jira/browse/NIFI-1626
> >
> > Thanks
> > Joe
> >
> > On Tue, Mar 15, 2016 at 11:56 AM, Aldrin Piri 
> wrote:
> >> Sounds great.  Will scope those out in the process.  Thanks, Tony.
> >>
> >> On Tue, Mar 15, 2016 at 11:37 AM, Tony Kurc  wrote:
> >>
> >>> Aldrin,
> >>> I did some crappy shell scripts to help, I'll try to put those up on
> the
> >>> wiki.
> >>>
> >>> Tony
> >>>
> >>> On Tue, Mar 15, 2016 at 11:29 AM, Aldrin Piri 
> >>> wrote:
> >>>
> >>> > In an effort to avoid me missing helpers and spamming the list, I
> will
> >>> > throw my hat into the ring to do this one.
> >>> >
> >>> > On Tue, Mar 15, 2016 at 11:28 AM, Joe Witt 
> wrote:
> >>> >
> >>> > > Team,
> >>> > >
> >>> > > Seeing good progress today.  Great!
> >>> > >
> >>> > > I know I volunteered to do the RM for the release but now I need
> to be
> >>> > > out of town and will have questionable Internet access.  Can
> someone
> >>> > > else grab it?
> >>> > >
> >>> > > Thanks
> >>> > > Joe
> >>> > >
> >>> > > On Mon, Mar 14, 2016 at 9:01 PM, Joe Witt 
> wrote:
> >>> > > > Team,
> >>> > > >
> >>> > > > Things are closing in well but we need to get pretty specific on
> >>> these
> >>> > > > to keep the release moving roughly along the lines of the
> schedule
> >>> > > > we've discussed previously.  Those of you who have tickets on
> here
> >>> > > > that can be moved to 0.7.0 please do so.  Otherwise, let's please
> >>> keep
> >>> > > > discussion/status information up so we can all see how we're
> trending
> >>> > > > toward release candidate readiness.
> >>> > > >
> >>> > > > The current items on the release based on JIRA fix version that
> >>> remain
> >>> > > are:
> >>> > > >
> >>> > > > - Supporting a new JMS producer/consumer style [1] [2] [3]
> >>> > > > [status]Great rounds of review with Oleg, Percivall, Moser.
> Looks
> >>> > close.
> >>> > > >
> >>> > > > - Processors to interact with Apache Cassandra [4]
> >>> > > > [status]Looks ready but needing review.
> >>> > > >
> >>> > > > - Additional functionality/cleanup for SplitText [5]
> >>> > > > [status]Still in discussions.  Recommend we move this change to
> >>> 0.7.0.
> >>> > > >
> >>> >

Re: Processor additional documentation

2016-03-19 Thread Russell Bateman


Uwe,

When I asked this question a few days ago. What you're doing is rolling 
your whole processor into a NAR instead of JARing it first, then rolling 
the JAR into a NAR. This doesn't work and still won't work when they fix 
it (not just for the reason of /additional//Details.html/ because 
there's other stuff that doesn't work either). So don't do it that way.


   Russell,

   There is a Maven archetype that will generate the project structure
   for you:

   
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-MavenProcessorArchetype

   That page also shows how the bundle should be laid out. Let us know
   if it doesn't make sense.

   -Bryan


On 03/17/2016 02:30 PM, Uwe Geercken wrote:

Hello,

I am writing my first processor.

As described in the documentation, I have added an HTML file to be used when the user 
selects "Usage":

docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html

This is located in the root or the Processors nar file.

The processor class is this:

com/datamelt/nifi/test/TemplateProcessor.class

The processor works, but selecting "Usage" won't show my HTML file.

I understood that I write the HTML file and Nifi will picks it up when it 
starts. Or is this not true?

Thanks for feedback,

Uwe

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/271#issuecomment-197617908
  
@olegz : checkstyle fails when I build with your latest commit
``
[INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ 
nifi-spring-processors ---
[WARNING] 
src/main/java/org/apache/nifi/spring/SpringContextFactory.java[44] (regexp) 
RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] 
src/main/java/org/apache/nifi/spring/SpringContextFactory.java[51] (regexp) 
RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] 
src/main/java/org/apache/nifi/spring/SpringContextFactory.java[61] (regexp) 
RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] 
src/test/java/org/apache/nifi/spring/SpringContextFactoryTests.java[20:15] 
(imports) UnusedImports: Unused import - org.junit.Assert.assertFalse.
``


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/271#discussion_r56446675
  
--- Diff: 
nifi-nar-bundles/nifi-spring-bundle/nifi-spring-processors/src/main/java/org/apache/nifi/spring/SpringContextProcessor.java
 ---
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.spring;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.spring.SpringDataExchanger.SpringResponse;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.MessageChannel;
+import org.springframework.messaging.PollableChannel;
+
+/**
+ * Implementation of {@link Processor} capable of sending and receiving 
data
+ * from application defined in Spring Application context. It does so via
+ * predefined in/out {@link MessageChannel}s (see spring-messaging module 
of
+ * Spring). Once such channels are defined user is free to implement the 
rest of
+ * the application any way they wish (e.g., custom code and/or using 
frameworks
+ * such as Spring Integration or Camel).
+ * 
+ * The requirement and expectations for channel types are:
+ * 
+ * Input channel must be of type {@link MessageChannel} and named 
"fromNiFi"
+ * (see {@link SpringNiFiConstants#FROM_NIFI})
+ * Output channel must be of type {@link PollableChannel} and named 
"toNiFi"
+ * (see {@link SpringNiFiConstants#TO_NIFI})
+ * 
+ * 
+ * Below is the example of sample configuration:
+ *
+ * 
+ * ?xml version="1.0" encoding="UTF-8"?
+ * beans xmlns="http://www.springframework.org/schema/beans;
+ *   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ *   xmlns:int="http://www.springframework.org/schema/integration;
+ *  xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
+ *  http://www.springframework.org/schema/integration 
http://www.springframework.org/schema/integration/spring-integration-4.2.xsd";
+ *
+ *  int:channel id="fromNiFi"/
+ *
+ *  . . . . .
+ *
+ *  int:channel id="toNiFi"
+ *  int:queue/
+ *  /int:channel
+ *
+ * /beans
+ * 
+ * 
+ * Defining {@link MessageChannel} is optional. That's why this processor
+ * supports 3 modes of interaction with Spring Application Context:
+ * 
+ * Headless â no channels

Re: patch and pull request reviewing question

2016-03-19 Thread Aldrin Piri

Sounds good to here as well.

Given the increasing number of contributions, which predominantly seem to
arrive via GitHub PR, I also created NIFI-1615 [1] a little while ago to
make a GitHub PR template [1].  Would like to distill the contributor guide
down into a few easy steps for folks to check out and make apprised of
before they commit the PR that I think can help our reviewers and
infrastructure.

[1] https://issues.apache.org/jira/browse/NIFI-1615
[2] https://github.com/blog/2111-issue-and-pull-request-templates

On Thu, Mar 17, 2016 at 9:55 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> +1 here as well
> My apologies Tony
>
> Sent from my iPhone
>
> > On Mar 17, 2016, at 21:42, Joe Witt  wrote:
> >
> > +1
> >
> >> On Thu, Mar 17, 2016 at 9:37 PM, Tony Kurc  wrote:
> >> As I was reviewing a pull request, i realized that it actually was a bit
> >> more mental effort in this instance to review a squashed and rebased
> set of
> >> commits than if it was if I could just review the changes in a commit.
> Does
> >> anyone object to me adding in the contributors guide something to the
> >> extent of "Although you may be asked to rebase or squash your
> contribution
> >> as part of the review process, don't feel the need to do so
> speculatively.
> >> The committer working on merging the contribution may prefer to do these
> >> types of operations as part of the merge process, and the history of
> your
> >> patch or pull request may aid in the review process"
> >>
> >> Tony
> >
>

[GitHub] nifi pull request: Nifi 1495 - AWS Kinesis Firehose

2016-03-19 Thread apiri

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/213#issuecomment-197294536
  
@mans2singh Looks great, thanks for getting those changes together and the 
contribution.  I'll be merging this into master shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/271#issuecomment-198126926
  
Also, the Camel example was added for testing here 
https://github.com/olegz/si-demo .@PuspenduBanerjee if/when you get a chance 
please check it out. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1488 Refactoring HBase Kerberos support

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/281#discussion_r56411887
  
--- Diff: 
nifi-commons/nifi-hadoop-utils/src/main/java/org/apache/nifi/hadoop/SecurityUtil.java
 ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.hadoop;
+
+import org.apache.commons.lang3.Validate;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.nifi.logging.ComponentLog;
+
+import java.io.IOException;
+
+/**
+ * Provides synchronized access to UserGroupInformation to avoid multiple 
processors/services from
+ * interfering with each other.
+ */
+public class SecurityUtil {
+
+/**
+ * Initializes UserGroupInformation with the given Configuration and 
performs the login for the given principal
+ * and keytab. All logins should happen through this class to ensure 
other threads are not concurrently modifying
+ * UserGroupInformation.
+ *
+ * @param config the configuration instance
+ * @param principal the principal to authenticate as
+ * @param keyTab the keytab to authenticate with
+ *
+ * @return the UGI for the given principal
+ *
+ * @throws IOException if login failed
+ */
+public static synchronized UserGroupInformation loginKerberos(final 
Configuration config, final String principal, final String keyTab)
+throws IOException {
+Validate.notNull(config);
+Validate.notNull(principal);
+Validate.notNull(keyTab);
+
+UserGroupInformation.setConfiguration(config);
+return 
UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal.trim(), 
keyTab.trim());
+}
+
+/**
+ * Initializes UserGroupInformation with the given Configuration and 
returns UserGroupInformation.getLoginUser().
+ * All logins should happen through this class to ensure other threads 
are not concurrently modifying
+ * UserGroupInformation.
+ *
+ * @param config the configuration instance
+ *
+ * @return the UGI for the given principal
+ *
+ * @throws IOException if login failed
+ */
+public static synchronized UserGroupInformation loginSimple(final 
Configuration config) throws IOException {
+Validate.notNull(config);
+UserGroupInformation.setConfiguration(config);
+return UserGroupInformation.getLoginUser();
+}
+
+/**
+ * Initializes UserGroupInformation with the given Configuration and 
returns UserGroupInformation.isSecurityEnabled().
+ *
+ * All checks for isSecurityEnabled() should happen through this 
method.
+ *
+ * @param config the given configuration
+ *
+ * @return true if kerberos is enabled on the given configuration, 
false otherwise
+ *
+ */
+public static synchronized boolean isSecurityEnabled(final 
Configuration config) {
+Validate.notNull(config);
+return 
"kerberos".equalsIgnoreCase(config.get("hadoop.security.authentication"));
+}
+
+/**
+ * Start a thread that periodically attempts to renew the current 
Kerberos user's ticket.
+ *
+ * Callers of this method should store the reference to the 
KerberosTicketRenewer and call stop() to stop the thread.
+ *
+ * @param clazz
+ *  The class that this renewal is for (i.e. PutHDFS, etc)
+ * @param ugi
+ *  The current Kerberos user.
+ * @param renewalPeriod
+ *  The amount of time between attempting renewals.
+ * @param logger
+ *  The logger to use with in the renewer
+ *
+ * @return the KerberosTicketRenewer Runnable
+ */
+public static KerberosTicketRenewer startTicketRenewalThread(final 
Class clazz, final

[GitHub] nifi pull request: NIFI-1617 Add source filename metadata to Ident...

2016-03-19 Thread jfrazee

Github user jfrazee closed the pull request at:

https://github.com/apache/nifi/pull/273


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56496561
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/kerberos/KerberosAuthenticationFilter.java
 ---
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.security.kerberos;
+
+import org.apache.nifi.web.security.InvalidAuthenticationException;
+import org.apache.nifi.web.security.NiFiAuthenticationFilter;
+import org.apache.nifi.web.security.token.NiFiAuthorizationRequestToken;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.core.AuthenticationException;
+import 
org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider;
+
+import javax.servlet.FilterChain;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.util.Arrays;
+
+/**
+ */
+public class KerberosAuthenticationFilter extends NiFiAuthenticationFilter 
{
--- End diff --

I don't believe this filter is necessary anymore as the SPNEGO negotiation 
is performed in the AccessResource at /access/kerberos.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/271#issuecomment-198171253
  
@olegz, I think @joewitt is still looking, but I think the code looks good, 
I put it through the paces using several applications. I'm +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1651 unit tests work on windows. removed c...

GitHub user trkurc opened a pull request:

https://github.com/apache/nifi/pull/291

NIFI-1651 unit tests work on windows. removed contributor name/package from 
data and schema



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trkurc/nifi NIFI-1651

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #291


commit bfd90e6587a449f23dd1ddc6e2cc36448e59e881
Author: trkurc 
Date:   2016-03-19T18:48:44Z

NIFI-1651 unit tests work on windows. removed contributor name/package from 
data and schema




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

GitHub user rickysaltzer opened a pull request:

https://github.com/apache/nifi/pull/285

NIFI-1636: Print Stacktrace When Unepected OnTrigger Exception Caught



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rickysaltzer/nifi NIFI-1636

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/285.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #285


commit e82464694435ee2e5f27cfbb00c0e6d7b53d89ce
Author: ricky 
Date:   2016-03-16T20:43:05Z

NIFI-1636: Print Stacktrace When Unepected OnTrigger Exception Caught




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: EndpointConnectionPool is not reusable.

2016-03-19 Thread Paresh Shah

Joe,

For our use case we want to be able to do the following.

1. In certain paths we want to be able to send the same FlowFiles to all
nodes of a remote cluster ( broadcast use case )
2. In another use case send it to one of the nodes of the remote cluster (
regular RPG use case ).

What we are working on is a general purpose DistributeProcessor that works
of a application defined strategy. This strategy in-turn would determine
which nodes need to be sent the data to.

We can set up a meeting to discuss this in more detail.

Paresh

On 3/16/16, 7:34 AM, "Joe Percivall" 
wrote:

>Hello Paresh,
>
>Just to be sure I understand your use-case correctly, you want to
>transfer FlowFiles from one NiFi instance to a NiFi cluster using site to
>site and you want to have the FlowFiles go to a certain node in the
>cluster based on an application specific strategy.
>
>I have not heard any plans to make the routing extensible but that's
>primarily because I don't think any one has asked for it before.
>
>Forgive me if this is a dumb question but, why not group similar nodes
>together into smaller clusters instead of treating them all the same?
>
>Sorry no one had gotten back to you earlier,
>
>Joe
>
>- - - - - -
>Joseph Percivall
>linkedin.com/in/Percivall
>e: joeperciv...@yahoo.com
>
>
>
>
>
>On Friday, March 11, 2016 10:19 AM, Paresh Shah
> wrote:
>Have not seen a response from the Nifi team on this. Can someone from the
>team, please share their views on this.
>
>Thanks
>Paresh
>
>
>From: Paresh Shah
>>
>Date: Tuesday, March 8, 2016 at 11:58 AM
>To: "dev@nifi.apache.org"
>>
>Subject: EndpointConnectionPool is not reusable.
>
>We have a use case where we would like to perform some routing based on
>application defined strategy. In order to be able to do this we need to
>have a mechanism to pass in the selcted nodes based on the app strategy.
>
>Ideally we would like to pass this strategy into the
>EndpointConnectionPool. Currently we see that the EndpointConnectionPool
>is not implemented as an interface and thus SocketClient explicitly
>contains an instance of this class. Also the logic to perform the load
>balancing is buried inside the EndpointConnectionPool.
>
>Are there any plans in the near term to make these components extensible.
>
>Thanks
>Paresh
>
>
>The information contained in this transmission may contain privileged and
>confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>

 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/271#discussion_r56445529
  
--- Diff: 
nifi-nar-bundles/nifi-spring-bundle/nifi-spring-processors/src/main/java/org/apache/nifi/spring/SpringContextProcessor.java
 ---
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.spring;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.spring.SpringDataExchanger.SpringResponse;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.MessageChannel;
+import org.springframework.messaging.PollableChannel;
+
+/**
+ * Implementation of {@link Processor} capable of sending and receiving 
data
+ * from application defined in Spring Application context. It does so via
+ * predefined in/out {@link MessageChannel}s (see spring-messaging module 
of
+ * Spring). Once such channels are defined user is free to implement the 
rest of
+ * the application any way they wish (e.g., custom code and/or using 
frameworks
+ * such as Spring Integration or Camel).
+ * 
+ * The requirement and expectations for channel types are:
+ * 
+ * Input channel must be of type {@link MessageChannel} and named 
"fromNiFi"
+ * (see {@link SpringNiFiConstants#FROM_NIFI})
+ * Output channel must be of type {@link PollableChannel} and named 
"toNiFi"
+ * (see {@link SpringNiFiConstants#TO_NIFI})
+ * 
+ * 
+ * Below is the example of sample configuration:
+ *
+ * 
+ * ?xml version="1.0" encoding="UTF-8"?
+ * beans xmlns="http://www.springframework.org/schema/beans;
+ *   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ *   xmlns:int="http://www.springframework.org/schema/integration;
+ *  xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
+ *  http://www.springframework.org/schema/integration 
http://www.springframework.org/schema/integration/spring-integration-4.2.xsd";
+ *
+ *  int:channel id="fromNiFi"/
+ *
+ *  . . . . .
+ *
+ *  int:channel id="toNiFi"
+ *  int:queue/
+ *  /int:channel
+ *
+ * /beans
+ * 
+ * 
+ * Defining {@link MessageChannel} is optional. That's why this processor
+ * supports 3 modes of interaction with Spring Application Context:
+ * 
+ * Headless â no channels are

[GitHub] nifi pull request: NIFI-1617 Add source filename metadata to Ident...

Github user mattyb149 commented on the pull request:

https://github.com/apache/nifi/pull/273#issuecomment-198185301
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] New Apache NiFi Committer Oleg Zhurakousky

2016-03-19 Thread Joe Skora

Oleg,

Congratulations, and thanks for all your help, past and future!

Joe

On Wed, Mar 16, 2016 at 10:11 AM, Pierre Villard <
pierre.villard...@gmail.com> wrote:

> Well deserved! Congrats!
> On Mar 16, 2016 13:27, "Joe Percivall" 
> wrote:
>
> > Congrats Oleg!
> >  - - - - - -
> > Joseph Percivall
> > linkedin.com/in/Percivall
> > e: joeperciv...@yahoo.com
> >
> >
> >
> >
> > On Tuesday, March 15, 2016 9:45 PM, Jeremy Dyer 
> wrote:
> > Congrats Oleg!
> >
> > Sent from my iPhone
> >
> >
> > > On Mar 15, 2016, at 9:01 PM, Ricky Saltzer  wrote:
> > >
> > > Big congrats!
> > >> On Mar 15, 2016 8:42 PM, "Michael Moser"  wrote:
> > >>
> > >> Yes, welcome Oleg!  Your hard work is very much appreciated by
> everyone.
> > >>
> > >> -- Mike
> > >>
> > >>
> > >>> On Tue, Mar 15, 2016 at 8:22 PM, Matt Burgess 
> > wrote:
> > >>>
> > >>> Congratulations! Well deserved.
> > >>>
> >  On Mar 15, 2016, at 8:17 PM, Tony Kurc  wrote:
> > 
> >  On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > >> Oleg
> >  Zhurakousky has accepted the PMC's invitation to become a committer
> on
> > >>> the
> >  Apache NiFi project. We greatly appreciate all of Oleg's hard work
> and
> >  generous contributions to the project. We look forward to his
> > continued
> >  involvement in the project.
> > 
> >  Some highlights of Oleg's contributions include identifying and
> > >>> mitigating
> >  some challenging framework bugs, amqp support in 0.5.x, and some
> > pretty
> >  exciting spring support slated for 0.6.x. As I have noticed, many of
> > >> you
> >  have already had the opportunity to interact with Oleg on the
> mailing
> >  lists, the jiras, and pull request comments. I, for one, look
> forward
> > >> to
> >  the continued community support!
> > 
> >  Welcome and congratulations!
> > 
> >  Tony
> > >>
> >
>

[GitHub] nifi pull request: NIFI-856 Implements experimental ListenLumberja...

2016-03-19 Thread trixpan

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/290

NIFI-856  Implements experimental ListenLumberjack Processor

ListenLumberjack processor should allow NiFi to act as a drop in 
replacement on pipelines 
using logstash-forwarder and Logstash's Lumberjack output to move data 
created by
distributed systems.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-856v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #290


commit 777852f1e994be737eeb3f8b832ffd3db25e02f7
Author: Andre F de Miranda 
Date:   2016-03-18T14:41:35Z

NIFI-856  Implements experimental ListenLumberjack Processor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Aw: Re: Re: Processor: User friendly vs system friendly design

Adam,

ok. I got it. This way one can use any flowfile attributes and merge them with 
the template.

I have worked all day on the two processors and it does work - very cool. Will 
have to do more error testing though.

So the SplitToAttribute processor does the splitting of the flowfile content. 
It has three properties:

Attribute prefix: how the attributes will be prefixed
Field seperator: used to split the content
Field Number format: how to format the positional number of the field

The MergeTemplate processor merges attributes from a flowfile with the Apache 
Velocity template. It has three properties:

Template path: where the template is stored
Template name: name of the template
Attribute filter: which fields to make available to the template engine

And then I have two relationships outgoing from the MergeTemplate processor: 
One for the original content of the flowfile and one for the result of the 
merge of the attributes with the template.

Will do more testing. And still my additionalDetails.html documentation does 
not work.

Rgds,

Uwe


> Gesendet: Freitag, 18. März 2016 um 22:41 Uhr
> Von: "Adam Taft" 
> An: dev@nifi.apache.org
> Betreff: Re: Re: Processor: User friendly vs system friendly design
>
> Uwe,
> 
> I'll take a look at your code sometime soon.  However, just to point you in
> the direction, I'd suggest extracting your single line CSV data into
> flowfile attributes named as you've demonstrated.  i.e.  create a processor
> which reads each CSV column as a flowfile attribute, using a configured
> naming convention.
> 
> For example, using "column" as your prefix with your example input, you'd
> end up with a single flowfile with attributes like:
> 
> column0 = Peterson
> column1 = Jenny
> column2 = New York
> column3 = USA
> 
> Flowfile attributes are effectively a Map.  So in your
> Velocity processor, you would pass the Map of flowfile attributes to the
> template engine and record the results to the flowfile content.
> 
> Using SplitText seems correct up front (though like you said, you lose the
> CSV header line).  You'd need two additional processors, from my
> perspective:
> 
> (input) -> SplitText -> ExtractCSVColumns -> ApplyVelocityTemplate ->
> (output)
> 
> It's the "split row into fields and merge with template" that we would
> want to separate into two processors instead of one.
> 
> You're very much on the right track, I believe.  If the above doesn't help,
> I'll try and jump in on a code example when I can.
> 
> Adam
> 
> 
> On Fri, Mar 18, 2016 at 5:04 PM, Uwe Geercken  wrote:
> 
> > Adam,
> >
> > I don't see an obvious way for your suggestion of "Read columns from a
> > single CSV line into flowfile attributes." - I would need your advice how I
> > can achieve it.
> >
> > Thinking about it in more detail, I have following issues:
> > - the incomming flowfile may have many columns. so adding the columns
> > manually as attributes with UpdateAttributes is not feasible
> > - I have setup a flow where I use SplitText to divide the flowfile into
> > multiple flowfiles, so there won't be a header row I can use to get the
> > column names. So I think I can only use abstract column names plus a
> > running number. e.g. column0, column1, etc.
> >
> > So for the moment I have coded the processor like described below. At the
> > moment I am still "thinking in CSV" but I will check it with other formats
> > later. The user can steer follwoing settings: path where the template is
> > stored, name of the template file, the label for the columns (I call it
> > prefix) and the separator based on which the split of the row is done.
> >
> > Example Flowfile content (user has chosen "comma" as separator:
> >
> > Peterson, Jenny, New York, USA
> >
> > Example template (user has chosen "column" as the prefix):
> >
> > {
> > "name": "$column0",
> > "first": "$column1",
> > "city": "$column2",
> > "country": "$column3"
> > }
> >
> > Example flow:
> >
> > GetFile: Get CSV File >> SplitText : split into multiple flowfiles, one
> > per row >> TemplateProcessor:
> > 
> > split row into fields and merge with template >> MergeContent: merge
> > flowfiles into one >> PutFile: put the file to the filesystem
> >
> > Example result:
> >
> > {
> > "name": "Peterson",
> > "first": "Jenny",
> > "city": "New York",
> > "country": "USA"
> >  }
> >
> > I will test the processor now for larger files, empty files and other
> > exceptions. If you are interested the code is here:
> >
> > https://github.com/uwegeercken/nifi_processors
> >
> > Greetings,
> >
> > Uwe
> >
> >
> >
> > > Gesendet: Freitag, 18. März 2016 um 18:58 Uhr
> > > Von: "Adam Taft" 
> > > An: dev@nifi.apache.org
> > > Betreff: Re: Processor: User friendly vs system friendly design
> > >
> > > Uwe,
> > >
> > > The Developer Guide[1] and Contributor Guide[2] are pretty solid.  The
> > > Developer

[GitHub] nifi pull request: NIFI-1575: Add QueryDatabaseTable processor

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/261#discussion_r56380052
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
 ---
@@ -0,0 +1,607 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.util.JdbcCommon;
+import org.apache.nifi.util.LongHolder;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.math.BigDecimal;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.text.DecimalFormat;
+import java.text.ParseException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import static java.sql.Types.ARRAY;
+import static java.sql.Types.BIGINT;
+import static java.sql.Types.BINARY;
+import static java.sql.Types.BIT;
+import static java.sql.Types.BLOB;
+import static java.sql.Types.BOOLEAN;
+import static java.sql.Types.CHAR;
+import static java.sql.Types.CLOB;
+import static java.sql.Types.DATE;
+import static java.sql.Types.DECIMAL;
+import static java.sql.Types.DOUBLE;
+import static java.sql.Types.FLOAT;
+import static java.sql.Types.INTEGER;
+import static java.sql.Types.LONGNVARCHAR;
+import static java.sql.Types.LONGVARBINARY;
+import static java.sql.Types.LONGVARCHAR;
+import static java.sql.Types.NCHAR;
+import static java.sql.Types.NUMERIC;
+import static java.sql.Types.NVARCHAR;
+import static java.sql.Types.REAL;
+import static java.sql.Types.ROWID;
+import static java.sql.Types.SMALLINT;
+import static java.sql.Types.TIME;
+import static java.sql.Types.TIMESTAMP;
+import static java.sql.Types.TINYINT;
+import static java.sql.Types.VARBINARY;
+import static java.sql.Types.VARCHAR;
+
+@EventDriven
+@InputRequirement(Requirement.INPUT_ALLOWED)
+@Tags({"sql", "select", "jdbc", "query", "database"})
+@CapabilityDescription("Execute provided SQL select query. Query result 
will be converted to Avro format."
++ " Streaming is used so arbitrarily large result sets are 
supported. This processor can be scheduled to run on "
++ "a timer, or cron expression,

[GitHub] nifi pull request: Nifi 1495 - AWS Kinesis Firehose

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/213


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/271#issuecomment-198134497
  
@olegz - there are a bunch of trailing whitespace and whitespace on blank 
lines in the xml and properties files. checkstyle doesn't complain as it checks 
the java code only, but it did cause warnings when I applied the patch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1617 Add source filename metadata to Ident...

Github user mattyb149 commented on the pull request:

https://github.com/apache/nifi/pull/273#issuecomment-198202258
  
I merged this but forgot to close the PR, @jfrazee can you do the honors?

Thanks for the contrib!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1629 downgraded Kafka back to 0.8

Github user rickysaltzer commented on the pull request:

https://github.com/apache/nifi/pull/282#issuecomment-197505410
  
+1 - tested and verified to work with 0.8.0 and 0.9.0 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/280#discussion_r56327650
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java
 ---
@@ -143,72 +199,82 @@ protected void init(final 
ProcessorInitializationContext context) {
 return properties;
 }
 
-private int readLines(final InputStream in, final int maxNumLines, 
final OutputStream out, final boolean keepAllNewLines) throws IOException {
+private int readLines(final InputStream in, final int maxNumLines, 
final long maxByteCount, final OutputStream out) throws IOException {
 int numLines = 0;
+long totalBytes = 0L;
 for (int i = 0; i < maxNumLines; i++) {
-final long bytes = countBytesToSplitPoint(in, out, 
keepAllNewLines || (i != maxNumLines - 1));
+final long bytes = countBytesToSplitPoint(in, out, totalBytes, 
maxByteCount);
+totalBytes += bytes;
 if (bytes <= 0) {
 return numLines;
 }
-
 numLines++;
+if (totalBytes >= maxByteCount && numLines > maxNumLines) {
--- End diff --

The logic here appears to be incorrect.
numLines is incremented for each iteration of the loop (unless we return 
before it is incremented).
This means that numLines <= i
The loop's condition indicates i < maxNumLines
So numLines <= i < maxNumLines
So it is always the case that numLines < maxNumLines, so this condition 
will never be satisfied because numLines will never be > maxNumLines

Now, looking through the code and doing a bit of testing, this does not 
appear to return an incorrect result, since countBytesToSplitPoint will handle 
the logic appropriately itself, but this should be fixed before it is merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: User friendly versus system friendly Question

2016-03-19 Thread Russell Bateman

Uwe,

While my usage is surely unrelated to yours, I do exactly this thing
which is to process a CSV producing as many new output flowfiles as
there are lines and the names of these flowfiles are, in my case,
conveniently chosen based on the significant content of each line (so no
problem coming up with names as there is a person's name and other
discriminating particulars in the result).

I have another processor that analyzes in-coming XML files and breaks
them each into several flowfiles according to dates associated in the
content I'm separating out. Again, no difficulty choosing easily
recognizable names for the output flowfiles (using hints from the
original flowfile name plus the resulting content date).

In terms of the pipeline, this all works very well moving on to other
processors as it does on its way to a database.

Not sure this reply helps, but it describes what I do.

Another very cool thing one gets almost for free with NiFi is an arc
(relationship) via which the original document can continue along a
different path, get stored, processed or whatever I want to do with it.
This is something that was less gracefully provided for in our previous
pipeline scheme.

Hope this helps.

Cheers

On 03/17/2016 08:09 AM, Uwe Geercken wrote:

Hello,

my first mailing here. I am a Java developer, using Apache Velocity, Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very much a data guy.

I have used Nifi for a while now and started yesterday of coding my first processor. I basically do it to widen my knowledge and learn something new.

I started with the idea of combining Apache Velocity - a template engine - with Nifi. So in comes a CSV file, it gets merged with a template containing formatting information and some placeholders (and some limited logic maybe) and out comes a new set of data, formatted differently. So it separates the processing logic from the formatting. One could create HTML, XML, Json or other text based formats from it. Easy to use and very efficient.

Now my question is: Should I rather implement the logic this way that I process a whole CSV file - which usually has multiple lines? That would be good for the user as he or she has to deal with only one processor doing the work. But the logic would be more specialized.

The other way around, I could code the processor to handle one row of the CSV file and the user will have to come up with a flow that divides the CSV file into multiple flowfiles before my processor can be used. That is not so specialized but it requires more preparation work from the user.

I tend to go the second way. Also because there is already a processor that will split a file into multiple flowfiles. But I wanted to hear your opinion of what is the best way to go. Do you have a recommendation for me? (Maybe the answer is to do both?!)

Thanks for sharing your thoughts.

Uwe

[GitHub] nifi pull request: NIFI-1337: Add Riemann Reporting Task

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/188#issuecomment-198151066
  
@rickysaltzer hey there.  I just realized that this does have a commit 
against it post review commentary.  My apologies for not engaging.  Do you 
happen to be using this consistently/etc.. and are getting good results/like 
how it is working?  I ask because the value of my own review will be fairly 
superficial and just want to make sure we have someone with good experience 
using it that is liking it (even if that is the author of it :))


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Re: Re: Processor additional documentation

2016-03-19 Thread dan bress

Uwe,
   No, the index.html is generated for you.  additionalDetails.html is your
responsibility only if you feel like the generated index.html doesn't fully
describe your processor.

   I would guess 80% of the included processors do not have
additionalDetails.html.  If you haven't browsed here [1] at examples of the
generated index.html and user supplied additionalDetails.html, it might
clear things up.

[1] https://nifi.apache.org/docs.html

Dan

On Fri, Mar 18, 2016 at 11:08 AM Uwe Geercken  wrote:

> Dan,
>
> but maybe I have a wrong understanding: do I have to create an index.html
> file? Currently I have only created an additionalDetails.html file.
>
> I will also try to reduce the html code to a minimum and see if it is a
> problem with my code.
>
> Bye,
>
> Uwe
>
> > Gesendet: Freitag, 18. März 2016 um 19:03 Uhr
> > Von: "dan bress" 
> > An: dev@nifi.apache.org
> > Betreff: Re: Re: Processor additional documentation
> >
> > Uwe,
> >No its not a problem to have both index.html and
> additionalDetails.html
> >  The NiFi framework generates nearly all of the documentation for your
> > processor for you.  It will generate information about the properties and
> > relationships your processor exposes to its users.  If you need to
> express
> > more about your processor, then that is where additionalDetails.html
> comes
> > into play.  For example, if your processor uses a custom query language.
> >
> > Generated index.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
> >
> > additionalDetails.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html
> >
> > On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken 
> wrote:
> >
> > > Bryan,
> > >
> > > all looks ok. I looked into the nifi-home/work/docs folder. There is
> > > nothing but a components folder. Inside there is a folder for my
> processor:
> > > com.datamelt.nifi.test.TemplateProcessor and inside the folder there
> is a
> > > file index.html and it contains the code of my additionalDetails.html
> file.
> > >
> > > when I open the file in the web browser it looks good. I looked at
> other
> > > index.html files and they look similar.
> > >
> > > but I noted that some folders have an inde.html file AND an
> > > additionalDetails.html file. maybe that is the problem?
> > >
> > > greetings,
> > >
> > > Uwe
> > >
> > >
> > >
> > > Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> > > Von: "Bryan Bende" 
> > > An: dev@nifi.apache.org
> > > Betreff: Re: Processor additional documentation
> > > Hi Uwe,
> > >
> > > Do you have the additionalDetails.html file in your processors jar
> project,
> > > under src/main/resources?
> > >
> > > Similar to this:
> > >
> > >
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
> > >
> > > The expected project structure is described here:
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> 
> > > <
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions%23MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> >
> > >
> > > If you think that part is setup correctly, can you check under
> > > nifi_home/work/docs and see if
> com.datamelt.nifi.test.TemplateProcessor is
> > > there?
> > >
> > > -Bryan
> > >
> > > On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken 
> > > wrote:
> > >
> > > >
> > > > Hello,
> > > >
> > > > I am writing my first processor. As described in the documentation, I
> > > have
> > > > added an HTML file to be used when the user selects "Usage":
> > > >
> > > > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> > > >
> > > > This is located in the root or the Processors nar file.
> > > >
> > > > The processor class is this:
> > > >
> > > > com/datamelt/nifi/test/TemplateProcessor.class
> > > >
> > > > The processor works, but selecting "Usage" won't show my HTML file.
> > > >
> > > > I understood that I write the HTML file and Nifi will picks it up
> when it
> > > > starts. Or is this not true?
> > > >
> > > > Thanks for feedback,
> > > >
> > > > Uwe
> > > >
> > >
> >
>

Re: Processor: User friendly vs system friendly design

Adam,

interesting and I agree. that sounds very good.

can you give me short tip of how to access attributes from code?

once I have something usable or for testing where would I publish it? just on 
my github site? or is there a place for sharing?

greetings

Uwe

Gesendet: Freitag, 18. März 2016 um 18:03 Uhr
Von: "Adam Taft" 
An: dev@nifi.apache.org
Betreff: Re: Processor: User friendly vs system friendly design
I'm probably on the far end of favoring composibility and processor reuse.
In this case, I would even go one step further and suggest that you're
talking about three separate operations:

1. Split a multi-line CSV input file into individual single line flowfiles.
2. Read columns from a single CSV line into flowfile attributes.
3. Pass flowfile attributes into the Velocity transform processor.

The point here, have you considered driving your Velocity template
transform using flowfile attributes as opposed to CSV? Flowfile attributes
are NIFI's lowest common data representation, many many processors create
attributes which would enable your Velocity processor to be used by more
than just CSV input.

Adam

On Fri, Mar 18, 2016 at 11:06 AM, Uwe Geercken  wrote:

>
> Hello,
>
> my first mailing here. I am a Java developer, using Apache Velocity,
> Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very much
> a data guy.
>
> I have used Nifi for a while now and started yesterday of coding my first
> processor. I basically do it to widen my knowledge and learn something new.
>
> I started with the idea of combining Apache Velocity - a template engine -
> with Nifi. So in comes a CSV file, it gets merged with a template
> containing formatting information and some placeholders (and some limited
> logic maybe) and out comes a new set of data, formatted differently. So it
> separates the processing logic from the formatting. One could create HTML,
> XML, Json or other text based formats from it. Easy to use and very
> efficient.
>
> Now my question is: Should I rather implement the logic this way that I
> process a whole CSV file - which usually has multiple lines? That would be
> good for the user as he or she has to deal with only one processor doing
> the work. But the logic would be more specialized.
>
> The other way around, I could code the processor to handle one row of the
> CSV file and the user will have to come up with a flow that divides the CSV
> file into multiple flowfiles before my processor can be used. That is not
> so specialized but it requires more preparation work from the user.
>
> I tend to go the second way. Also because there is already a processor
> that will split a file into multiple flowfiles. But I wanted to hear your
> opinion of what is the best way to go. Do you have a recommendation for me?
> (Maybe the answer is to do both?!)
>
> Thanks for sharing your thoughts.
>
> Uwe
>

[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...

2016-03-19 Thread markobean

Github user markobean commented on a diff in the pull request:

https://github.com/apache/nifi/pull/280#discussion_r56328470
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java
 ---
@@ -143,72 +199,82 @@ protected void init(final 
ProcessorInitializationContext context) {
 return properties;
 }
 
-private int readLines(final InputStream in, final int maxNumLines, 
final OutputStream out, final boolean keepAllNewLines) throws IOException {
+private int readLines(final InputStream in, final int maxNumLines, 
final long maxByteCount, final OutputStream out) throws IOException {
 int numLines = 0;
+long totalBytes = 0L;
 for (int i = 0; i < maxNumLines; i++) {
-final long bytes = countBytesToSplitPoint(in, out, 
keepAllNewLines || (i != maxNumLines - 1));
+final long bytes = countBytesToSplitPoint(in, out, totalBytes, 
maxByteCount);
+totalBytes += bytes;
 if (bytes <= 0) {
 return numLines;
 }
-
 numLines++;
+if (totalBytes >= maxByteCount && numLines > maxNumLines) {
--- End diff --

Agreed. "&& numLines > maxNumLines" condition can be removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

patch and pull request reviewing question

2016-03-19 Thread Tony Kurc

As I was reviewing a pull request, i realized that it actually was a bit
more mental effort in this instance to review a squashed and rebased set of
commits than if it was if I could just review the changes in a commit. Does
anyone object to me adding in the contributors guide something to the
extent of "Although you may be asked to rebase or squash your contribution
as part of the review process, don't feel the need to do so speculatively.
The committer working on merging the contribution may prefer to do these
types of operations as part of the merge process, and the history of your
patch or pull request may aid in the review process"

Tony

[GitHub] nifi pull request: NIFI-901: Fixed unit test to verify IP address ...

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/286

NIFI-901: Fixed unit test to verify IP address of Cassandra contact point

The test was checking the IP against getHostName(), but if there were a 
DNS/hosts entry for that IP, the test would fail. This fix changes the asserts 
to compare the expected IP against the actual IP.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-901

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #286


commit 395c84d27ba0c83f5d5fe7e1fb597c3ad9a8ed29
Author: Matt Burgess 
Date:   2016-03-17T17:46:53Z

NIFI-901: Fixed unit test to verify IP address of Cassandra contact point




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56410434
  
--- Diff: 
nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-ui/src/main/java/org/apache/nifi/update/attributes/api/RuleResource.java
 ---
@@ -16,66 +16,63 @@
  */
 package org.apache.nifi.update.attributes.api;
 
-import java.text.Collator;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Locale;
-import java.util.Set;
-import java.util.UUID;
-
-import javax.servlet.ServletContext;
-import javax.servlet.http.HttpServletRequest;
-import javax.ws.rs.Consumes;
-import javax.ws.rs.DELETE;
-import javax.ws.rs.DefaultValue;
-import javax.ws.rs.GET;
-import javax.ws.rs.POST;
-import javax.ws.rs.PUT;
-import javax.ws.rs.Path;
-import javax.ws.rs.PathParam;
-import javax.ws.rs.Produces;
-import javax.ws.rs.QueryParam;
-import javax.ws.rs.WebApplicationException;
-import javax.ws.rs.core.CacheControl;
-import javax.ws.rs.core.Context;
-import javax.ws.rs.core.MediaType;
-import javax.ws.rs.core.Response;
-import javax.ws.rs.core.Response.ResponseBuilder;
-import javax.ws.rs.core.UriBuilder;
-import javax.ws.rs.core.UriInfo;
-
+import com.sun.jersey.api.NotFoundException;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.nifi.update.attributes.Action;
 import org.apache.nifi.update.attributes.Condition;
 import org.apache.nifi.update.attributes.Criteria;
+import org.apache.nifi.update.attributes.FlowFilePolicy;
 import org.apache.nifi.update.attributes.Rule;
 import org.apache.nifi.update.attributes.UpdateAttributeModelFactory;
 import org.apache.nifi.update.attributes.dto.DtoFactory;
 import org.apache.nifi.update.attributes.dto.RuleDTO;
 import org.apache.nifi.update.attributes.entity.ActionEntity;
 import org.apache.nifi.update.attributes.entity.ConditionEntity;
+import org.apache.nifi.update.attributes.entity.EvaluationContextEntity;
 import org.apache.nifi.update.attributes.entity.RuleEntity;
 import org.apache.nifi.update.attributes.entity.RulesEntity;
 import org.apache.nifi.update.attributes.serde.CriteriaSerDe;
-import org.apache.nifi.web.InvalidRevisionException;
-import org.apache.nifi.web.Revision;
-import org.apache.commons.lang3.StringUtils;
-
-import com.sun.jersey.api.NotFoundException;
-
-import org.apache.nifi.update.attributes.FlowFilePolicy;
-import org.apache.nifi.update.attributes.entity.EvaluationContextEntity;
 import org.apache.nifi.web.ComponentDetails;
 import org.apache.nifi.web.HttpServletConfigurationRequestContext;
 import org.apache.nifi.web.HttpServletRequestContext;
+import org.apache.nifi.web.InvalidRevisionException;
 import org.apache.nifi.web.NiFiWebConfigurationContext;
 import org.apache.nifi.web.NiFiWebConfigurationRequestContext;
 import org.apache.nifi.web.NiFiWebRequestContext;
+import org.apache.nifi.web.Revision;
 import org.apache.nifi.web.UiExtensionType;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.DELETE;
+import javax.ws.rs.DefaultValue;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.PUT;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.QueryParam;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.CacheControl;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.Response.ResponseBuilder;
+import javax.ws.rs.core.UriBuilder;
+import javax.ws.rs.core.UriInfo;
+import java.text.Collator;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import java.util.UUID;
--- End diff --

From what I can tell, the only changes to this file are to the formatting 
of the imports. In general extraneous modifications should be limited.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1488 Added hbase kerb auth with ugi

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/253#issuecomment-198146996
  
Hello @lordjc This ended up being merged and incorporated but the 'This 
closes #253' was not added to the commit log so the close hook didn't fire.  
Can you please close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56420919
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

Right.  So i think the short version is in the UI now.  But you want the 
stacktrace too.  We should make that available in the logs.  And this class is 
what controls that I believe 
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-logging-utils/src/main/java/org/apache/nifi/logging/NiFiLog.java
   You'll see it has some ifDebugEnabled guards.  We should probably just toss 
those out.  @markap14 what say you?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1605 Adjust documentation and resources to...

Github user mattyb149 commented on the pull request:

https://github.com/apache/nifi/pull/263#issuecomment-198165221
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1575: Add QueryDatabaseTable processor

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/261#issuecomment-198147766
  
@mattyb149 believe this PR is closed but apparently the commit message 
didn't reflect it.  Can you please close it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...

2016-03-19 Thread jskora

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/280#issuecomment-197388774
  
I think the issues are all covered now and tests have been expanded to get 
all critical logic and edge cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...

2016-03-19 Thread jskora

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/252#issuecomment-198427593
  
I don't love all the attribute names, I might have come up with a cleaner 
taxonomy myself.  But, letting Tika handle it leverages the work done on 
another active Apache project by people familiar with the formats.  
Incorporating Tika was a little harder than writing custom code to only handle 
WAV files would have been, but it provided significantly more functionality.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56496662
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/NiFiAuthenticationFilter.java
 ---
@@ -64,7 +64,7 @@ public void doFilter(final ServletRequest request, final 
ServletResponse respons
 
 }
 
-private boolean requiresAuthentication(final HttpServletRequest 
request) {
--- End diff --

This can stay private since there is no need to override anymore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

2016-03-19 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/271#issuecomment-198245248
  
@olegz Sorry for the delay in response, I shall look into that during next 
week.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: patch and pull request reviewing question

2016-03-19 Thread Oleg Zhurakousky

+1 here as well
My apologies Tony

Sent from my iPhone

> On Mar 17, 2016, at 21:42, Joe Witt  wrote:
> 
> +1
> 
>> On Thu, Mar 17, 2016 at 9:37 PM, Tony Kurc  wrote:
>> As I was reviewing a pull request, i realized that it actually was a bit
>> more mental effort in this instance to review a squashed and rebased set of
>> commits than if it was if I could just review the changes in a commit. Does
>> anyone object to me adding in the contributors guide something to the
>> extent of "Although you may be asked to rebase or squash your contribution
>> as part of the review process, don't feel the need to do so speculatively.
>> The committer working on merging the contribution may prefer to do these
>> types of operations as part of the merge process, and the history of your
>> patch or pull request may aid in the review process"
>> 
>> Tony
>

Re: Re: Processor: User friendly vs system friendly design

2016-03-19 Thread Adam Taft

Uwe,

I'll take a look at your code sometime soon.  However, just to point you in
the direction, I'd suggest extracting your single line CSV data into
flowfile attributes named as you've demonstrated.  i.e.  create a processor
which reads each CSV column as a flowfile attribute, using a configured
naming convention.

For example, using "column" as your prefix with your example input, you'd
end up with a single flowfile with attributes like:

column0 = Peterson
column1 = Jenny
column2 = New York
column3 = USA

Flowfile attributes are effectively a Map.  So in your
Velocity processor, you would pass the Map of flowfile attributes to the
template engine and record the results to the flowfile content.

Using SplitText seems correct up front (though like you said, you lose the
CSV header line).  You'd need two additional processors, from my
perspective:

(input) -> SplitText -> ExtractCSVColumns -> ApplyVelocityTemplate ->
(output)

It's the "split row into fields and merge with template" that we would
want to separate into two processors instead of one.

You're very much on the right track, I believe.  If the above doesn't help,
I'll try and jump in on a code example when I can.

Adam


On Fri, Mar 18, 2016 at 5:04 PM, Uwe Geercken  wrote:

> Adam,
>
> I don't see an obvious way for your suggestion of "Read columns from a
> single CSV line into flowfile attributes." - I would need your advice how I
> can achieve it.
>
> Thinking about it in more detail, I have following issues:
> - the incomming flowfile may have many columns. so adding the columns
> manually as attributes with UpdateAttributes is not feasible
> - I have setup a flow where I use SplitText to divide the flowfile into
> multiple flowfiles, so there won't be a header row I can use to get the
> column names. So I think I can only use abstract column names plus a
> running number. e.g. column0, column1, etc.
>
> So for the moment I have coded the processor like described below. At the
> moment I am still "thinking in CSV" but I will check it with other formats
> later. The user can steer follwoing settings: path where the template is
> stored, name of the template file, the label for the columns (I call it
> prefix) and the separator based on which the split of the row is done.
>
> Example Flowfile content (user has chosen "comma" as separator:
>
> Peterson, Jenny, New York, USA
>
> Example template (user has chosen "column" as the prefix):
>
> {
> "name": "$column0",
> "first": "$column1",
> "city": "$column2",
> "country": "$column3"
> }
>
> Example flow:
>
> GetFile: Get CSV File >> SplitText : split into multiple flowfiles, one
> per row >> TemplateProcessor:
> 
> split row into fields and merge with template >> MergeContent: merge
> flowfiles into one >> PutFile: put the file to the filesystem
>
> Example result:
>
> {
> "name": "Peterson",
> "first": "Jenny",
> "city": "New York",
> "country": "USA"
>  }
>
> I will test the processor now for larger files, empty files and other
> exceptions. If you are interested the code is here:
>
> https://github.com/uwegeercken/nifi_processors
>
> Greetings,
>
> Uwe
>
>
>
> > Gesendet: Freitag, 18. März 2016 um 18:58 Uhr
> > Von: "Adam Taft" 
> > An: dev@nifi.apache.org
> > Betreff: Re: Processor: User friendly vs system friendly design
> >
> > Uwe,
> >
> > The Developer Guide[1] and Contributor Guide[2] are pretty solid.  The
> > Developer Guide has a section dealing with reading & writing flowfile
> > attributes.  Please check these out, and then if you have any specific
> > questions, please feel free to reply.
> >
> > For inclusion in NIFI directly, you'd want to create a NIFI Jira ticket
> > mentioning the new feature, and then fork the NIFI project in Github and
> > send a Pull Request referencing the ticket.  However, if you just want
> some
> > feedback on suitability and consideration for inclusion, using your own
> > personal Github project and sending a link would be fine.
> >
> > Having a template conversion processor would be a nice addition.  Making
> it
> > generic to support Velocity, FreeMarker, and others might be really nice.
> > Extra bonus points for Markdown or Asciidoc transforms as well (but these
> > might be too separate of a use case).
> >
> > Hope this helps.
> >
> > Adam
> >
> > [1]  http://nifi.apache.org/developer-guide.html
> >
> > [2]  https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
> >
> >
> >
> >
> > On Fri, Mar 18, 2016 at 1:36 PM, Uwe Geercken 
> wrote:
> >
> > > Adam,
> > >
> > > interesting and I agree. that sounds very good.
> > >
> > > can you give me short tip of how to access attributes from code?
> > >
> > > once I have something usable or for testing where would I publish it?
> just
> > > on my github site? or is there a place for sharing?
> > >
> > > greetings
> > >
> > > Uwe
> > >
> > >
> > >
> > >

Processor: User friendly vs system friendly design


Hello,

my first mailing here. I am a Java developer, using Apache Velocity, Drill, 
Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very much a data guy.

I have used Nifi for a while now and started yesterday of coding my first 
processor. I basically do it to widen my knowledge and learn something new.

I started with the idea of combining Apache Velocity - a template engine - with 
Nifi. So in comes a CSV file, it gets merged with a template containing 
formatting information and some placeholders (and some limited logic maybe) and 
out comes a new set of data, formatted differently. So it separates the 
processing logic from the formatting. One could create HTML, XML, Json or other 
text based formats from it. Easy to use and very efficient.

Now my question is: Should I rather implement the logic this way that I process 
a whole CSV file - which usually has multiple lines? That would be good for the 
user as he or she has to deal with only one processor doing the work. But the 
logic would be more specialized.

The other way around, I could code the processor to handle one row of the CSV 
file and the user will have to come up with a flow that divides the CSV file 
into multiple flowfiles before my processor can be used. That is not so 
specialized but it requires more preparation work from the user.

I tend to go the second way. Also because there is already a processor that 
will split a file into multiple flowfiles. But I wanted to hear your opinion of 
what is the best way to go. Do you have a recommendation for me? (Maybe the 
answer is to do both?!)

Thanks for sharing your thoughts.

Uwe

Re: patch and pull request reviewing question

2016-03-19 Thread Tony Kurc

Aldrin,
Very cool
On Mar 17, 2016 10:21 PM, "Aldrin Piri"  wrote:

> Sounds good to here as well.
>
> Given the increasing number of contributions, which predominantly seem to
> arrive via GitHub PR, I also created NIFI-1615 [1] a little while ago to
> make a GitHub PR template [1].  Would like to distill the contributor guide
> down into a few easy steps for folks to check out and make apprised of
> before they commit the PR that I think can help our reviewers and
> infrastructure.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1615
> [2] https://github.com/blog/2111-issue-and-pull-request-templates
>
> On Thu, Mar 17, 2016 at 9:55 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
> > +1 here as well
> > My apologies Tony
> >
> > Sent from my iPhone
> >
> > > On Mar 17, 2016, at 21:42, Joe Witt  wrote:
> > >
> > > +1
> > >
> > >> On Thu, Mar 17, 2016 at 9:37 PM, Tony Kurc  wrote:
> > >> As I was reviewing a pull request, i realized that it actually was a
> bit
> > >> more mental effort in this instance to review a squashed and rebased
> > set of
> > >> commits than if it was if I could just review the changes in a commit.
> > Does
> > >> anyone object to me adding in the contributors guide something to the
> > >> extent of "Although you may be asked to rebase or squash your
> > contribution
> > >> as part of the review process, don't feel the need to do so
> > speculatively.
> > >> The committer working on merging the contribution may prefer to do
> these
> > >> types of operations as part of the merge process, and the history of
> > your
> > >> patch or pull request may aid in the review process"
> > >>
> > >> Tony
> > >
> >
>

Re: Re: Processor: User friendly vs system friendly design

This sounds pretty cool and definitely think you're both on a good
track design-wise.

On Fri, Mar 18, 2016 at 2:04 PM, Uwe Geercken  wrote:
> Adam,
>
> thanks again.
>
> I didn't know about the contributors guide - I was always looking in the docs 
> inside Nifi and there is a reference to the developer guide only.
>
> I will try to make a good processor for velocity first. The next step would 
> then be to include also freemarker. I will try to keep that in mind during 
> design and coding. I don't know anything about Markdown or Asciidoc. So I 
> will have to have a look first.
>
> Regards,
>
> Uwe
>
>
>
>> Gesendet: Freitag, 18. März 2016 um 18:58 Uhr
>> Von: "Adam Taft" 
>> An: dev@nifi.apache.org
>> Betreff: Re: Processor: User friendly vs system friendly design
>>
>> Uwe,
>>
>> The Developer Guide[1] and Contributor Guide[2] are pretty solid.  The
>> Developer Guide has a section dealing with reading & writing flowfile
>> attributes.  Please check these out, and then if you have any specific
>> questions, please feel free to reply.
>>
>> For inclusion in NIFI directly, you'd want to create a NIFI Jira ticket
>> mentioning the new feature, and then fork the NIFI project in Github and
>> send a Pull Request referencing the ticket.  However, if you just want some
>> feedback on suitability and consideration for inclusion, using your own
>> personal Github project and sending a link would be fine.
>>
>> Having a template conversion processor would be a nice addition.  Making it
>> generic to support Velocity, FreeMarker, and others might be really nice.
>> Extra bonus points for Markdown or Asciidoc transforms as well (but these
>> might be too separate of a use case).
>>
>> Hope this helps.
>>
>> Adam
>>
>> [1]  http://nifi.apache.org/developer-guide.html
>>
>> [2]  https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
>>
>>
>>
>>
>> On Fri, Mar 18, 2016 at 1:36 PM, Uwe Geercken  wrote:
>>
>> > Adam,
>> >
>> > interesting and I agree. that sounds very good.
>> >
>> > can you give me short tip of how to access attributes from code?
>> >
>> > once I have something usable or for testing where would I publish it? just
>> > on my github site? or is there a place for sharing?
>> >
>> > greetings
>> >
>> > Uwe
>> >
>> >
>> >
>> > Gesendet: Freitag, 18. März 2016 um 18:03 Uhr
>> > Von: "Adam Taft" 
>> > An: dev@nifi.apache.org
>> > Betreff: Re: Processor: User friendly vs system friendly design
>> > I'm probably on the far end of favoring composibility and processor reuse.
>> > In this case, I would even go one step further and suggest that you're
>> > talking about three separate operations:
>> >
>> > 1. Split a multi-line CSV input file into individual single line flowfiles.
>> > 2. Read columns from a single CSV line into flowfile attributes.
>> > 3. Pass flowfile attributes into the Velocity transform processor.
>> >
>> > The point here, have you considered driving your Velocity template
>> > transform using flowfile attributes as opposed to CSV? Flowfile attributes
>> > are NIFI's lowest common data representation, many many processors create
>> > attributes which would enable your Velocity processor to be used by more
>> > than just CSV input.
>> >
>> > Adam
>> >
>> >
>> >
>> > On Fri, Mar 18, 2016 at 11:06 AM, Uwe Geercken 
>> > wrote:
>> >
>> > >
>> > > Hello,
>> > >
>> > > my first mailing here. I am a Java developer, using Apache Velocity,
>> > > Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very
>> > much
>> > > a data guy.
>> > >
>> > > I have used Nifi for a while now and started yesterday of coding my first
>> > > processor. I basically do it to widen my knowledge and learn something
>> > new.
>> > >
>> > > I started with the idea of combining Apache Velocity - a template engine
>> > -
>> > > with Nifi. So in comes a CSV file, it gets merged with a template
>> > > containing formatting information and some placeholders (and some limited
>> > > logic maybe) and out comes a new set of data, formatted differently. So
>> > it
>> > > separates the processing logic from the formatting. One could create
>> > HTML,
>> > > XML, Json or other text based formats from it. Easy to use and very
>> > > efficient.
>> > >
>> > > Now my question is: Should I rather implement the logic this way that I
>> > > process a whole CSV file - which usually has multiple lines? That would
>> > be
>> > > good for the user as he or she has to deal with only one processor doing
>> > > the work. But the logic would be more specialized.
>> > >
>> > > The other way around, I could code the processor to handle one row of the
>> > > CSV file and the user will have to come up with a flow that divides the
>> > CSV
>> > > file into multiple flowfiles before my processor can be used. That is not
>> > > so specialized but it requires more preparation work from the user.
>> > >
>> > > I tend to go the

[GitHub] nifi pull request: NIFI-856 - Implements experimental ListenLumber...

2016-03-19 Thread trixpan

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/289

NIFI-856 - Implements experimental ListenLumberjack Processor

ListenLumberjack processor should potentially allow NiFi to act as a drop 
in replacement 
on pipelines using logstash-forwarder and Logstash's Lumberjack output to 
move data 
created by distributed systems.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-856

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #289


commit c92d5f618feb9115f71293c302f8efc1849ce3dd
Author: Andre F de Miranda 
Date:   2016-03-18T13:21:45Z

NIFI-856 - Implements initial and experimental ListenLumberjack Processor, 
allowing NiFi to
   replace Logstash as a pipeline for logstash-forwarder and 
Logstash's Lumberjack
 output

commit d1d9a4d41e0af5d507df3d1a5199cfc383b856f6
Author: Andre F de Miranda 
Date:   2016-03-18T13:26:53Z

Remove .gitignore and README.md left behind during transtion from 
standalone git




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Rollbacks

Devin,

Good stuff.  Unless the api call to change the content completes
normally the transformation effectively did not happen.  Take a look
at CompressContent [1] for an example.

It reads in the original content and streams out new content.  If that
process of transformation fails it routes the flow file to failure.
What gets routed is the original representation of the data.

Go ahead and give a really simple form of your case a try.  Do
something like always throw an exception in your callback to test it
and then you can validate the behavior meets your needs.  In fact,
that could be a great test case.

[1] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/CompressContent.java

Thanks
Joe

On Thu, Mar 17, 2016 at 12:36 PM, Devin Fisher
 wrote:
> Thanks for the detailed response. The details about the batching were very
> helpful. I've changed my processors to take advantage of the batching that
> framework provides since my use case fits what you described to a tee.
>
> But I have a question about the failure relationship. Maybe a concrete
> example might help. So let's say I have an XML document that I'm processing
> with a Processor that converts XML to JSON. This Processor makes use an SAX
> parser so it doesn't have to read the whole XML document into memory. The
> SAX parser read from an input stream provided by Nifi and calls SAX
> callbacks base on the XML it encounters. These callbacks write the JSON to
> an output stream provided by Nifi. So as soon as the Processor starts it
> has written to the current Session. If towards the end of the XML document
> it hits a syntax error, let say there is a text element with an '&' that
> has not been encoded as an entity. The SAX parser will fail and will not be
> able to complete successfully. (the parser is very strict) But the
> processor will have already written most of the resulting JSON but not all.
> At this point if I transfer it the failed relationship without rolling back
> what will I get? Will I get the incomplete JSON or will I get the invalid
> but complete XML? I'm guessing that it will be half written JSON. What I
> would think you would want is the complete XML so that a processor that
> fixes the '&' entity issue could handle the issue and route it back. Or a
> different processor could log it or something else.
>
> Anyway, I hope that example explains the confusion that I still have about
> handling error cases. Rolling back here would not be helpful from what I
> understand because it will just get processed again by the same Processor
> with the SAX parser that can't handle the '&' over and over again. The
> penalty will make sure that other documents get process before it is tried
> again but that one malformed XML document will never progress because it
> will always fail in the same way each time and be rollback again.
>
> Again thanks for the reply. The info was really useful and helped my
> understanding.
>
> Devin
>
> On Tue, Mar 15, 2016 at 7:27 PM, Joe Witt  wrote:
>
>> Devin,
>>
>> So the session follows the unit of work pattern that Martin Fowler
>> describes [1].  The idea here is that whether you access one flow file
>> or many flow files or anything in between you are doing some session
>> of work.  You can commit all the things you do on that session or
>> rollback all the things you do on that session.
>>
>> This pattern/concept provides a nice and safe environment in which the
>> contract between the developer and framework is well understood.
>> Generally, rollback is only necessary when some unplanned or
>> unexpected exception occurs and it is largely there so that the
>> framework can ensure all things are returned to a safe state.  It can
>> also do things like penalize that processor/extension so that if it is
>> some programming error that it will reduce its impact on the system
>> overall.
>>
>> So, with that said there are two ways to think about your case.  It
>> appears you are doing your own batching and probably this is for
>> higher throughput and also it appears you'd really like to treat each
>> flow file independently in terms of logic/handling.
>>
>> This is precisely why in addition to this nice clean unit of work
>> pattern we also support automated session batching (this is what
>> Andrew was referring to).  In this mode you can add an annotation to
>> your processor called @SupportsBatching which signals to the framework
>> that it may attempt to automatically combine subsequent calls to
>> commit into small batches and commit them in a single batch.  In this
>> way you can build your processor in a very simple single flow file
>> sort of manner and call commit.  But the framework will combine a
>> series of commits in a very small time window together to get higher
>> throughput.  In the UI a user can signal their willingness to let

[GitHub] nifi pull request: Nifi 1274

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56425560
  
--- Diff: 
nifi-nar-bundles/nifi-kerberos-iaa-providers-bundle/nifi-kerberos-iaa-providers/src/main/java/org/apache/nifi/kerberos/KerberosProvider.java
 ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.kerberos;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.authentication.AuthenticationResponse;
+import org.apache.nifi.authentication.LoginCredentials;
+import org.apache.nifi.authentication.LoginIdentityProvider;
+import 
org.apache.nifi.authentication.LoginIdentityProviderConfigurationContext;
+import 
org.apache.nifi.authentication.LoginIdentityProviderInitializationContext;
+import org.apache.nifi.authentication.exception.IdentityAccessException;
+import 
org.apache.nifi.authentication.exception.InvalidLoginCredentialsException;
+import org.apache.nifi.authorization.exception.ProviderCreationException;
+import 
org.apache.nifi.authorization.exception.ProviderDestructionException;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.core.AuthenticationException;
+import 
org.springframework.security.kerberos.authentication.KerberosAuthenticationProvider;
+import 
org.springframework.security.kerberos.authentication.sun.SunJaasKerberosClient;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Kerberos-based implementation of a login identity provider.
+ */
+public class KerberosProvider implements LoginIdentityProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(KerberosProvider.class);
+
+private KerberosAuthenticationProvider provider;
+private String issuer;
+private long expiration;
+
+@Override
+public final void initialize(final 
LoginIdentityProviderInitializationContext initializationContext) throws 
ProviderCreationException {
+this.issuer = getClass().getSimpleName();
+}
+
+@Override
+public final void onConfigured(final 
LoginIdentityProviderConfigurationContext configurationContext) throws 
ProviderCreationException {
+final String rawExpiration = 
configurationContext.getProperty("Authentication Expiration");
+if (StringUtils.isBlank(rawExpiration)) {
+throw new ProviderCreationException("The Authentication 
Expiration must be specified.");
+}
+
+try {
+expiration = FormatUtils.getTimeDuration(rawExpiration, 
TimeUnit.MILLISECONDS);
+} catch (final IllegalArgumentException iae) {
+throw new ProviderCreationException(String.format("The 
Expiration Duration '%s' is not a valid time duration", rawExpiration));
+}
+
+provider = new KerberosAuthenticationProvider();
+SunJaasKerberosClient client = new SunJaasKerberosClient();
+client.setDebug(true);
+provider.setKerberosClient(client);
+provider.setUserDetailsService(new KerberosUserDetailsService());
+}
+
+@Override
+public final AuthenticationResponse authenticate(final 
LoginCredentials credentials) throws InvalidLoginCredentialsException, 
IdentityAccessException {
+if (provider == null) {
+throw new IdentityAccessException("The Kerberos authentication 
provider is not initialized.");
+}
+
+try {
+// TODO: Remove debug statements
+logger.info("[REMOVE] Attempting to authenticate Kerberos user 
{} with password {}", credentials.getUsername(), credentials.getPassword());
+
+// Perform the authentication
+final

Aw: Re: Processor: User friendly vs system friendly design

Adam,

thanks again.

I didn't know about the contributors guide - I was always looking in the docs 
inside Nifi and there is a reference to the developer guide only.

I will try to make a good processor for velocity first. The next step would 
then be to include also freemarker. I will try to keep that in mind during 
design and coding. I don't know anything about Markdown or Asciidoc. So I will 
have to have a look first.

Regards,

Uwe



> Gesendet: Freitag, 18. März 2016 um 18:58 Uhr
> Von: "Adam Taft" 
> An: dev@nifi.apache.org
> Betreff: Re: Processor: User friendly vs system friendly design
>
> Uwe,
> 
> The Developer Guide[1] and Contributor Guide[2] are pretty solid.  The
> Developer Guide has a section dealing with reading & writing flowfile
> attributes.  Please check these out, and then if you have any specific
> questions, please feel free to reply.
> 
> For inclusion in NIFI directly, you'd want to create a NIFI Jira ticket
> mentioning the new feature, and then fork the NIFI project in Github and
> send a Pull Request referencing the ticket.  However, if you just want some
> feedback on suitability and consideration for inclusion, using your own
> personal Github project and sending a link would be fine.
> 
> Having a template conversion processor would be a nice addition.  Making it
> generic to support Velocity, FreeMarker, and others might be really nice.
> Extra bonus points for Markdown or Asciidoc transforms as well (but these
> might be too separate of a use case).
> 
> Hope this helps.
> 
> Adam
> 
> [1]  http://nifi.apache.org/developer-guide.html
> 
> [2]  https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
> 
> 
> 
> 
> On Fri, Mar 18, 2016 at 1:36 PM, Uwe Geercken  wrote:
> 
> > Adam,
> >
> > interesting and I agree. that sounds very good.
> >
> > can you give me short tip of how to access attributes from code?
> >
> > once I have something usable or for testing where would I publish it? just
> > on my github site? or is there a place for sharing?
> >
> > greetings
> >
> > Uwe
> >
> >
> >
> > Gesendet: Freitag, 18. März 2016 um 18:03 Uhr
> > Von: "Adam Taft" 
> > An: dev@nifi.apache.org
> > Betreff: Re: Processor: User friendly vs system friendly design
> > I'm probably on the far end of favoring composibility and processor reuse.
> > In this case, I would even go one step further and suggest that you're
> > talking about three separate operations:
> >
> > 1. Split a multi-line CSV input file into individual single line flowfiles.
> > 2. Read columns from a single CSV line into flowfile attributes.
> > 3. Pass flowfile attributes into the Velocity transform processor.
> >
> > The point here, have you considered driving your Velocity template
> > transform using flowfile attributes as opposed to CSV? Flowfile attributes
> > are NIFI's lowest common data representation, many many processors create
> > attributes which would enable your Velocity processor to be used by more
> > than just CSV input.
> >
> > Adam
> >
> >
> >
> > On Fri, Mar 18, 2016 at 11:06 AM, Uwe Geercken 
> > wrote:
> >
> > >
> > > Hello,
> > >
> > > my first mailing here. I am a Java developer, using Apache Velocity,
> > > Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very
> > much
> > > a data guy.
> > >
> > > I have used Nifi for a while now and started yesterday of coding my first
> > > processor. I basically do it to widen my knowledge and learn something
> > new.
> > >
> > > I started with the idea of combining Apache Velocity - a template engine
> > -
> > > with Nifi. So in comes a CSV file, it gets merged with a template
> > > containing formatting information and some placeholders (and some limited
> > > logic maybe) and out comes a new set of data, formatted differently. So
> > it
> > > separates the processing logic from the formatting. One could create
> > HTML,
> > > XML, Json or other text based formats from it. Easy to use and very
> > > efficient.
> > >
> > > Now my question is: Should I rather implement the logic this way that I
> > > process a whole CSV file - which usually has multiple lines? That would
> > be
> > > good for the user as he or she has to deal with only one processor doing
> > > the work. But the logic would be more specialized.
> > >
> > > The other way around, I could code the processor to handle one row of the
> > > CSV file and the user will have to come up with a flow that divides the
> > CSV
> > > file into multiple flowfiles before my processor can be used. That is not
> > > so specialized but it requires more preparation work from the user.
> > >
> > > I tend to go the second way. Also because there is already a processor
> > > that will split a file into multiple flowfiles. But I wanted to hear your
> > > opinion of what is the best way to go. Do you have a recommendation for
> > me?
> > > (Maybe the answer is to do both?!)
> > >
> > > Thanks for sharing

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56420228
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

that is actually the current behavior/intent.  I think the change needed is 
to simply get rid of the 'if debug enabled' stuff we did in the logger and 
instead just always log the stacktrace when it is there.  This is what Adam 
Taft had advocated for previously as I recall.  I now see why he was saying it. 
 So yeah in the UI it should be a short and ideally meaningful message (not 
always possible) and if a throwable shows up we should put the whole stack 
trace in the logs.

The current idea that a user will go in an do debug enabledjust has 
proven to be a failed experiment in my view and as someone who advocated for 
that I now think i was wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/284

Nifi 1274

Adds support for Kerberos single sign-on via SPNEGO negotiation with 
fallback to Kerberos LoginIdentityProvider (similar to LDAP credential login). 

Initial PR for review. I am adding documentation and will rebase after 
feedback is provided. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-1274

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/284.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #284


commit 522787dd5f6a11a0e6b8e97266106dfc66d9c09e
Author: Adam Lamar 
Date:   2016-02-21T06:12:56Z

NIFI-1180: Modify PutS3Object to enable encryption

commit f3395b2757305dd516c34f2d6573ad125760ecab
Author: Andy LoPresto 
Date:   2016-02-26T00:21:34Z

Merge branch 'NIFI-1180' of https://github.com/adamonduty/nifi

commit 50fc5b6bc256aa160bbe663f059eca6f71e49bb9
Author: Andy LoPresto 
Date:   2016-03-01T18:26:47Z

Merge branch 'master' of https://github.com/apache/nifi

commit 68c86ad9c86363eba60afae19711a4400888bfc4
Author: Andy LoPresto 
Date:   2016-03-11T23:11:19Z

NIFI-1274 Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos 
plugin.

commit 2fb038d9c2d5d470e6e007d5131d6130f6ccae35
Author: Andy LoPresto 
Date:   2016-03-12T06:05:02Z

NIFI-1274 Added kerberos module dependencies to nifi/pom.xml and 
nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.

commit 513a0f0bcd3e7c8808194106c45a6f3e7098c6d3
Author: Andy LoPresto 
Date:   2016-03-12T06:09:40Z

NIFI-1274 Added working configuration files to test/resources in kerberos 
module to document necessary config. This version requires the user to enter 
their Kerberos username (without realm) and password into the NiFi login screen 
and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a 
secure instance.

commit b97f754d6c4bc65d430d255231e8609163bfab7c
Author: Andy LoPresto 
Date:   2016-03-15T05:28:48Z

NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO 
authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location 
for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without 
dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to 
pom.xml.

commit 0733574a0d1cd72b7663eadd0fd89c9297731003
Author: Andy LoPresto 
Date:   2016-03-15T18:40:48Z

NIFI-1274 Added temporary solution for Rules Resource access via Kerberos 
ticket.

commit 6670b8bf33bf8018366972217dde8a0956e88194
Author: Andy LoPresto 
Date:   2016-03-15T19:13:53Z

NIFI-1274 Removed temporary solution for Rules Resource access via Kerberos 
ticket.

commit 794b9be508d1fe2042c70ba27a775eee0f4aab32
Author: Andy LoPresto 
Date:   2016-03-15T19:19:25Z

NIFI-1274 Renamed Kerberos discovery method to be explicit about service 
vs. credential login.

commit 22ff40b58994b2cdc0bedcf03d85ec744a37dfd5
Author: Andy LoPresto 
Date:   2016-03-15T19:50:38Z

NIFI-1274 Added check to only instantiate beans when Kerberos enabled to 
allow access control integration tests to pass.

commit ff50eaf0ffc2355e0794135b1aa6610ee562bf3f
Author: Andy LoPresto 
Date:   2016-03-16T02:33:24Z

NIFI-1274 Kerberos SPNEGO works without additional filter (new entry 
endpoint accepts Kerberos ticket in Authorization header and returns JWT so the 
rest of the application functions the same as LDAP).

commit 49d8063bc69e0c73d4ac039e7d0047f865e17fc0
Author: Andy LoPresto 
Date:   2016-03-16T04:01:51Z

NIFI-1274 Fixed canvas call to only attempt Kerberos login if JWT not 
present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in 
nifi-web-security-context.xml for consistency.

commit 2a33ded7c896f73d4c1a203f115e0e6d21f432f4
Author: Andy LoPresto 
Date:   2016-03-16T17:33:36Z

NIFI-1274 Added NiFi properties for Kerberos SSO.

commit 710318a43c7e0c05dc182ba9c7f406bb4eaa7722
Author: Andy LoPresto

[GitHub] nifi pull request: Nifi 1274

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56511388
  
--- Diff: 
nifi-nar-bundles/nifi-kerberos-iaa-providers-bundle/nifi-kerberos-iaa-providers/src/test/resources/nifi.properties
 ---
@@ -0,0 +1,185 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Core Properties #
+nifi.version=0.6.0-SNAPSHOT
+nifi.flow.configuration.file=./conf/flow.xml.gz
+nifi.flow.configuration.archive.dir=./conf/archive/
+nifi.flowcontroller.autoResumeState=true
+nifi.flowcontroller.graceful.shutdown.period=10 sec
+nifi.flowservice.writedelay.interval=500 ms
+nifi.administrative.yield.duration=30 sec
+# If a component has no work to do (is "bored"), how long should we wait 
before checking again for work?
+nifi.bored.yield.duration=10 millis
+
+nifi.authority.provider.configuration.file=./conf/authority-providers.xml

+nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
+nifi.templates.directory=./conf/templates
+nifi.ui.banner.text=
+nifi.ui.autorefresh.interval=30 sec
+nifi.nar.library.directory=./lib
+nifi.nar.working.directory=./work/nar/
+nifi.documentation.working.directory=./work/docs/components
+
+
+# State Management #
+
+nifi.state.management.configuration.file=./conf/state-management.xml
+# The ID of the local state provider
+nifi.state.management.provider.local=local-provider
+# The ID of the cluster-wide state provider. This will be ignored if NiFi 
is not clustered but must be populated if running in a cluster.
+nifi.state.management.provider.cluster=zk-provider
+# Specifies whether or not this instance of NiFi should run an embedded 
ZooKeeper server
+nifi.state.management.embedded.zookeeper.start=false
+# Properties file that provides the ZooKeeper properties to use if 
 is set to true

+nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
+
+
+# H2 Settings
+nifi.database.directory=./database_repository
+nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
+
+# FlowFile Repository

+nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
+nifi.flowfile.repository.directory=./flowfile_repository
+nifi.flowfile.repository.partitions=256
+nifi.flowfile.repository.checkpoint.interval=2 mins
+nifi.flowfile.repository.always.sync=false
+

+nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
+nifi.queue.swap.threshold=2
+nifi.swap.in.period=5 sec
+nifi.swap.in.threads=1
+nifi.swap.out.period=5 sec
+nifi.swap.out.threads=4
+
+# Content Repository

+nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
+nifi.content.claim.max.appendable.size=10 MB
+nifi.content.claim.max.flow.files=100
+nifi.content.repository.directory.default=./content_repository
+nifi.content.repository.archive.max.retention.period=12 hours
+nifi.content.repository.archive.max.usage.percentage=50%
+nifi.content.repository.archive.enabled=true
+nifi.content.repository.always.sync=false
+nifi.content.viewer.url=/nifi-content-viewer/
+
+# Provenance Repository Properties

+nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
+
+# Persistent Provenance Repository Properties
+nifi.provenance.repository.directory.default=./provenance_repository
+nifi.provenance.repository.max.storage.time=24 hours
+nifi.provenance.repository.max.storage.size=1 GB
+nifi.provenance.repository.rollover.time=30 secs
+nifi.provenance.repository.rollover.size=100 MB
+nifi.provenance.repository.query.threads=2
+nifi.provenance.repository.index.threads=1
+nifi.provenance.repository.compress.on.rollover=true
+nifi.provenance.repository.always.sync=false
+nifi.provenance.repository.journal.count=16
+#

[GitHub] nifi pull request: Nifi 1274

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/284#discussion_r56411928
  
--- Diff: 
nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-ui/src/main/java/org/apache/nifi/update/attributes/api/RuleResource.java
 ---
@@ -16,66 +16,63 @@
  */
 package org.apache.nifi.update.attributes.api;
 
-import java.text.Collator;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Locale;
-import java.util.Set;
-import java.util.UUID;
-
-import javax.servlet.ServletContext;
-import javax.servlet.http.HttpServletRequest;
-import javax.ws.rs.Consumes;
-import javax.ws.rs.DELETE;
-import javax.ws.rs.DefaultValue;
-import javax.ws.rs.GET;
-import javax.ws.rs.POST;
-import javax.ws.rs.PUT;
-import javax.ws.rs.Path;
-import javax.ws.rs.PathParam;
-import javax.ws.rs.Produces;
-import javax.ws.rs.QueryParam;
-import javax.ws.rs.WebApplicationException;
-import javax.ws.rs.core.CacheControl;
-import javax.ws.rs.core.Context;
-import javax.ws.rs.core.MediaType;
-import javax.ws.rs.core.Response;
-import javax.ws.rs.core.Response.ResponseBuilder;
-import javax.ws.rs.core.UriBuilder;
-import javax.ws.rs.core.UriInfo;
-
+import com.sun.jersey.api.NotFoundException;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.nifi.update.attributes.Action;
 import org.apache.nifi.update.attributes.Condition;
 import org.apache.nifi.update.attributes.Criteria;
+import org.apache.nifi.update.attributes.FlowFilePolicy;
 import org.apache.nifi.update.attributes.Rule;
 import org.apache.nifi.update.attributes.UpdateAttributeModelFactory;
 import org.apache.nifi.update.attributes.dto.DtoFactory;
 import org.apache.nifi.update.attributes.dto.RuleDTO;
 import org.apache.nifi.update.attributes.entity.ActionEntity;
 import org.apache.nifi.update.attributes.entity.ConditionEntity;
+import org.apache.nifi.update.attributes.entity.EvaluationContextEntity;
 import org.apache.nifi.update.attributes.entity.RuleEntity;
 import org.apache.nifi.update.attributes.entity.RulesEntity;
 import org.apache.nifi.update.attributes.serde.CriteriaSerDe;
-import org.apache.nifi.web.InvalidRevisionException;
-import org.apache.nifi.web.Revision;
-import org.apache.commons.lang3.StringUtils;
-
-import com.sun.jersey.api.NotFoundException;
-
-import org.apache.nifi.update.attributes.FlowFilePolicy;
-import org.apache.nifi.update.attributes.entity.EvaluationContextEntity;
 import org.apache.nifi.web.ComponentDetails;
 import org.apache.nifi.web.HttpServletConfigurationRequestContext;
 import org.apache.nifi.web.HttpServletRequestContext;
+import org.apache.nifi.web.InvalidRevisionException;
 import org.apache.nifi.web.NiFiWebConfigurationContext;
 import org.apache.nifi.web.NiFiWebConfigurationRequestContext;
 import org.apache.nifi.web.NiFiWebRequestContext;
+import org.apache.nifi.web.Revision;
 import org.apache.nifi.web.UiExtensionType;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.DELETE;
+import javax.ws.rs.DefaultValue;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.PUT;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.QueryParam;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.CacheControl;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.Response.ResponseBuilder;
+import javax.ws.rs.core.UriBuilder;
+import javax.ws.rs.core.UriInfo;
+import java.text.Collator;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import java.util.UUID;
--- End diff --

Yes, there were other changes here that were reverted and this was my IDE. 
I'll just revert this file. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1629 downgraded Kafka back to 0.8

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/282


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/252#issuecomment-198149231
  
@jskora do the attribute names that come from the media bundles have 
anything special to them in terms of how tika handles them or are they purely 
as found in the metadata of the raw entities?  Just want to make sure there 
isn't some special mapping/normalization to worry about as versions of tika 
evolves.

Also, i've not built this yet but do you know how large those parsers end 
up being when pulled in for the  nar?  I recall for some reason they can be 
quite huge which is why we have avoided them so far.  Thinking being the are 
perfectly fine once we have the registry.  Might be fine now too but curious.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user mcgilman commented on the pull request:

https://github.com/apache/nifi/pull/284#issuecomment-197860863
  
Looks really good overall. Found a couple things lingering that I believe 
can be removed prior to merging. Additionally, there are a number of 
src/test/resources in the nifi-kerberos-iaa-providers but there are no 
corresponding test cases. Are we comfortable removing these test resources?

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Processor: User friendly vs system friendly design

2016-03-19 Thread Adam Taft

I'm probably on the far end of favoring composibility and processor reuse.
In this case, I would even go one step further and suggest that you're
talking about three separate operations:

1.  Split a multi-line CSV input file into individual single line flowfiles.
2.  Read columns from a single CSV line into flowfile attributes.
3.  Pass flowfile attributes into the Velocity transform processor.

The point here, have you considered driving your Velocity template
transform using flowfile attributes as opposed to CSV?  Flowfile attributes
are NIFI's lowest common data representation, many many processors create
attributes which would enable your Velocity processor to be used by more
than just CSV input.

Adam

On Fri, Mar 18, 2016 at 11:06 AM, Uwe Geercken  wrote:

>
> Hello,
>
> my first mailing here. I am a Java developer, using Apache Velocity,
> Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very much
> a data guy.
>
> I have used Nifi for a while now and started yesterday of coding my first
> processor. I basically do it to widen my knowledge and learn something new.
>
> I started with the idea of combining Apache Velocity - a template engine -
> with Nifi. So in comes a CSV file, it gets merged with a template
> containing formatting information and some placeholders (and some limited
> logic maybe) and out comes a new set of data, formatted differently. So it
> separates the processing logic from the formatting. One could create HTML,
> XML, Json or other text based formats from it. Easy to use and very
> efficient.
>
> Now my question is: Should I rather implement the logic this way that I
> process a whole CSV file - which usually has multiple lines? That would be
> good for the user as he or she has to deal with only one processor doing
> the work. But the logic would be more specialized.
>
> The other way around, I could code the processor to handle one row of the
> CSV file and the user will have to come up with a flow that divides the CSV
> file into multiple flowfiles before my processor can be used. That is not
> so specialized but it requires more preparation work from the user.
>
> I tend to go the second way. Also because there is already a processor
> that will split a file into multiple flowfiles. But I wanted to hear your
> opinion of what is the best way to go. Do you have a recommendation for me?
> (Maybe the answer is to do both?!)
>
> Thanks for sharing your thoughts.
>
> Uwe
>

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user rickysaltzer commented on the pull request:

https://github.com/apache/nifi/pull/285#issuecomment-197998896
  
@joewitt posted a new version of the patch. I tested it out on my local 
machine and it seems to work as expected. That is, the stacktrace is logged to 
the `nifi-app.log`, while the the shortened version is logged to the UI. The 
`contrib-check` also checks out ;) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1332 Fixed HTTP-204 handling + unit test

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/259


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-615 - Create a processor to extract WAV fi...

2016-03-19 Thread jskora

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/252#issuecomment-198408663
  
@joewitt, the attribute names are created by Tika and seem to be based on 
the source file types.  The core Tika module was already in the project and the 
Tika parsers module appears to add ~600KB to the NAR.  Also, this is named 
"ExtractMediaMetadata" because using Tika allows it to support over 1,000 file 
types, instead of just WAV files, and allows it to be extended with custom 
parsers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1575: Add QueryDatabaseTable processor

Github user mattyb149 closed the pull request at:

https://github.com/apache/nifi/pull/261


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1631 increased timeouts on test

GitHub user olegz opened a pull request:

https://github.com/apache/nifi/pull/283

NIFI-1631 increased timeouts on test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/olegz/nifi NIFI-1631

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/283.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #283


commit 3d6fb4e9d6fb249dfe2ab91b8a45cd7db4d4937a
Author: Oleg Zhurakousky 
Date:   2016-03-16T14:32:07Z

NIFI-1631 increased timeouts on test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1631 increased timeouts on test

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/283


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user rickysaltzer commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56418891
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

That makes sense @joewitt. Should legitimate unexpected exceptions be 
surfaced to `INFO` by default? I feel like these are uncommon enough to warrant 
it, but I could be wrong. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-901: Add QueryCassandra and PutCassandraQL...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/237


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-901: Fixed unit test to verify IP address ...

Github user mcgilman commented on the pull request:

https://github.com/apache/nifi/pull/286#issuecomment-197997930
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Aw: Re: Processor additional documentation

Bryan,

all looks ok. I looked into the nifi-home/work/docs folder. There is nothing 
but a components folder. Inside there is a folder for my processor: 
com.datamelt.nifi.test.TemplateProcessor and inside the folder there is a file 
index.html and it contains the code of my additionalDetails.html file.

when I open the file in the web browser it looks good. I looked at other 
index.html files and they look similar.

but I noted that some folders have an inde.html file AND an 
additionalDetails.html file. maybe that is the problem?

greetings,

Uwe

Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
Von: "Bryan Bende" 
An: dev@nifi.apache.org
Betreff: Re: Processor additional documentation
Hi Uwe,

Do you have the additionalDetails.html file in your processors jar project,
under src/main/resources?

Similar to this:
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources

The expected project structure is described here:
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]

If you think that part is setup correctly, can you check under
nifi_home/work/docs and see if com.datamelt.nifi.test.TemplateProcessor is
there?

-Bryan

On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken  wrote:

>
> Hello,
>
> I am writing my first processor. As described in the documentation, I have
> added an HTML file to be used when the user selects "Usage":
>
> docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
>
> This is located in the root or the Processors nar file.
>
> The processor class is this:
>
> com/datamelt/nifi/test/TemplateProcessor.class
>
> The processor works, but selecting "Usage" won't show my HTML file.
>
> I understood that I write the HTML file and Nifi will picks it up when it
> starts. Or is this not true?
>
> Thanks for feedback,
>
> Uwe
>

[GitHub] nifi pull request: NIFI-1488 Refactoring HBase Kerberos support

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/281#discussion_r56410960
  
--- Diff: 
nifi-commons/nifi-hadoop-utils/src/main/java/org/apache/nifi/hadoop/KerberosTicketRenewer.java
 ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.hadoop;
+
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.nifi.logging.ComponentLog;
+
+import java.io.IOException;
+
+/**
+ * Periodically attempts to renew the Kerberos user's ticket for the given 
UGI.
+ */
+public class KerberosTicketRenewer implements Runnable {
+
+private final UserGroupInformation ugi;
+private final long renewalPeriod;
+private final ComponentLog logger;
+
+private volatile boolean stopped = false;
+
+public KerberosTicketRenewer(final UserGroupInformation ugi, final 
long renewalPeriod, final ComponentLog logger) {
+this.ugi = ugi;
+this.renewalPeriod = renewalPeriod;
+this.logger = logger;
+}
+
+@Override
+public void run() {
+stopped = false;
+while (!stopped) {
+try {
+logger.debug("Invoking renewal attempt for Kerberos 
ticket");
+// While we run this "frequently", the Hadoop 
implementation will only perform the login at 80% of ticket lifetime.
+ugi.checkTGTAndReloginFromKeytab();
+} catch (IOException e) {
+// Should failures to renew the ticket be retried more 
quickly?
--- End diff --

I definitely think they should. I believe 24 hours is a fairly common 
renewal period, but we don't want to wait 24 hours on failure. Perhaps provide 
a new constructor arg for retryPeriod?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1337: Add Riemann Reporting Task

Github user rickysaltzer commented on the pull request:

https://github.com/apache/nifi/pull/188#issuecomment-198464162
  
hey @joewitt  - not an issue. I haven't actually put this patch into 
production yet, but I have a good use case to try it out today / over the 
weekend. Let me give that a shot and report my findings. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56419500
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

I would say generally no.  They are uncommon in general but when they 
happen they happen in bursts.  As a general rule I think we should strive to 
make what the user sees be something a lot more friendly than a wild stack 
trace.  But, the logs should have them.  We did discuss this on dev list a 
while back and there were some great inputs from various folks.  We def need to 
do something here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: patch and pull request reviewing question

2016-03-19 Thread Mark Payne

+1 Yes please!

Sent from my iPhone

> On Mar 17, 2016, at 9:38 PM, Tony Kurc  wrote:
> 
> As I was reviewing a pull request, i realized that it actually was a bit
> more mental effort in this instance to review a squashed and rebased set of
> commits than if it was if I could just review the changes in a commit. Does
> anyone object to me adding in the contributors guide something to the
> extent of "Although you may be asked to rebase or squash your contribution
> as part of the review process, don't feel the need to do so speculatively.
> The committer working on merging the contribution may prefer to do these
> types of operations as part of the merge process, and the history of your
> patch or pull request may aid in the review process"
> 
> Tony

Re: patch and pull request reviewing question

+1

On Thu, Mar 17, 2016 at 9:37 PM, Tony Kurc  wrote:
> As I was reviewing a pull request, i realized that it actually was a bit
> more mental effort in this instance to review a squashed and rebased set of
> commits than if it was if I could just review the changes in a commit. Does
> anyone object to me adding in the contributors guide something to the
> extent of "Although you may be asked to rebase or squash your contribution
> as part of the review process, don't feel the need to do so speculatively.
> The committer working on merging the contribution may prefer to do these
> types of operations as part of the merge process, and the history of your
> patch or pull request may aid in the review process"
>
> Tony

[GitHub] nifi pull request: NIFI-901: Fixed unit test to verify IP address ...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/286


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1636: Print Stacktrace When Unepected OnTr...

Github user rickysaltzer commented on a diff in the pull request:

https://github.com/apache/nifi/pull/285#discussion_r56420502
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java ---
@@ -27,7 +30,12 @@ public final void onTrigger(final ProcessContext 
context, final ProcessSessionFa
 onTrigger(context, session);
 session.commit();
 } catch (final Throwable t) {
-getLogger().error("{} failed to process due to {}; rolling 
back session", new Object[]{this, t});
+StringWriter stacktraceWriter = new StringWriter();
--- End diff --

So if I understand correctly, this happens now but only in `DEBUG` mode. 
What we would like to see is that behavior but even if we're in `INFO` mode. 
That is, shortened version logged to the UI and the stacktrace logged to the 
log file? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/284#issuecomment-197610965
  
I removed all added TODO statements. I created NIFI-1637 to refactor the 
duplicated `KerberosUserDetailsService` and 
`AlternateKerberosUserDetailsService` to a shared utility module. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1571 initial commit of SpringContext suppo...

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/271#discussion_r56446521
  
--- Diff: 
nifi-nar-bundles/nifi-spring-bundle/nifi-spring-processors/src/main/java/org/apache/nifi/spring/SpringContextProcessor.java
 ---
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.spring;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.spring.SpringDataExchanger.SpringResponse;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.MessageChannel;
+import org.springframework.messaging.PollableChannel;
+
+/**
+ * Implementation of {@link Processor} capable of sending and receiving 
data
+ * from application defined in Spring Application context. It does so via
+ * predefined in/out {@link MessageChannel}s (see spring-messaging module 
of
+ * Spring). Once such channels are defined user is free to implement the 
rest of
+ * the application any way they wish (e.g., custom code and/or using 
frameworks
+ * such as Spring Integration or Camel).
+ * 
+ * The requirement and expectations for channel types are:
+ * 
+ * Input channel must be of type {@link MessageChannel} and named 
"fromNiFi"
+ * (see {@link SpringNiFiConstants#FROM_NIFI})
+ * Output channel must be of type {@link PollableChannel} and named 
"toNiFi"
+ * (see {@link SpringNiFiConstants#TO_NIFI})
+ * 
+ * 
+ * Below is the example of sample configuration:
+ *
+ * 
+ * ?xml version="1.0" encoding="UTF-8"?
+ * beans xmlns="http://www.springframework.org/schema/beans;
+ *   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ *   xmlns:int="http://www.springframework.org/schema/integration;
+ *  xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
+ *  http://www.springframework.org/schema/integration 
http://www.springframework.org/schema/integration/spring-integration-4.2.xsd";
+ *
+ *  int:channel id="fromNiFi"/
+ *
+ *  . . . . .
+ *
+ *  int:channel id="toNiFi"
+ *  int:queue/
+ *  /int:channel
+ *
+ * /beans
+ * 
+ * 
+ * Defining {@link MessageChannel} is optional. That's why this processor
+ * supports 3 modes of interaction with Spring Application Context:
+ * 
+ * Headless â no channels are

Re: Closing in on the Apache NiFi 0.6.0 release

Team,

Ok sooo close.  We have 6 tickets remaining.

- Additional functionality/cleanup for SplitText [1]
[status] Still in discussions. Recommend we move this change to 0.7.0.
Solid effort on both code contributor and reviewer side but this is a
tricky one.

- Support Kerberos based authentication to REST API [2]
[status] PR is in. Reviewing and PR tweaking appears active.  Looks
quite close and comments indicate great results.

- Add Kerberos support to HBase processors [3]
[status] Patch in. Under review.  Running on live test system with
great results.

- Add support for Spring Context loaded processors (Spring
Integrations, Camel, ...) [4]
[status] Appears ready. Getting review feedback.

- Support based database snapshot/query/change capture [5]
[status] Appears ready. Needs review.

- Zookeeper interaction for NiFI state management should limit state to 1MB [6]
[status] Patch is in and review under way.  Looks close.

[1] https://issues.apache.org/jira/browse/NIFI-1118
[2] https://issues.apache.org/jira/browse/NIFI-1274
[3] https://issues.apache.org/jira/browse/NIFI-1488
[4] https://issues.apache.org/jira/browse/NIFI-1571
[5] https://issues.apache.org/jira/browse/NIFI-1575
[6] https://issues.apache.org/jira/browse/NIFI-1626

Thanks
Joe

On Tue, Mar 15, 2016 at 11:56 AM, Aldrin Piri  wrote:
> Sounds great.  Will scope those out in the process.  Thanks, Tony.
>
> On Tue, Mar 15, 2016 at 11:37 AM, Tony Kurc  wrote:
>
>> Aldrin,
>> I did some crappy shell scripts to help, I'll try to put those up on the
>> wiki.
>>
>> Tony
>>
>> On Tue, Mar 15, 2016 at 11:29 AM, Aldrin Piri 
>> wrote:
>>
>> > In an effort to avoid me missing helpers and spamming the list, I will
>> > throw my hat into the ring to do this one.
>> >
>> > On Tue, Mar 15, 2016 at 11:28 AM, Joe Witt  wrote:
>> >
>> > > Team,
>> > >
>> > > Seeing good progress today.  Great!
>> > >
>> > > I know I volunteered to do the RM for the release but now I need to be
>> > > out of town and will have questionable Internet access.  Can someone
>> > > else grab it?
>> > >
>> > > Thanks
>> > > Joe
>> > >
>> > > On Mon, Mar 14, 2016 at 9:01 PM, Joe Witt  wrote:
>> > > > Team,
>> > > >
>> > > > Things are closing in well but we need to get pretty specific on
>> these
>> > > > to keep the release moving roughly along the lines of the schedule
>> > > > we've discussed previously.  Those of you who have tickets on here
>> > > > that can be moved to 0.7.0 please do so.  Otherwise, let's please
>> keep
>> > > > discussion/status information up so we can all see how we're trending
>> > > > toward release candidate readiness.
>> > > >
>> > > > The current items on the release based on JIRA fix version that
>> remain
>> > > are:
>> > > >
>> > > > - Supporting a new JMS producer/consumer style [1] [2] [3]
>> > > > [status]Great rounds of review with Oleg, Percivall, Moser.  Looks
>> > close.
>> > > >
>> > > > - Processors to interact with Apache Cassandra [4]
>> > > > [status]Looks ready but needing review.
>> > > >
>> > > > - Additional functionality/cleanup for SplitText [5]
>> > > > [status]Still in discussions.  Recommend we move this change to
>> 0.7.0.
>> > > >
>> > > > - Support Kerberos based authentication to REST API [6]
>> > > > [status]Update suggests it is almost ready.  A suggested key feature
>> > > > of this release.
>> > > >
>> > > > - Add Kerberos support to HBase processors [7]
>> > > > [status]Appears nearly ready.  Great discussions and group contribs.
>> > > >
>> > > > - Add support for Spring Context loaded processors (Spring
>> > > > Integrations, Camel, ...) [8]
>> > > > [status]Appears ready.  Needs review.
>> > > >
>> > > > - Support based database snapshot/query/change capture [9]
>> > > > [status] Appears ready.  Needs review.
>> > > >
>> > > > - Allow empty content posting w/o content type header [10]
>> > > > [status] Discussions led to changing behavior a bit.  Possibly slide
>> > > > to 0.7.0 as an alternative approach exists for now.
>> > > >
>> > > > - Support Kinesis Firehose interaction [11]
>> > > > [status] Appears to be a candidate to slide to 0.7.0
>> > > >
>> > > > - Support Apache Spark 1.6 for Spark receiver [12]
>> > > > [status] Appears ready but needing review.
>> > > >
>> > > > - Support SSL with AMQP processors [13]
>> > > > [status] Appears ready but needs review as it adds to a commonly used
>> > > > SSL interface.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/NIFI-614
>> > > > [2] https://issues.apache.org/jira/browse/NIFI-1424
>> > > > [3] https://issues.apache.org/jira/browse/NIFI-1425
>> > > > [4] https://issues.apache.org/jira/browse/NIFI-901
>> > > > [5] https://issues.apache.org/jira/browse/NIFI-1118
>> > > > [6] https://issues.apache.org/jira/browse/NIFI-1274
>> > > > [7] https://issues.apache.org/jira/browse/NIFI-1488
>> > > > [8] https://issues.apache.org/jira/browse/NIFI-1571
>> > > > [9]

[GitHub] nifi pull request: Nifi 1274

Github user JPercivall commented on the pull request:

https://github.com/apache/nifi/pull/284#issuecomment-197944422
  
+1 (with Gilman's last modifications)

Used Gilman's branch to do a contrib check build, reviewed that comments 
were addressed and ran through setting up a kerberos secure nifi instance. 
Looks good.

Thanks for contributing this awesome new feature!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1274