Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Marc]

Congrats Mike!

On Fri, Nov 10, 2017 at 8:33 AM, Vincent Russell 
wrote:

> congrats Mike
>
> On Fri, Nov 10, 2017 at 8:20 AM, Matt Burgess 
> wrote:
>
> > Congratulations Mike, glad to have you aboard!
> >
> > On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Mike
> > > Hogue has accepted the PMC's invitation to become a committer on the
> > Apache
> > > NiFi project. We greatly appreciate all of Mike's hard work and
> generous
> > > contributions to the project. We look forward to his continued
> > involvement
> > > in the project.
> > >
> > > In the past several months, Mike has contributed on a number of fronts.
> > He
> > > was primary author on several new features in 1.4.0 to include the gRPC
> > > bundle. He also improved SSL/TLS controller services, and made great
> > > documentation improvements on LICENSE and NOTICE best practices and has
> > > helped work down our pull request backlog by reviewing new
> contributions.
> > >
> > > Welcome and congratulations!
> > >
> > > Tony
> >
>

Re: Ingest data into SQL database using NiFi

[Pierre Villard]

Hi Tina,

For QueryRecord processor, I'd recommend you reading this excellent article
[1].

[1] https://blogs.apache.org/nifi/entry/real-time-sql-on-event

Hope this helps,
Pierre

2017-11-10 21:59 GMT+01:00 tzhu :

> Hi Koji,
>
> I was not working on NiFi for a while, and now I'm back with more
> questions...
>
> In QueryRecord, how should I write a SQL query? It doesn't seem to give me
> the choice to use SQL anywhere, even if I choose the script option. And
> what
> exactly should I do for the reader and writer?
>
> Thanks,
>
> Tina
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: How to count the number of occurrences of a certain string in file

[Mark Payne]

Tina,

My apologies - I was conflating ExecuteScript with ExecuteStreamCommand.
In ExecuteStream, the contents of the FlowFile are not read from StdIn but 
rather
you'd want to use the session to read the contents. So in Groovy we'd do 
something like:

import org.apache.nifi.processors.script.ExecuteScript
def flowFile = session.get()
def in = session.read(flowFile)
def totalDisconnections = 0
def totalConnections = 0

try {
// Here, 'in' is the InputStream that contains the data
finally {
in.close()
}

flowFile = session.putAttribute("TOTAL_DISCONNECTIONS", totalDisconnections)
flowFile = session.putAttribute("TOTAL_CONNECTIONS", totalConnections)
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)

Sorry, I am not familiar enough with Python to provide any sort of meaningful 
suggestions on what
to do there.

Thanks
-Mark



On Nov 10, 2017, at 4:01 PM, tzhu 
> wrote:

Hi Mark,

This is the script I'm using currently:

import sys
TOTAL_DISCONNECTIONS = 0
TOTAL_CONNECTIONS = 0
flowFile = sys.stdin
if (flowFile != None):
   for line in flowFile:
   if "Lost connection to server" in line:
   TOTAL_DISCONNECTIONS += 1
   if "Established connection to server" in line:
TOTAL_CONNECTIONS += 1
attrMap = {"TOTAL_DISCONNECTIONS":TOTAL_DISCONNECTIONS,
  "TOTAL_CONNECTIONS":TOTAL_CONNECTIONS}
flowFile = session.putAllAttributes(flowFile, attrMap)



Does it make sense to you? The error message says "filereader" object is not
iterable. So what is in flowFile now? How should I access the content in
filwFile?

Thanks,

Tina



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: How to count the number of occurrences of a certain string in file

[tzhu]

Hi Mark,

This is the script I'm using currently:

import sys
TOTAL_DISCONNECTIONS = 0 
TOTAL_CONNECTIONS = 0 
flowFile = sys.stdin
if (flowFile != None):
for line in flowFile:
if "Lost connection to server" in line:
TOTAL_DISCONNECTIONS += 1
if "Established connection to server" in line:
 TOTAL_CONNECTIONS += 1
attrMap = {"TOTAL_DISCONNECTIONS":TOTAL_DISCONNECTIONS,
   "TOTAL_CONNECTIONS":TOTAL_CONNECTIONS}  
flowFile = session.putAllAttributes(flowFile, attrMap)



Does it make sense to you? The error message says "filereader" object is not
iterable. So what is in flowFile now? How should I access the content in
filwFile?

Thanks,

Tina



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Ingest data into SQL database using NiFi

[tzhu]

Hi Koji,

I was not working on NiFi for a while, and now I'm back with more
questions...

In QueryRecord, how should I write a SQL query? It doesn't seem to give me
the choice to use SQL anywhere, even if I choose the script option. And what
exactly should I do for the reader and writer?

Thanks,

Tina



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Forcing FlowFiles to Expire at a Connection

[Mark Payne]

Mike,

FlowFile expiration can happen in two ways: 

1) There is a background thread that runs every 30 seconds or so that can 
trigger Expiration
2) Whenever a Processor (or other component) pulls from a queue, any expired 
FlowFiles are
expired instead of being handed to the Processor.

So if the destination of the connection is not running, you will see it expire 
only once every 30
seconds.

Thanks
-Mark


> On Nov 10, 2017, at 9:32 AM, Michael Hogue  
> wrote:
> 
> All,
> 
>   I'm working on NIFI-620 ,
> which is to add expired FlowFile count to the Status History UI for
> connections, and i've gotten to a point that i'm ready to test this but i'm
> having trouble forcing FlowFiles to expire. I've created this flow [1] with
> 1ns FlowFile expirations, but no luck.
> 
>   Is there an easy way to force FlowFiles to be expired at a connection so
> i can verify that my changes are being presented in the UI? Reading through
> the code, it seems that it shouldn't be too difficult but possibly i'm
> missing something.
> 
> Thanks,
> Mike
> 
> [1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad

Re: Forcing FlowFiles to Expire at a Connection

[Pierre Villard]

Hi Mike,

What I would suggest is :
GFF (scheduled every 1s) -> LogAttribute (scheduled every 5s)
And expiration set to 1s on success relationship.
You should have about 1FF for 5 going through LogAttribute.

Pierre


2017-11-10 15:32 GMT+01:00 Michael Hogue :

> All,
>
>I'm working on NIFI-620  >,
> which is to add expired FlowFile count to the Status History UI for
> connections, and i've gotten to a point that i'm ready to test this but i'm
> having trouble forcing FlowFiles to expire. I've created this flow [1] with
> 1ns FlowFile expirations, but no luck.
>
>Is there an easy way to force FlowFiles to be expired at a connection so
> i can verify that my changes are being presented in the UI? Reading through
> the code, it seems that it shouldn't be too difficult but possibly i'm
> missing something.
>
> Thanks,
> Mike
>
> [1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad
>

Forcing FlowFiles to Expire at a Connection

[Michael Hogue]

All,

   I'm working on NIFI-620 ,
which is to add expired FlowFile count to the Status History UI for
connections, and i've gotten to a point that i'm ready to test this but i'm
having trouble forcing FlowFiles to expire. I've created this flow [1] with
1ns FlowFile expirations, but no luck.

   Is there an easy way to force FlowFiles to be expired at a connection so
i can verify that my changes are being presented in the UI? Reading through
the code, it seems that it shouldn't be too difficult but possibly i'm
missing something.

Thanks,
Mike

[1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad

Re: How to count the number of occurrences of a certain string in file

[Mark Payne]

Tina,

In NiFi, a FlowFile is made up of two parts: Content (the payload, or bytes) 
and Attributes (metadata about the content).
When you are using the Expression Language, you are operating on FlowFile 
Attributes, not the content. So if you are
wanting to count the number of occurrences of some string in the content, the 
Expression Language will not help you.

This is why I was suggesting using the ExecuteScript processor. You can have a 
script that reads the content from
StdIn (this will provide you the content of the FlowFile) and count the number 
of occurrences of each word/phrase of
interest. Once you have counted the number of occurrences, you can add those to 
the FlowFile as attributes.
Or, alternatively, you could write out the data to the contents of the 
FlowFile, from within your script.

Thanks
-Mark


> On Nov 9, 2017, at 4:05 PM, tzhu  wrote:
> 
> Hi Mark,
> 
> According to the  language guide
> 
>  
> , the count can be done by:
> ${allMatchingAttributes("abc","xyz"):contains("world"):count()}
> 
> I'm wondering if I can contain this one in one of the processor, as a side
> product of "update attribute" or some other processor.
> 
> Thanks,
> 
> Tina
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Vincent Russell]

congrats Mike

On Fri, Nov 10, 2017 at 8:20 AM, Matt Burgess  wrote:

> Congratulations Mike, glad to have you aboard!
>
> On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that Mike
> > Hogue has accepted the PMC's invitation to become a committer on the
> Apache
> > NiFi project. We greatly appreciate all of Mike's hard work and generous
> > contributions to the project. We look forward to his continued
> involvement
> > in the project.
> >
> > In the past several months, Mike has contributed on a number of fronts.
> He
> > was primary author on several new features in 1.4.0 to include the gRPC
> > bundle. He also improved SSL/TLS controller services, and made great
> > documentation improvements on LICENSE and NOTICE best practices and has
> > helped work down our pull request backlog by reviewing new contributions.
> >
> > Welcome and congratulations!
> >
> > Tony
>

Re: [ANNOUNCE] New Apache NiFi Committer Peter Wicks

[Matt Burgess]

Congrats Peter, well deserved!

On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Peter
> Wicks has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Peter's hard work and generous
> contributions to the project. We look forward to his continued involvement
> in the project.
>
> Peter has been contributing for over a year. Many of you have benefited
> from useful improvements and new features centered mainly around databases
> and record-based processors. I'm sure many of you have had the pleasure of
> interacting with him on the mailing lists, as he's always been willing to
> pitch in and make the Apache NiFi community stronger.
>
> Welcome and congratulations!
>
> Tony

Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Matt Burgess]

Congratulations Mike, glad to have you aboard!

On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Mike
> Hogue has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Mike's hard work and generous
> contributions to the project. We look forward to his continued involvement
> in the project.
>
> In the past several months, Mike has contributed on a number of fronts. He
> was primary author on several new features in 1.4.0 to include the gRPC
> bundle. He also improved SSL/TLS controller services, and made great
> documentation improvements on LICENSE and NOTICE best practices and has
> helped work down our pull request backlog by reviewing new contributions.
>
> Welcome and congratulations!
>
> Tony

Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Scott Aslan]

Congrats!

On Fri, Nov 10, 2017 at 3:22 AM, Pierre Villard  wrote:

> Congrats Mike, welcome on board!
>
> 2017-11-10 5:27 GMT+01:00 Jeff :
>
> > Congrats, Mike!
> >
> > On Thu, Nov 9, 2017 at 7:19 PM Jeremy Dyer  wrote:
> >
> > > Mike glad to have you and looking forward to working with you on things
> > in
> > > the future!
> > >
> > > On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> > >
> > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Mike
> > > > Hogue has accepted the PMC's invitation to become a committer on the
> > > Apache
> > > > NiFi project. We greatly appreciate all of Mike's hard work and
> > generous
> > > > contributions to the project. We look forward to his continued
> > > involvement
> > > > in the project.
> > > >
> > > > In the past several months, Mike has contributed on a number of
> fronts.
> > > He
> > > > was primary author on several new features in 1.4.0 to include the
> gRPC
> > > > bundle. He also improved SSL/TLS controller services, and made great
> > > > documentation improvements on LICENSE and NOTICE best practices and
> has
> > > > helped work down our pull request backlog by reviewing new
> > contributions.
> > > >
> > > > Welcome and congratulations!
> > > >
> > > > Tony
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Peter Wicks

[Scott Aslan]

Congrats!

On Fri, Nov 10, 2017 at 3:21 AM, Pierre Villard  wrote:

> Congrats Peter, well deserved!
>
> 2017-11-10 5:26 GMT+01:00 Jeff :
>
> > Congrats, Peter!
> >
> > On Thu, Nov 9, 2017 at 7:18 PM Jeremy Dyer  wrote:
> >
> > > Welcome Peter excited to have you on board!
> > >
> > > On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> > >
> > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > > Peter
> > > > Wicks has accepted the PMC's invitation to become a committer on the
> > > Apache
> > > > NiFi project. We greatly appreciate all of Peter's hard work and
> > generous
> > > > contributions to the project. We look forward to his continued
> > > involvement
> > > > in the project.
> > > >
> > > > Peter has been contributing for over a year. Many of you have
> benefited
> > > > from useful improvements and new features centered mainly around
> > > databases
> > > > and record-based processors. I'm sure many of you have had the
> pleasure
> > > of
> > > > interacting with him on the mailing lists, as he's always been
> willing
> > to
> > > > pitch in and make the Apache NiFi community stronger.
> > > >
> > > > Welcome and congratulations!
> > > >
> > > > Tony
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Pierre Villard]

Congrats Mike, welcome on board!

2017-11-10 5:27 GMT+01:00 Jeff :

> Congrats, Mike!
>
> On Thu, Nov 9, 2017 at 7:19 PM Jeremy Dyer  wrote:
>
> > Mike glad to have you and looking forward to working with you on things
> in
> > the future!
> >
> > On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Mike
> > > Hogue has accepted the PMC's invitation to become a committer on the
> > Apache
> > > NiFi project. We greatly appreciate all of Mike's hard work and
> generous
> > > contributions to the project. We look forward to his continued
> > involvement
> > > in the project.
> > >
> > > In the past several months, Mike has contributed on a number of fronts.
> > He
> > > was primary author on several new features in 1.4.0 to include the gRPC
> > > bundle. He also improved SSL/TLS controller services, and made great
> > > documentation improvements on LICENSE and NOTICE best practices and has
> > > helped work down our pull request backlog by reviewing new
> contributions.
> > >
> > > Welcome and congratulations!
> > >
> > > Tony
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Peter Wicks

[Pierre Villard]

Congrats Peter, well deserved!

2017-11-10 5:26 GMT+01:00 Jeff :

> Congrats, Peter!
>
> On Thu, Nov 9, 2017 at 7:18 PM Jeremy Dyer  wrote:
>
> > Welcome Peter excited to have you on board!
> >
> > On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc  wrote:
> >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Peter
> > > Wicks has accepted the PMC's invitation to become a committer on the
> > Apache
> > > NiFi project. We greatly appreciate all of Peter's hard work and
> generous
> > > contributions to the project. We look forward to his continued
> > involvement
> > > in the project.
> > >
> > > Peter has been contributing for over a year. Many of you have benefited
> > > from useful improvements and new features centered mainly around
> > databases
> > > and record-based processors. I'm sure many of you have had the pleasure
> > of
> > > interacting with him on the mailing lists, as he's always been willing
> to
> > > pitch in and make the Apache NiFi community stronger.
> > >
> > > Welcome and congratulations!
> > >
> > > Tony
> > >
> >
>