Re: [ANNOUNCE] New NiFi PMC member Marc Parisi

[Jeff]

Congrats, Marc!  Treat yourself to some Kobe beef. :)

On Thu, Dec 14, 2017 at 9:47 PM Mike Hogue  wrote:

> Congrats, Marc! Well deserved!
> On Thu, Dec 14, 2017 at 20:41 Jeremy Dyer  wrote:
>
> > Congrats Marc! Lots of good work I have seen from you and appreciate all
> > you have given the community!
> >
> > - Jeremy Dyer
> >
> > > On Dec 14, 2017, at 8:29 PM, Scott Aslan 
> wrote:
> > >
> > > Congrats!
> > >
> > >> On Thu, Dec 14, 2017 at 6:50 PM, Matt Burgess 
> > wrote:
> > >>
> > >> Congratulations Marc with a C :) Welcome aboard, looking forward to
> > >> more kick-arse contributions!
> > >>
> > >>> On Thu, Dec 14, 2017 at 6:47 PM, Tony Kurc  wrote:
> > >>> NiFi community,
> > >>> On behalf of the Apache NiFi PMC, I am pleased to announce that Marc
> > >> Parisi
> > >>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> > >> greatly
> > >>> appreciate all of Marc's hard work and generous contributions to the
> > >>> project. We look forward to continued involvement in the project.
> > >>>
> > >>> Marc is a major contributor to MiNiFi CPP; in addition to his code
> and
> > >>> review contributions, he recently release managed the 0.3.0 release.
> > He's
> > >>> super friendly, active across the NiFi project, and can lift a small
> > car.
> > >>>
> > >>> Congratulations and welcome, Marc!
> > >>>
> > >>> Tony
> > >>
> >
>

Re: [ANNOUNCE] New NiFi PMC member Marc Parisi

[Mike Hogue]

Congrats, Marc! Well deserved!
On Thu, Dec 14, 2017 at 20:41 Jeremy Dyer  wrote:

> Congrats Marc! Lots of good work I have seen from you and appreciate all
> you have given the community!
>
> - Jeremy Dyer
>
> > On Dec 14, 2017, at 8:29 PM, Scott Aslan  wrote:
> >
> > Congrats!
> >
> >> On Thu, Dec 14, 2017 at 6:50 PM, Matt Burgess 
> wrote:
> >>
> >> Congratulations Marc with a C :) Welcome aboard, looking forward to
> >> more kick-arse contributions!
> >>
> >>> On Thu, Dec 14, 2017 at 6:47 PM, Tony Kurc  wrote:
> >>> NiFi community,
> >>> On behalf of the Apache NiFi PMC, I am pleased to announce that Marc
> >> Parisi
> >>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> >> greatly
> >>> appreciate all of Marc's hard work and generous contributions to the
> >>> project. We look forward to continued involvement in the project.
> >>>
> >>> Marc is a major contributor to MiNiFi CPP; in addition to his code and
> >>> review contributions, he recently release managed the 0.3.0 release.
> He's
> >>> super friendly, active across the NiFi project, and can lift a small
> car.
> >>>
> >>> Congratulations and welcome, Marc!
> >>>
> >>> Tony
> >>
>

Re: [ANNOUNCE] New NiFi PMC member Marc Parisi

[Jeremy Dyer]

Congrats Marc! Lots of good work I have seen from you and appreciate all you 
have given the community!

- Jeremy Dyer

> On Dec 14, 2017, at 8:29 PM, Scott Aslan  wrote:
> 
> Congrats!
> 
>> On Thu, Dec 14, 2017 at 6:50 PM, Matt Burgess  wrote:
>> 
>> Congratulations Marc with a C :) Welcome aboard, looking forward to
>> more kick-arse contributions!
>> 
>>> On Thu, Dec 14, 2017 at 6:47 PM, Tony Kurc  wrote:
>>> NiFi community,
>>> On behalf of the Apache NiFi PMC, I am pleased to announce that Marc
>> Parisi
>>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
>> greatly
>>> appreciate all of Marc's hard work and generous contributions to the
>>> project. We look forward to continued involvement in the project.
>>> 
>>> Marc is a major contributor to MiNiFi CPP; in addition to his code and
>>> review contributions, he recently release managed the 0.3.0 release. He's
>>> super friendly, active across the NiFi project, and can lift a small car.
>>> 
>>> Congratulations and welcome, Marc!
>>> 
>>> Tony
>> 

Re: [ANNOUNCE] New NiFi PMC member Marc Parisi

[Scott Aslan]

Congrats!

On Thu, Dec 14, 2017 at 6:50 PM, Matt Burgess  wrote:

> Congratulations Marc with a C :) Welcome aboard, looking forward to
> more kick-arse contributions!
>
> On Thu, Dec 14, 2017 at 6:47 PM, Tony Kurc  wrote:
> > NiFi community,
> > On behalf of the Apache NiFi PMC, I am pleased to announce that Marc
> Parisi
> > has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly
> > appreciate all of Marc's hard work and generous contributions to the
> > project. We look forward to continued involvement in the project.
> >
> > Marc is a major contributor to MiNiFi CPP; in addition to his code and
> > review contributions, he recently release managed the 0.3.0 release. He's
> > super friendly, active across the NiFi project, and can lift a small car.
> >
> > Congratulations and welcome, Marc!
> >
> > Tony
>

Re: Unable to checkpoint FlowFile Repository (No space left on device)

[尹文才]

Thanks Michael, I could try turning off the content repository archive
feature. One thing I'm curious though is when I watched the content
repository folder when my nifi flow is still actively running, it kept
writing new empty files to the archive folder and some of them are old
flowfiles' content after I checked(only a very small portion, most are
empty files, I used the default archive option of 12 hours and 50%), the
speed of creating these files differ slightly when I tune
the interval of my data extracting processor. The most important thing is
My custom processor that extracts data has no more data to extract at the
moment, so I didn't know what's going on inside NiFI that kept increasing
the use of inodes.

/Ben

2017-12-15 0:27 GMT+08:00 Michael Moser :

> Maximum number of inodes is defined when you build a file system on a
> device.  You would have to backup your data, rebuild your file system and
> tell it to allocate more inodes than the default (an mkfs option, I
> think?), then restore your data.
>
> You can turn off the NiFi content_repository archive, if you don't need
> that feature, by setting nifi.content.repository.archive.enabled=false in
> your nifi.properties.
>
> -- Mike
>
>
> On Thu, Dec 14, 2017 at 2:22 AM, 尹文才  wrote:
>
> > One Strange thing one of our testers found was that we're using the
> default
> > 12hours archive and 50% disk space configuration, he noticed that when
> nifi
> > removed the archived files inside the content_repository, he checked the
> > inodes ' count didn't go down, then he tried to remove some of the
> archived
> > files inside using the rm command, the inodes's count did go down.
> >
> > /Ben
> >
> > 2017-12-14 9:49 GMT+08:00 尹文才 :
> >
> > > Hi Michael, the no space left on device occurred again and I checked
> the
> > > inodes at the time and found it was indeed full, why would the inodes
> > > become full and are there any solutions to get around this problem?
> > Thanks.
> > >
> > > /Ben
> > >
> > > 2017-12-13 13:36 GMT+08:00 尹文才 :
> > >
> > >> Hi Michael, I checked the system available inodes by running df -i
> > >> command and there're quite enough inodes in the system. I then removed
> > all
> > >> the files in all repository folders and restarted
> > >> the system, I couldn't see the error again. I will continue to track
> the
> > >> problem to see what's causing it, but it seems not relevant to the
> inode
> > >> use-up reason you mentioned. Thanks.
> > >>
> > >> /Ben
> > >>
> > >> 2017-12-12 23:45 GMT+08:00 Michael Moser :
> > >>
> > >>> Greetings Ben,
> > >>>
> > >>> The "No space left on device" error can also be caused by running out
> > of
> > >>> inodes on your device.  You can check this with "df -i".
> > >>>
> > >>> -- Mike
> > >>>
> > >>>
> > >>> On Tue, Dec 12, 2017 at 1:36 AM, 尹文才  wrote:
> > >>>
> > >>> > sorry that I forgot to mention the environment that caused this
> > >>> problem,
> > >>> > I'm using the latest nifi 1.4.0 release and installed it on centos
> 7.
> > >>> >
> > >>> > 2017-12-12 14:35 GMT+08:00 尹文才 :
> > >>> >
> > >>> > > Hi guys, I'm running into a very weird problem, I wrote a
> processor
> > >>> > > specifically to extract some data
> > >>> > > and I found starting from yesterday it kept showing errors in the
> > >>> log, as
> > >>> > > below:
> > >>> > >
> > >>> > > 2017-12-12 14:01:04,661 INFO [pool-10-thread-1] o.a.n.c.r.
> > >>> > WriteAheadFlowFileRepository
> > >>> > > Initiating checkpoint of FlowFile Repository
> > >>> > > 2017-12-12 14:01:04,676 ERROR [pool-10-thread-1] o.a.n.c.r.
> > >>> > WriteAheadFlowFileRepository
> > >>> > > Unable to checkpoint FlowFile Repository due to
> > >>> > > java.io.FileNotFoundException: ../flowfile_repository/
> > >>> > partition-5/96.journal
> > >>> > > (No space left on device)
> > >>> > > java.io.FileNotFoundException: ../flowfile_repository/
> > >>> > partition-5/96.journal
> > >>> > > (No space left on device)
> > >>> > > at java.io.FileOutputStream.open0(Native Method)
> > >>> > > at java.io.FileOutputStream.open(
> > FileOutputStream.java:270)
> > >>> > > at java.io.FileOutputStream. > >>> >(FileOutputStream.java:213)
> > >>> > > at java.io.FileOutputStream. > >>> >(FileOutputStream.java:162)
> > >>> > > at org.wali.MinimalLockingWriteAheadLog$
> > Partition.rollover(
> > >>> > > MinimalLockingWriteAheadLog.java:779)
> > >>> > > at org.wali.MinimalLockingWriteAheadLog.checkpoint(
> > >>> > > MinimalLockingWriteAheadLog.java:528)
> > >>> > > at org.apache.nifi.controller.repository.
> > >>> > > WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRe
> > >>> pository.
> > >>> > > java:451)
> > >>> > > at org.apache.nifi.controller.repository.
> > >>> > > WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.
> > >>> > java:423)
> > >>> > >   

Re: [ANNOUNCE] New NiFi PMC member Marc Parisi

[Matt Burgess]

Congratulations Marc with a C :) Welcome aboard, looking forward to
more kick-arse contributions!

On Thu, Dec 14, 2017 at 6:47 PM, Tony Kurc  wrote:
> NiFi community,
> On behalf of the Apache NiFi PMC, I am pleased to announce that Marc Parisi
> has accepted the PMC's invitation to join the Apache NiFi PMC.  We greatly
> appreciate all of Marc's hard work and generous contributions to the
> project. We look forward to continued involvement in the project.
>
> Marc is a major contributor to MiNiFi CPP; in addition to his code and
> review contributions, he recently release managed the 0.3.0 release. He's
> super friendly, active across the NiFi project, and can lift a small car.
>
> Congratulations and welcome, Marc!
>
> Tony

Re: Questions re: customizing a NiFi processor

[Mike Thomsen]

Jim,

I'd recommend just forking NiFi and updating it with a patch. Writing your
own processor for this sounds like massive overkill for what you need. If
you know how to use Git, you'll be fine. You can add me on GitHub as a
reviewer when you're done (I can at least do the initial pass for Andy or
someone else).

The easiest way to do the compliance check when you think you're done is to
have another terminal open to $NIFI_ROOT and run this:

mvn clean install -Pcontrib-check -pl :some-module-name

Note that ":" at the start. It's required for Maven to identify it as a
module name. That'll build only the module you want to test.

You can also run just your own tests with:

mvn test -Dtest=org.apache.nifi.x.y.z.MyTest

Sometimes that's necessary if other tests are broken for whatever reason
and they're not related to your code.

On Sat, Dec 2, 2017 at 6:43 PM, James McMahon  wrote:

> Good evening. I am familiar with NiFi and have developed a number of
> workflows using out -of-the-box processors. I now need to develop a custom
> version of PublishAMQP that permits the use of attributes in expression
> language to set specs for the AMQP target.
>
> I am not very familiar with Maven, but have done some initial research and
> hope I can bootstrap my effort following this link:
> https://community.hortonworks.com/articles/4318/build-
> custom-nifi-processor.html
>
> Lacking any experience developing custom processors, I'd like to understand
> how I would establish in my maven project the source code for the existing
> PublishAMQP. I'd like to modify that proven code base to allow the use of
> expression language to set other configuration properties.
>
> Can anyone help me with an example where source code for an existing
> processor is brought in from a github resource or the like to use as my
> starting point?
>
> Thank you in advance for any assistance. -Jim
>

Re: Unable to checkpoint FlowFile Repository (No space left on device)

[Michael Moser]

Maximum number of inodes is defined when you build a file system on a
device.  You would have to backup your data, rebuild your file system and
tell it to allocate more inodes than the default (an mkfs option, I
think?), then restore your data.

You can turn off the NiFi content_repository archive, if you don't need
that feature, by setting nifi.content.repository.archive.enabled=false in
your nifi.properties.

-- Mike


On Thu, Dec 14, 2017 at 2:22 AM, 尹文才  wrote:

> One Strange thing one of our testers found was that we're using the default
> 12hours archive and 50% disk space configuration, he noticed that when nifi
> removed the archived files inside the content_repository, he checked the
> inodes ' count didn't go down, then he tried to remove some of the archived
> files inside using the rm command, the inodes's count did go down.
>
> /Ben
>
> 2017-12-14 9:49 GMT+08:00 尹文才 :
>
> > Hi Michael, the no space left on device occurred again and I checked the
> > inodes at the time and found it was indeed full, why would the inodes
> > become full and are there any solutions to get around this problem?
> Thanks.
> >
> > /Ben
> >
> > 2017-12-13 13:36 GMT+08:00 尹文才 :
> >
> >> Hi Michael, I checked the system available inodes by running df -i
> >> command and there're quite enough inodes in the system. I then removed
> all
> >> the files in all repository folders and restarted
> >> the system, I couldn't see the error again. I will continue to track the
> >> problem to see what's causing it, but it seems not relevant to the inode
> >> use-up reason you mentioned. Thanks.
> >>
> >> /Ben
> >>
> >> 2017-12-12 23:45 GMT+08:00 Michael Moser :
> >>
> >>> Greetings Ben,
> >>>
> >>> The "No space left on device" error can also be caused by running out
> of
> >>> inodes on your device.  You can check this with "df -i".
> >>>
> >>> -- Mike
> >>>
> >>>
> >>> On Tue, Dec 12, 2017 at 1:36 AM, 尹文才  wrote:
> >>>
> >>> > sorry that I forgot to mention the environment that caused this
> >>> problem,
> >>> > I'm using the latest nifi 1.4.0 release and installed it on centos 7.
> >>> >
> >>> > 2017-12-12 14:35 GMT+08:00 尹文才 :
> >>> >
> >>> > > Hi guys, I'm running into a very weird problem, I wrote a processor
> >>> > > specifically to extract some data
> >>> > > and I found starting from yesterday it kept showing errors in the
> >>> log, as
> >>> > > below:
> >>> > >
> >>> > > 2017-12-12 14:01:04,661 INFO [pool-10-thread-1] o.a.n.c.r.
> >>> > WriteAheadFlowFileRepository
> >>> > > Initiating checkpoint of FlowFile Repository
> >>> > > 2017-12-12 14:01:04,676 ERROR [pool-10-thread-1] o.a.n.c.r.
> >>> > WriteAheadFlowFileRepository
> >>> > > Unable to checkpoint FlowFile Repository due to
> >>> > > java.io.FileNotFoundException: ../flowfile_repository/
> >>> > partition-5/96.journal
> >>> > > (No space left on device)
> >>> > > java.io.FileNotFoundException: ../flowfile_repository/
> >>> > partition-5/96.journal
> >>> > > (No space left on device)
> >>> > > at java.io.FileOutputStream.open0(Native Method)
> >>> > > at java.io.FileOutputStream.open(
> FileOutputStream.java:270)
> >>> > > at java.io.FileOutputStream. >>> >(FileOutputStream.java:213)
> >>> > > at java.io.FileOutputStream. >>> >(FileOutputStream.java:162)
> >>> > > at org.wali.MinimalLockingWriteAheadLog$
> Partition.rollover(
> >>> > > MinimalLockingWriteAheadLog.java:779)
> >>> > > at org.wali.MinimalLockingWriteAheadLog.checkpoint(
> >>> > > MinimalLockingWriteAheadLog.java:528)
> >>> > > at org.apache.nifi.controller.repository.
> >>> > > WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRe
> >>> pository.
> >>> > > java:451)
> >>> > > at org.apache.nifi.controller.repository.
> >>> > > WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.
> >>> > java:423)
> >>> > > at java.util.concurrent.Executors$RunnableAdapter.
> >>> > > call(Executors.java:511)
> >>> > > at java.util.concurrent.FutureTask.runAndReset(
> >>> > > FutureTask.java:308)
> >>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
> >>> > > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.
> java:180)
> >>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
> >>> > > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >>> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> >>> > > ThreadPoolExecutor.java:1142)
> >>> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >>> > > ThreadPoolExecutor.java:617)
> >>> > > at java.lang.Thread.run(Thread.java:745)
> >>> > >
> >>> > >
> >>> > > I noticed the log mentioned no space left on device and I went to
> >>> check
> >>> > > the available space and found 33G left. Does anyone know what could
> >>> > > possibly cause this and how to resolve this