Re: How to stop Nifi workflow automatically

[Joe Witt]

Selvam

NiFi isn't  oriented around jobs which get executed and finish.  It is
oriented around continuous streams of data delivery.  So with that
thinking there is no mechanism to say "run this processor for X time
units".  There was some discussion a while back about adding that to
the scheduling criteria though such that a source processor

Could you describe your use case a bit more?  In your case will you
manually start the flow and you just want it to stop executing after a
period of time?  What about data that might still be in the flow?
Does it have to stop or would it be ok for only the source processor
to automatically stop so that the others will process data until it is
gone?

Thanks
Joe

On Fri, Oct 7, 2016 at 9:01 AM, Selvam Raman  wrote:
> Hi,
>
> I am running a NIFI workflow which will run for 2 to 3 hour. is there
> provision to stop the flow automatically once it is done without manual
> intervention.
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"

Re: Nifi hardware recommendation

[Joe Witt]

Ali,

You have a lot of nice resources to work with there.  I'd recommend the
series of RAID-1 configuration personally provided you keep in mind this
means you can only lose a single disk for any one partition.  As long as
they're being monitored and would be quickly replaced this in practice
works well.  If there could be lapses in monitoring or time to replace then
it is perhaps safer to go with more redundancy or an alternative RAID type.

I'd say do the OS, app installs w/user and audit db stuff, application logs
on one physical RAID volume.  Have a dedicated physical volume for the flow
file repository.  It will not be able to use all the space but it certainly
could benefit from having no other contention.  This could be a great thing
to have SSDs for actually.  And for the remaining volumes split them up for
content and provenance as you have.  You get to make the overall
performance versus retention decision.  Frankly, you have a great system to
work with and I suspect you're going to see excellent results anyway.

Conservatively speaking expect say 50MB/s of throughput per volume in the
content repository so if you end up with 8 of them could achieve upwards of
400MB/s sustained.  You'll also then want to make sure you have a good 10G
based network setup as well.  Or, you could dial back on the speed tradeoff
and simply increase retention or disk loss tolerance.  Lots of ways to play
the game.

There are no published SSD vs HDD performance benchmarks that I am aware of
though this is a good idea.  Having a hybrid of SSDs and HDDs could offer a
really solid performance/retention/cost tradeoff.  For example having SSDs
for the OS/logs/provenance/flowfile with HDDs for the content - that would
be quite nice.  At that rate to take full advantage of the system you'd
need to have very strong network infrastructure between NiFi and any
systems it is interfacing with  and your flows would need to be well tuned
for GC/memory efficiency.

Thanks
Joe

On Thu, Oct 13, 2016 at 2:50 AM, Ali Nazemian  wrote:

> Dear Nifi Users/ developers,
> Hi,
>
> I was wondering is there any benchmark about the question that is it
> better to dedicate disk control to Nifi or using RAID for this purpose? For
> example, which of these scenarios is recommended from the performance point
> of view?
> Scenario 1:
> 24 disk in total
> 2 disk- raid 1 for OS and fileflow repo
> 2 disk- raid 1 for provenance repo1
> 2 disk- raid 1 for provenance repo2
> 2 disk- raid 1 for content repo1
> 2 disk- raid 1 for content repo2
> 2 disk- raid 1 for content repo3
> 2 disk- raid 1 for content repo4
> 2 disk- raid 1 for content repo5
> 2 disk- raid 1 for content repo6
> 2 disk- raid 1 for content repo7
> 2 disk- raid 1 for content repo8
> 2 disk- raid 1 for content repo9
>
>
> Scenario 2:
> 24 disk in total
> 2 disk- raid 1 for OS and fileflow repo
> 4 disk- raid 10 for provenance repo1
> 18 disk- raid 10 for content repo1
>
> Moreover, is there any benchmark for SSD vs HDD performance for Nifi?
> Thank you very much.
>
> Best regards,
> Ali
>

Re: Nifi hardware recommendation

[Joe Witt]

Ali,

I agree with your assumption.  It would be great to test that out and
provide some numbers but intuitively I agree.

I could envision certain scatter/gather data flows that could challenge
that sequential access assumption but honestly with how awesome disk
caching is in Linux these days in think practically speaking this is the
right way to think about it.

Thanks
Joe

On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian  wrote:

> Dear Joe,
>
> Thank you very much. That was a really great explanation.
> I investigated the Nifi architecture, and it seems that most of the
> read/write operations for flow file repo and provenance repo are random.
> However, for content repo most of the read/write operations are sequential.
> Let's say cost does not matter. In this case, even choosing SSD for content
> repo can not provide huge performance gain instead of HDD. Am I right?
> Hence, it would be better to spend content repo SSD money on network
> infrastructure.
>
> Best regards,
> Ali
>
> On Thu, Oct 13, 2016 at 10:22 PM, Joe Witt  wrote:
>
>> Ali,
>>
>> You have a lot of nice resources to work with there.  I'd recommend the
>> series of RAID-1 configuration personally provided you keep in mind this
>> means you can only lose a single disk for any one partition.  As long as
>> they're being monitored and would be quickly replaced this in practice
>> works well.  If there could be lapses in monitoring or time to replace then
>> it is perhaps safer to go with more redundancy or an alternative RAID type.
>>
>> I'd say do the OS, app installs w/user and audit db stuff, application
>> logs on one physical RAID volume.  Have a dedicated physical volume for the
>> flow file repository.  It will not be able to use all the space but it
>> certainly could benefit from having no other contention.  This could be a
>> great thing to have SSDs for actually.  And for the remaining volumes split
>> them up for content and provenance as you have.  You get to make the
>> overall performance versus retention decision.  Frankly, you have a great
>> system to work with and I suspect you're going to see excellent results
>> anyway.
>>
>> Conservatively speaking expect say 50MB/s of throughput per volume in the
>> content repository so if you end up with 8 of them could achieve upwards of
>> 400MB/s sustained.  You'll also then want to make sure you have a good 10G
>> based network setup as well.  Or, you could dial back on the speed tradeoff
>> and simply increase retention or disk loss tolerance.  Lots of ways to play
>> the game.
>>
>> There are no published SSD vs HDD performance benchmarks that I am aware
>> of though this is a good idea.  Having a hybrid of SSDs and HDDs could
>> offer a really solid performance/retention/cost tradeoff.  For example
>> having SSDs for the OS/logs/provenance/flowfile with HDDs for the content -
>> that would be quite nice.  At that rate to take full advantage of the
>> system you'd need to have very strong network infrastructure between NiFi
>> and any systems it is interfacing with  and your flows would need to be
>> well tuned for GC/memory efficiency.
>>
>> Thanks
>> Joe
>>
>> On Thu, Oct 13, 2016 at 2:50 AM, Ali Nazemian 
>> wrote:
>>
>>> Dear Nifi Users/ developers,
>>> Hi,
>>>
>>> I was wondering is there any benchmark about the question that is it
>>> better to dedicate disk control to Nifi or using RAID for this purpose? For
>>> example, which of these scenarios is recommended from the performance point
>>> of view?
>>> Scenario 1:
>>> 24 disk in total
>>> 2 disk- raid 1 for OS and fileflow repo
>>> 2 disk- raid 1 for provenance repo1
>>> 2 disk- raid 1 for provenance repo2
>>> 2 disk- raid 1 for content repo1
>>> 2 disk- raid 1 for content repo2
>>> 2 disk- raid 1 for content repo3
>>> 2 disk- raid 1 for content repo4
>>> 2 disk- raid 1 for content repo5
>>> 2 disk- raid 1 for content repo6
>>> 2 disk- raid 1 for content repo7
>>> 2 disk- raid 1 for content repo8
>>> 2 disk- raid 1 for content repo9
>>>
>>>
>>> Scenario 2:
>>> 24 disk in total
>>> 2 disk- raid 1 for OS and fileflow repo
>>> 4 disk- raid 10 for provenance repo1
>>> 18 disk- raid 10 for content repo1
>>>
>>> Moreover, is there any benchmark for SSD vs HDD performance for Nifi?
>>> Thank you very much.
>>>
>>> Best regards,
>>> Ali
>>>
>>
>>
>
>
> --
> A.Nazemian
>

Re: NiFi for backup solution

[Joe Witt]

Rai,

NiFi can certainly be used for some data replication scenarios and
quite often is.  If you can treat the source like a continuous data
source then there is some way to keep state about what has been pulled
already, what has changed or needs yet to be pulled, and it can just
keep running then generally speaking it will work out well.  Depending
on how the flow is setup, error conditions that can occur in remote
delivery, and cluster topology NiFi won't be able to ensure the order
that data is received is the order in which data is delivered.  So, if
you need to ensure data is copied in precisely the same order (like
log replication) and each object/message/event is on the order of KBs
in size then I'd recommend looking at Apache Kafka and Kafka Connect's
support for keeping things ordered within the same partition of the
same topic.

Thanks
Joe

On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr  wrote:
> Hi All,
> I am learning NiFi as well as trying to deploy it in production for few  use
> cases. One of the use case is ETL and another use case is, using NiFi as a
> backup solution, where it takes the data from one source and moves to
> another database|file. Is anyone using NiFi for this purpose? Does NiFi
> support incremental data move?
> It would be awesome if someone can point me to right documentation.
> Thanks
> Rai

Re: NiFi for backup solution

[Joe Witt]

You'd only need to do that if you have strict ordering requirements like
reading directly from a transaction log and replicating it.  If yes I'd
skip nifi unless your also doing other cases with it.

Sounds like Matts path gets you going though so that might work out just
fine.

Thanks
Joe

On Oct 13, 2016 11:25 AM, "Gop Krr"  wrote:

> Thanks Joe and Matt.
> @Joe, based on your comment, I need to use NiFi as a producer which puts
> the data on Kafka queue and then have NiFi consumer, which writes the data
> back to the destination. Is my understanding correct?
>
> @Matt, My use case is for the DynamoDB. I will look into whether
> incremental copy is supported for Dynamodb.
> Thanks again and felt so good to see the vibrant community. I got my
> questions answered within five minutes. Kudos to NiFi community.
>
> On Thu, Oct 13, 2016 at 8:17 AM, Matt Burgess 
> wrote:
>
>> Rai,
>>
>> There are incremental data movement processors in NiFi depending on
>> your source/target. For example, if your sources are files, you can
>> use ListFile in combination with FetchFile, the former will keep track
>> of which files it has found thus far, so if you put new files into the
>> location (or update existing ones), only those new/updated files will
>> be processed the next time.
>>
>> For database (RDBMS) sources, there are the QueryDatabaseTable and
>> GenerateTableFetch processors, which support the idea of "maximum
>> value columns", such that for each of said columns, the processor(s)
>> will keep track of the maximum value observed in that column, then for
>> future executions of the processor, only rows whose values in those
>> columns exceed the currently-observed maximum will be retrieved, then
>> the maximum will be updated, and so forth.
>>
>> The Usage documentation for these processors can be found at
>> https://nifi.apache.org/docs.html (left-hand side under Processors).
>>
>> Regards,
>> Matt
>>
>> On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr  wrote:
>> > Hi All,
>> > I am learning NiFi as well as trying to deploy it in production for
>> few  use
>> > cases. One of the use case is ETL and another use case is, using NiFi
>> as a
>> > backup solution, where it takes the data from one source and moves to
>> > another database|file. Is anyone using NiFi for this purpose? Does NiFi
>> > support incremental data move?
>> > It would be awesome if someone can point me to right documentation.
>> > Thanks
>> > Rai
>>
>
>

Re: Nifi hardware recommendation

[Joe Witt]

I'd also add to Mark's great reply that another good use of RAM beyond the
HEAP and disk caching and avoiding swapping is that you can do things like
off-heap native storage of things like reference datasets that can wired
into NiFi flows for high speed enrichment where you can even do hot
swapping of older and newer versions of those references sets.

On Fri, Oct 14, 2016 at 8:41 AM, Mark Payne  wrote:

> Hi Ali,
>
> Typically, we see people using a 4-8 GB heap with NiFi. 8 GB is pretty
> typical for a flow that is expected to have
> pretty high throughput in terms of the number of FlowFiles, or a large
> number of processors. However, one thing
> that you will want to consider in terms of RAM is disk caching. While NiFi
> may not use a huge amount of RAM
> directly, the operating system's disk cache is immensely valuable. Because
> the content of FlowFiles is written
> to disk, having a small amount of RAM can result in the next processor
> needing to read that content from disk.
> However, with a sufficient amount of RAM, you will see by looking at
> operating system metrics such as (iostat -xmh 5, for linux)
> that NiFi almost never reads FlowFile content from disk. Instead, it is
> able to get all it needs from the disk cache.
> Frequently querying provenance data also shows a huge difference in
> performance if you have enough RAM.
>
> So the ideal case, I would say, is to have enough RAM for NiFi's heap, as
> well as the content size of all FlowFiles
> that will be actively in your flow at once, plus all other things that
> need to go on, on that box. That said, NiFi should
> certainly work fine reading the content from disk if it needs to - just
> with lower performance.
>
> Does this answer your question?
>
> Thanks
> -Mark
>
>
> On Oct 13, 2016, at 7:47 PM, Ali Nazemian  wrote:
>
> Hi,
>
> I have another question regarding the hardware recommendation. As far as I
> found out, Nifi uses on-heap memory currently, and it will not try to load
> the whole object in memory. From the garbage collection perspective, it is
> not recommended to dedicate more than 8-10 GB to JVM heap space. In this
> case, may I say spending money on system memory is useless? Probably 16 GB
> per each system is enough according to this architecture. Unless some
> architecture changes appear in the future to use off-heap memory as well.
> However, I found some articles about best practices, and in terms of memory
> recommendation it does not make sense. Would you please clarify this part
> for me?
> Thank you very much.
>
> Best regards,
> Ali
>
>
> On Thu, Oct 13, 2016 at 11:38 PM, Ali Nazemian 
> wrote:
>
>> Thank you very much.
>> I would be more than happy to provide some benchmark results after the
>> implementation.
>> Sincerely yours,
>> Ali
>>
>> On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt  wrote:
>>
>>> Ali,
>>>
>>> I agree with your assumption.  It would be great to test that out and
>>> provide some numbers but intuitively I agree.
>>>
>>> I could envision certain scatter/gather data flows that could challenge
>>> that sequential access assumption but honestly with how awesome disk
>>> caching is in Linux these days in think practically speaking this is the
>>> right way to think about it.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian 
>>> wrote:
>>>
>>>> Dear Joe,
>>>>
>>>> Thank you very much. That was a really great explanation.
>>>> I investigated the Nifi architecture, and it seems that most of the
>>>> read/write operations for flow file repo and provenance repo are random.
>>>> However, for content repo most of the read/write operations are sequential.
>>>> Let's say cost does not matter. In this case, even choosing SSD for content
>>>> repo can not provide huge performance gain instead of HDD. Am I right?
>>>> Hence, it would be better to spend content repo SSD money on network
>>>> infrastructure.
>>>>
>>>> Best regards,
>>>> Ali
>>>>
>>>> On Thu, Oct 13, 2016 at 10:22 PM, Joe Witt  wrote:
>>>>
>>>>> Ali,
>>>>>
>>>>> You have a lot of nice resources to work with there.  I'd recommend
>>>>> the series of RAID-1 configuration personally provided you keep in mind
>>>>> this means you can only lose a single disk for any one partition.  As long
>>>>> as they're being monitored and would be quickly replaced this in p

Re: Nifi hardware recommendation

[Joe Witt]

Russ,

You can definitely find a lot of material on the Internet about Java heap
sizes, types of garbage collectors, application usage patterns.  By all
means please do experiment with different sizes appropriate for your case.
We're not saying NiFi itself has any problem with large heaps.

Thanks
Joe

On Fri, Oct 14, 2016 at 12:44 PM, Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Ali,
>
> "not recommended to dedicate more than 8-10 GM to JVM heap space" by whom?
> Do you have links/references establishing this? I couldn't find anyone
> saying this or why.
>
> Russ
>
> On 10/13/2016 05:47 PM, Ali Nazemian wrote:
>
> Hi,
>
> I have another question regarding the hardware recommendation. As far as I
> found out, Nifi uses on-heap memory currently, and it will not try to load
> the whole object in memory. From the garbage collection perspective, it is
> not recommended to dedicate more than 8-10 GB to JVM heap space. In this
> case, may I say spending money on system memory is useless? Probably 16 GB
> per each system is enough according to this architecture. Unless some
> architecture changes appear in the future to use off-heap memory as well.
> However, I found some articles about best practices, and in terms of memory
> recommendation it does not make sense. Would you please clarify this part
> for me?
> Thank you very much.
>
> Best regards,
> Ali
>
>
> On Thu, Oct 13, 2016 at 11:38 PM, Ali Nazemian 
> wrote:
>
>> Thank you very much.
>> I would be more than happy to provide some benchmark results after the
>> implementation.
>> Sincerely yours,
>> Ali
>>
>> On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt < 
>> joe.w...@gmail.com> wrote:
>>
>>> Ali,
>>>
>>> I agree with your assumption.  It would be great to test that out and
>>> provide some numbers but intuitively I agree.
>>>
>>> I could envision certain scatter/gather data flows that could challenge
>>> that sequential access assumption but honestly with how awesome disk
>>> caching is in Linux these days in think practically speaking this is the
>>> right way to think about it.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian 
>>> wrote:
>>>
>>>> Dear Joe,
>>>>
>>>> Thank you very much. That was a really great explanation.
>>>> I investigated the Nifi architecture, and it seems that most of the
>>>> read/write operations for flow file repo and provenance repo are random.
>>>> However, for content repo most of the read/write operations are sequential.
>>>> Let's say cost does not matter. In this case, even choosing SSD for content
>>>> repo can not provide huge performance gain instead of HDD. Am I right?
>>>> Hence, it would be better to spend content repo SSD money on network
>>>> infrastructure.
>>>>
>>>> Best regards,
>>>> Ali
>>>>
>>>> On Thu, Oct 13, 2016 at 10:22 PM, Joe Witt < 
>>>> joe.w...@gmail.com> wrote:
>>>>
>>>>> Ali,
>>>>>
>>>>> You have a lot of nice resources to work with there.  I'd recommend
>>>>> the series of RAID-1 configuration personally provided you keep in mind
>>>>> this means you can only lose a single disk for any one partition.  As long
>>>>> as they're being monitored and would be quickly replaced this in practice
>>>>> works well.  If there could be lapses in monitoring or time to replace 
>>>>> then
>>>>> it is perhaps safer to go with more redundancy or an alternative RAID 
>>>>> type.
>>>>>
>>>>> I'd say do the OS, app installs w/user and audit db stuff, application
>>>>> logs on one physical RAID volume.  Have a dedicated physical volume for 
>>>>> the
>>>>> flow file repository.  It will not be able to use all the space but it
>>>>> certainly could benefit from having no other contention.  This could be a
>>>>> great thing to have SSDs for actually.  And for the remaining volumes 
>>>>> split
>>>>> them up for content and provenance as you have.  You get to make the
>>>>> overall performance versus retention decision.  Frankly, you have a great
>>>>> system to work with and I suspect you're going to see excellent results
>>>>> anyway.
>>>>>
>>>>> Conservatively speakin

Re: Nifi hardware recommendation

[Joe Witt]

The validity of that advice depends on a lot of factors.  G1 changed the
game a bit for pause times for sure but you can still see larger pause
times than acceptable for some cases.  In any event I agree that we should
be more careful with how we describe heap usage.

Thanks
Joe

On Oct 14, 2016 7:10 PM, "Russell Bateman" <
russell.bate...@perfectsearchcorp.com> wrote:

Yeah, I spent a bit of time this morning before posting looking for a magic
8-10Gb advisory and generally for GC gotchas related to larger heap sizes
in the 64-bit world, but couldn't find any. We're using 12Gb right now for
NiFi and haven't noticed any trouble. We vaguely conceive of increasing
this amount in the future as needed as our servers tend to run large
amounts of memory.

The statement yesterday on this thread warning against using that much is
what sent me into Google-it mode. I think this advice is a red herring.

Russ

On 10/14/2016 03:03 PM, Corey Flowers wrote:

We actually use heap sizes from 32 to 64Gb for ours but our volumes and
graphs are both extremely large. Although I believe the smaller heap sizes
were a limitation of the garbage collection in Java 7. We also moved to ssd
drives, which did help through put quite a bit. Our systems were actually
requesting the creation and removal of file handles faster than traditional
disks could keep up with (we believe). In addition, unlike with traditional
drives where we tired to minimize caching, we actually forced more disk
caching when we moved to ssds. Still waiting to see the results of that on
our volumes, although it does seemed to have help. Also remember, depending
on how you code them, individual processors can use system memory outside
of the heap. So you need to take that into consideration when designing the
servers.

Sent from my iPhone

On Oct 14, 2016, at 1:36 PM, Joe Witt  wrote:

Russ,

You can definitely find a lot of material on the Internet about Java heap
sizes, types of garbage collectors, application usage patterns.  By all
means please do experiment with different sizes appropriate for your case.
We're not saying NiFi itself has any problem with large heaps.

Thanks
Joe

On Fri, Oct 14, 2016 at 12:44 PM, Russell Bateman  wrote:

> Ali,
>
> "not recommended to dedicate more than 8-10 GM to JVM heap space" by whom?
> Do you have links/references establishing this? I couldn't find anyone
> saying this or why.
>
> Russ
>
> On 10/13/2016 05:47 PM, Ali Nazemian wrote:
>
> Hi,
>
> I have another question regarding the hardware recommendation. As far as I
> found out, Nifi uses on-heap memory currently, and it will not try to load
> the whole object in memory. From the garbage collection perspective, it is
> not recommended to dedicate more than 8-10 GB to JVM heap space. In this
> case, may I say spending money on system memory is useless? Probably 16 GB
> per each system is enough according to this architecture. Unless some
> architecture changes appear in the future to use off-heap memory as well.
> However, I found some articles about best practices, and in terms of memory
> recommendation it does not make sense. Would you please clarify this part
> for me?
> Thank you very much.
>
> Best regards,
> Ali
>
>
> On Thu, Oct 13, 2016 at 11:38 PM, Ali Nazemian 
> wrote:
>
>> Thank you very much.
>> I would be more than happy to provide some benchmark results after the
>> implementation.
>> Sincerely yours,
>> Ali
>>
>> On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt  wrote:
>>
>>> Ali,
>>>
>>> I agree with your assumption.  It would be great to test that out and
>>> provide some numbers but intuitively I agree.
>>>
>>> I could envision certain scatter/gather data flows that could challenge
>>> that sequential access assumption but honestly with how awesome disk
>>> caching is in Linux these days in think practically speaking this is the
>>> right way to think about it.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian 
>>> wrote:
>>>
>>>> Dear Joe,
>>>>
>>>> Thank you very much. That was a really great explanation.
>>>> I investigated the Nifi architecture, and it seems that most of the
>>>> read/write operations for flow file repo and provenance repo are random.
>>>> However, for content repo most of the read/write operations are sequential.
>>>> Let's say cost does not matter. In this case, even choosing SSD for content
>>>> repo can not provide huge performance gain instead of HDD. Am I right?
>>>> Hence, it would be better to spend content repo SSD money on network
>>>> infrastruct

Re: Calculating the theoretical throughput of a Nifi server

[Joe Witt]

Ali

I suspect bottlenecks in the software itself and the flow design will
become a factor before you 800 MB/s. You'd likely hit CPU efficiency
issues before this caused by the flow processors themselves and due to
garbage collection.  Probably the most important factor though will be
the transaction rate and whether the flow is configured to tradeoff
some latency for higher throughput.  So many variables at play but
under idealized conditions and a system like you describe it is
theoretically feasible to hit that value.

Practically speaking I think you'd be looking at a couple hundred MB/s
per server like this on real flows on a sustained basis.

Thanks
Joe

On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian  wrote:
> Dear Nifi users/developers,
> Hi,
>
> I was wondering how can I calculate the theoretical throughput of a Nifi
> server? let's suppose we can eliminate different bottlenecks such as the
> file flow rep and provenance repo bottleneck by using a very high-end SSD.
> Moreover, assume that a very high-end network infrastructure is available.
> In this case, is it possible to reach 800MB throughput per second per each
> server? Suppose each server comes with 24 disk slots. 16 disk slots are used
> for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the
> content repo. Let's say each content repo can achieve 100 MB throughput. May
> I say the total throughput per each server can be 8x100=800MBps?  Is it
> possible to reach this amount of throughput practically?
> Thank you very much.
>
> Best regards,
> Ali

Re: Calculating the theoretical throughput of a Nifi server

[Joe Witt]

Thanks Lee.  Your response was awesome and really made me want to get
hands on a set of boxes like this so we could do some testing.

Thanks
Joe

On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim  wrote:
> Joe,
> Good points regarding throughput on real flows and sustained basis.  My test
> was only pushing one aspect of the system.
>
> That said, I would be interested discussing/developing a more comprehensive
> test flow to capture more real world use cases. I'll check to see if that
> conversation has started.
>
> Thanks,
> Lee
>
>
>
>
>
> Lee Laim
> 610-864-1657
>
> On Oct 17, 2016, at 9:55 PM, Ali Nazemian  wrote:
>
> Dear Joe,
> Thank you very much.
>
> Best regards
>
>
> On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt  wrote:
>>
>> Ali
>>
>> I suspect bottlenecks in the software itself and the flow design will
>> become a factor before you 800 MB/s. You'd likely hit CPU efficiency
>> issues before this caused by the flow processors themselves and due to
>> garbage collection.  Probably the most important factor though will be
>> the transaction rate and whether the flow is configured to tradeoff
>> some latency for higher throughput.  So many variables at play but
>> under idealized conditions and a system like you describe it is
>> theoretically feasible to hit that value.
>>
>> Practically speaking I think you'd be looking at a couple hundred MB/s
>> per server like this on real flows on a sustained basis.
>>
>> Thanks
>> Joe
>>
>> On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian 
>> wrote:
>> > Dear Nifi users/developers,
>> > Hi,
>> >
>> > I was wondering how can I calculate the theoretical throughput of a Nifi
>> > server? let's suppose we can eliminate different bottlenecks such as the
>> > file flow rep and provenance repo bottleneck by using a very high-end
>> > SSD.
>> > Moreover, assume that a very high-end network infrastructure is
>> > available.
>> > In this case, is it possible to reach 800MB throughput per second per
>> > each
>> > server? Suppose each server comes with 24 disk slots. 16 disk slots are
>> > used
>> > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the
>> > content repo. Let's say each content repo can achieve 100 MB throughput.
>> > May
>> > I say the total throughput per each server can be 8x100=800MBps?  Is it
>> > possible to reach this amount of throughput practically?
>> > Thank you very much.
>> >
>> > Best regards,
>> > Ali
>
>
>
>
> --
> A.Nazemian

Re: Calculating the theoretical throughput of a Nifi server

[Joe Witt]

Ali,

Without knowing the details of the data streams, nature of each event
and the operations that will be performed against them, or how the
processors themselves will work, I cannot give you a solid answer.  Do
I think it is possible?  Absolutely.  Do I think there will be hurdles
to overcome to reach and sustain such a rate?  Absolutely.

Thanks
Joe

On Mon, Oct 17, 2016 at 9:28 PM, Lee Laim  wrote:
> Ali,
> I used the pcie for all repos and the PutFile destination.
>
>
>
> On Oct 18, 2016, at 8:38 AM, Ali Nazemian  wrote:
>
> Hi Lee,
>
> I was wondering, did you use PCIe for file flow repo or provenance repo or
> content repo? or all of them?
>
> Joe,
>
> The ETL is not very complicated ETL, so do you think isn't it possible to
> reach 800MBps in production even if I use PCIe for file flow repo? Is it
> worth spending money on PCIe for the file flow repo?
>
> Best regards
>
> On Tue, Oct 18, 2016 at 2:36 AM, Joe Witt  wrote:
>>
>> Thanks Lee.  Your response was awesome and really made me want to get
>> hands on a set of boxes like this so we could do some testing.
>>
>> Thanks
>> Joe
>>
>> On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim  wrote:
>> > Joe,
>> > Good points regarding throughput on real flows and sustained basis.  My
>> > test
>> > was only pushing one aspect of the system.
>> >
>> > That said, I would be interested discussing/developing a more
>> > comprehensive
>> > test flow to capture more real world use cases. I'll check to see if
>> > that
>> > conversation has started.
>> >
>> > Thanks,
>> > Lee
>> >
>> >
>> >
>> >
>> >
>> > Lee Laim
>> > 610-864-1657
>> >
>> > On Oct 17, 2016, at 9:55 PM, Ali Nazemian  wrote:
>> >
>> > Dear Joe,
>> > Thank you very much.
>> >
>> > Best regards
>> >
>> >
>> > On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt  wrote:
>> >>
>> >> Ali
>> >>
>> >> I suspect bottlenecks in the software itself and the flow design will
>> >> become a factor before you 800 MB/s. You'd likely hit CPU efficiency
>> >> issues before this caused by the flow processors themselves and due to
>> >> garbage collection.  Probably the most important factor though will be
>> >> the transaction rate and whether the flow is configured to tradeoff
>> >> some latency for higher throughput.  So many variables at play but
>> >> under idealized conditions and a system like you describe it is
>> >> theoretically feasible to hit that value.
>> >>
>> >> Practically speaking I think you'd be looking at a couple hundred MB/s
>> >> per server like this on real flows on a sustained basis.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian 
>> >> wrote:
>> >> > Dear Nifi users/developers,
>> >> > Hi,
>> >> >
>> >> > I was wondering how can I calculate the theoretical throughput of a
>> >> > Nifi
>> >> > server? let's suppose we can eliminate different bottlenecks such as
>> >> > the
>> >> > file flow rep and provenance repo bottleneck by using a very high-end
>> >> > SSD.
>> >> > Moreover, assume that a very high-end network infrastructure is
>> >> > available.
>> >> > In this case, is it possible to reach 800MB throughput per second per
>> >> > each
>> >> > server? Suppose each server comes with 24 disk slots. 16 disk slots
>> >> > are
>> >> > used
>> >> > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the
>> >> > content repo. Let's say each content repo can achieve 100 MB
>> >> > throughput.
>> >> > May
>> >> > I say the total throughput per each server can be 8x100=800MBps?  Is
>> >> > it
>> >> > possible to reach this amount of throughput practically?
>> >> > Thank you very much.
>> >> >
>> >> > Best regards,
>> >> > Ali
>> >
>> >
>> >
>> >
>> > --
>> > A.Nazemian
>
>
>
>
> --
> A.Nazemian

Re: Nifi processeror's input form

[Joe Witt]

Yari

Definitely sounds like a bug and I agree it is 'unhandy' when stated in the
nicest possible terms.  Do you mind creating a JIRA for this?

Thanks
Joe

On Tue, Oct 18, 2016 at 11:32 AM, Yari Marchetti <
yari.marche...@buongiorno.com> wrote:

> Hello,
> I need to configure several processors with some very long text and, even
> if it's very easy to create them, I'm finding the UI for reading a little
> "unhandy". The reason is that:
>
> - As long as the text is only one line (even if very very long), there's
> no problem: a scrolling bar is created but the resize handle available in
> all major browser is visible so I can simply resize the text area to a
> suitable dimension. Same is true horizontally (I think scroll bar kicks in
> after something like 5 or 6)
>
> ​​
> - But when the text exceed the max dimension both horizontally and
> vertically, the resize handle vanish and there's no way to resize the text
> are anymore.
> -
>
>
>
> - I'm not a CSS/JS expert but as far as I can see it's all due to this
> line:
>
> 
>
> which, if removed, brings back the resize handle.
>
> ​
> So my question is: Is there any way already implemented to resize the
> text, under the circumstances I explained above? or this is an unintended
> bug?
>
> Thanks,
> Yari
>

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

[Joe Witt]

Conrad,

For s2s now you can just point at any of the nodes in the cluster.
Have you tried changing the URL or removing and adding new RPG
entries?

Thanks
Joe

On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
 wrote:
> Hi,
>
> I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>
> 6 nodes with a NCM.
>
> With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> my Remote Process Groups work as they previously did because they were
> configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>
> I have tried converting my NCM to a node but whilst I can get it running
> (sort of) when I try and connect to the cluster I get something like this in
> my logs…
>
>
>
> 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect node to cluster because local flow is different than cluster
> flow.
>
> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> Cluster Coordinator,5,main]: java.lang.NullPointerException
>
> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi
>
> java.lang.NullPointerException: null
>
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> ~[na:1.8.0_51]
>
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>
> 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> Failed to close Leader Selector for Cluster Coordinator
>
> java.lang.IllegalStateException: Already closed or has not been started
>
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> ~[curator-recipes-2.11.0.jar:na]
>
> at
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controll

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

[Joe Witt]

thanks conrad - did get it.  Bryan is being more helpful that I so I
went silent :-)

On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
 wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with 
> one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being 
> available for the ports and therefore couldn’t actually enable the port I 
> needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping 
> getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt"  wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>  wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 
> 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue 
> where none of
> > my Remote Process Groups work as they previously did because they 
> were
> > configured to connect to the NCM (as the RPG url) which now doesn’t 
> exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it 
> running
> > (sort of) when I try and connect to the cluster I get something 
> like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] 
> o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to 
> connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to 
> connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: 
> Proposed
> > Authorizer is not inheritable by the flow controller because of 
> Authorizer
> > differences: Proposed Authorizations do not match current 
> Authorizations
> >
> > at
> > 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] 
> o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

[Joe Witt]

n this – two days
> of frustration with nothing to show for it.
>
>
>
> Any more suggestions please??
>
> Thanks for your patience.
>
> Conrad
>
>
>
>
>
> From: Andy LoPresto 
> Reply-To: "users@nifi.apache.org" 
> Date: Wednesday, 19 October 2016 at 18:24
> To: "users@nifi.apache.org" 
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>> Old Signed by an unknown key
>
> Hi Conrad,
>
>
>
> Bryan is correct that changing the certificates (and the encapsulating
> keystores and truststores) will not affect any data held in the nodes.
>
>
>
> Regenerating everything using the TLS toolkit should hopefully not be too
> challenging, but I am also curious as to why you are getting these handshake
> exceptions now. As Bryan pointed out, adding the following line to
> bootstrap.conf will provide substantial additional log output which should
> help trace the issue.
>
>
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
>
>
> You can also imitate the node connecting to the (previous) NCM via this
> command:
>
>
>
> $ openssl s_client -connect  -debug -state -cert
>  -key  -CAfile
> 
>
>
>
> Where:
>
>
>
>  = the hostname and port of the “NCM”
>  = the public key used to identify the “node” (can be
> exported from the node keystore [1])
>  = the private key used to identify the “node” (can be
> exported from the node keystore via 2 step process)
>  = the public key used to sign the “NCM”
> certificate (could be a 3rd party like Verisign or DigiCert, or an internal
> organization CA if you have one)
>
>
>
> If you’ve already regenerated everything and it works, that’s fine. But if
> you have the time to try and investigate the old certs, we are interested
> and prepared to help. Thanks.
>
>
>
> [1] https://security.stackexchange.com/a/66865/16485
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> alopresto.apa...@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Oct 19, 2016, at 1:03 PM, Bryan Bende  wrote:
>
>
>
> That is definitely weird that it is only an issue on the node that used to
> be the NCM.
>
> Might be worth double checking the keystore and truststore of that one node
> and make sure it has what you would expect, and also double check
> nifi.properties compared to the others to see if anything seems different.
>
>
>
> Changing all of the keystores, truststores, etc should be fine from a data
> perspective...
>
>
>
> If you decide to go that route it would probably be easiest to start back
> over from a security perspective, meaning:
>
> - Stop all the nodes and delete the users.xml and authorizations.xml from
> all nodes
>
> - Configure authorizers.xml with the appropriate initial admin (or legacy
> file) and node identities based on the new certs
>
> - Ensure authorizers.xml is the same on all nodes
>
> - Then restart everything
>
>
>
> Alternatively, you might be able to manually add users for all of the new
> certs before shutting everything down and give them the appropriate
> policies, then restart everything, but requires more manual work to get
> everything lined up.
>
>
>
>
>
> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
>  wrote:
>
> Hi,
>
> As a plan for tomorrow – I have generated new keystores, truststores, client
> certts  etc. for all nodes in my cluster using the
>
>
>
> From: Bryan Bende 
> Reply-To: "users@nifi.apache.org" 
> Date: Wednesday, 19 October 2016 at 15:33
>
>
> To: "users@nifi.apache.org" 
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and
> the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
>  wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> From: Bryan Bende 
> Reply-To: "users@nifi.apache.org" 
> Date: Wednesday, 19 October 2016 at 15:20
> To: "users@nifi.apache.org" 
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt  wrote:
>
> thanks conrad - did get it.  Bryan is being more helpful that I so I
> went silent

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

[Joe Witt]

It's ok.  If you see it again would love to get the stack trace.  NPEs
automatically should turn into JIRAs as they should definitely never
happen.  I'm not suggesting tracking this down will help your s2s/rpg
struggles but there is clearly something we can tighten up there.

On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
 wrote:
> I’ve just tried to replicate but without success. Using a RPG with RAW 
> transfer protocol.
>
> If you want I can try removing the additional lines in my nifi.properties 
> (the ones for http.port and host that I added which appeared to get this 
> working?
>
> Regards
> Conrad
>
>
> On 21/10/2016, 15:24, "Joe Witt"  wrote:
>
> Whoa there can you turn on debug logging in conf/logback.xml by adding
>
> level="DEBUG"/>
>
> Can you submit a JIRA for the log output with the stacktrace for those
> NPEs please.
>
> Thanks
> Joe
>
> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
>  wrote:
> > Hi,
> >
> > I realise this may be getting boring for most but…
> >
> > I didn’t find any resolution to the upgrade so I have cleanly installed
> > v1.0.0 and oddly experienced the same issue with RPGs in that although 
> the
> > RPGs didn’t show any errors (in so much as they had no warning on them 
> and
> > reported that S2S was secure) the errors reported were about not being 
> able
> > to determine other nodes in cluster.
> >
> > This is a section of log that showed an error that I don’t think I saw
> > before but only including here in case one of you fine folks need it 
> for any
> > reason…
> >
> >
> >
> > ERROR [Site-to-Site Worker Thread-194]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with 
> remote
> > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> > 
> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with 
> remote
> > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> > 
> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with 
> remote
> > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> > 
> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with 
> remote
> > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> > 
> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with 
> remote
> > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> > 
> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
> > due to java.lang.NullPointerException; closing connection
> >
> >
> >
> > in my naivety this suggests a problem with the socket (RAW) connection
> > protocol. The relevant section for S2S connection in my nifi.properties 
> is
> >
> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
> > different for each node obviously
> >
> > nifi.remote.input.socket.port=10443
> >
> > nifi.remote.input.secure=true
> >
> > nifi.remote.input.http.enabled=false
> >
> > nifi.remote.input.http.transaction.ttl=30 sec
> >
> >
> >
> > the errors suggest that the port specified here aren’t being used, but 
> some
> > random ports are being used instead. Of course this is complete 
> supposition
> > and probably a red herring.
> >
> >
> >
> > Anyway, I updated my nifi.properties to
> >
> &g

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

[Joe Witt]

Thanks for following up Conrad and sorry for the challenges you went
through getting there.  We have to make this important feature easier
to setup.

Joe

On Mon, Oct 24, 2016 at 8:46 AM, Conrad Crampton
 wrote:
> Hi,
>
> Yes, I think this is the issue.
>
> I have changed the nifi.remote.input.socket.host to be
> nifi.remote.input.host.
>
> I think I have over-thought the comment at the top of the admin docs for
> Site to Site
>
> “Properties named with nifi.remote.input.socket.* are RAW transport protocol
> specific. Similarly, nifi.remote.input.http.* are HTTP transport protocol
> specific properties.
>
> ” and made the leap of ‘logic’ to include
>
> a)   Keep the ….input.socket.host from previous version
>
> b)   Include a …input.http.host
>
> c)   Include a nifi.remote.input.http.port
>
> I don’t suppose b & c matter too much as they are most likely ignored, but
> not having the correct setting for a) probably doesn’t help.
>
> With a vanilla install I still think not having http enables causes the NPE
> when using RAW.
>
> If someone can confirm if there is an outstanding JIRA for this (or fixed on
> master branch) then I won’t record another one. If I do, probably will be a
> bit garbled given the probably root cause as above.
>
>
>
> Thanks again for everyone’s input on this. I have now reverted back to my
> original 0.6.1 version then successfully upgraded to 1.0.0.
>
>
>
> Conrad
>
>
>
> From: Bryan Bende 
> Reply-To: "users@nifi.apache.org" 
> Date: Friday, 21 October 2016 at 16:09
> To: "users@nifi.apache.org" 
> Subject: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Conrad,
>
>
>
> I'm glad you were able to get something working. I am definitely confused by
> the properties that were required though.
>
>
>
> My understanding is that for RAW site-to-site it uses:
>
>
>
> nifi.remote.input.host=
>
> nifi.remote.input.socket.port=
>
>
>
> And for http based site-to-site it uses the web http or https port that you
> have configured (same as the UI).
>
>
>
> The properties you added seem to be slightly different. One thing I noticed
> was that nifi.remote.input.socket.host changed to nifi.remote.input.host in
> 1.0.0, so maybe that was a problem.
>
>
>
> The latest base nifi.properties is here:
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
>
>
>
> On Friday, October 21, 2016, Joe Witt  wrote:
>
> It's ok.  If you see it again would love to get the stack trace.  NPEs
> automatically should turn into JIRAs as they should definitely never
> happen.  I'm not suggesting tracking this down will help your s2s/rpg
> struggles but there is clearly something we can tighten up there.
>
> On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
>  wrote:
>> I’ve just tried to replicate but without success. Using a RPG with RAW
>> transfer protocol.
>>
>> If you want I can try removing the additional lines in my nifi.properties
>> (the ones for http.port and host that I added which appeared to get this
>> working?
>>
>> Regards
>> Conrad
>>
>>
>> On 21/10/2016, 15:24, "Joe Witt"  wrote:
>>
>> Whoa there can you turn on debug logging in conf/logback.xml by
>> adding
>>
>>> level="DEBUG"/>
>>
>> Can you submit a JIRA for the log output with the stacktrace for those
>> NPEs please.
>>
>> Thanks
>> Joe
>>
>> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
>>  wrote:
>> > Hi,
>> >
>> > I realise this may be getting boring for most but…
>> >
>> > I didn’t find any resolution to the upgrade so I have cleanly
>> installed
>> > v1.0.0 and oddly experienced the same issue with RPGs in that
>> although the
>> > RPGs didn’t show any errors (in so much as they had no warning on
>> them and
>> > reported that S2S was secure) the errors reported were about not
>> being able
>> > to determine other nodes in cluster.
>> >
>> > This is a section of log that showed an error that I don’t think I
>> saw
>> > before but only including here in case one of you fine folks need it
>> for any
>> > reason…
>> >
>> >
>> >
>> > ERROR [Site-to-Site Worker Thread-194]
>> > o.a.nifi.remote.Socket

Re: start it ONCE

[Joe Witt]

Alessio,

We don't presently support cron scheduling for the primary node only
processors.  This does come up fairly often and a nice PR/contrib came
in a while ago from the community
https://issues.apache.org/jira/browse/NIFI-401.  It is now getting
some good traction so hopefully it will be in an upcoming release.

Thanks
Joe

On Mon, Oct 24, 2016 at 6:13 AM, Alessio Palma
 wrote:
> Hello All, here is a question which is driving me crazy...
> I'm using the PutHiveQL processor to create some tables and view into
> HIVE. This processor need the SQL to execute into a flowfile.
>
> I'm using a GenerateFlowFile to build an empty flowfile which is later
> filled using the ReplaceText processor.
>
> It Works but the cluster has 3 host and I'm getting the same query 3 times.
>
> If I use the "On primany node" scheduling stretegy into the
> GenerateFlowFile, anything works but I'm no longer able to use the CRON
> scheduling option.
>
> So.. There is a way to produce a SINGLE FlowFile into a cluster using a
> "cron scheduling" option ?
>

Re: start it ONCE

[Joe Witt]

Alessio

It just isn't supported yet as that was never a case that was needed.
Folks would like it now.  So it is a new feature or enhancement rather
than a bug.

It is fine to increase the priority of it.  I just bumped it major.

In either case the key now is to join the discussion as needed to
share your perspective on what you need to do with it and perhaps
contribute in other ways that you might be interested in.  Since there
is solid traction happening now I think you'll be in good shape soon.

Thanks
Joe

On Tue, Oct 25, 2016 at 9:13 AM, Alessio Palma
 wrote:
> I see on https://issues.apache.org/jira/browse/NIFI-401:
>
> priority : Minor
> Type: Improvement
>
> Are we sure about this ?
> I think this is a priority bug.
> Is it just  me who is facing the problem to start ONE flowfile  into the
> cluster?
>
>
> On 24/10/2016 15:01, Joe Witt wrote:
>> Alessio,
>>
>> We don't presently support cron scheduling for the primary node only
>> processors.  This does come up fairly often and a nice PR/contrib came
>> in a while ago from the community
>> https://issues.apache.org/jira/browse/NIFI-401.  It is now getting
>> some good traction so hopefully it will be in an upcoming release.
>>
>> Thanks
>> Joe
>>
>> On Mon, Oct 24, 2016 at 6:13 AM, Alessio Palma
>>  wrote:
>>> Hello All, here is a question which is driving me crazy...
>>> I'm using the PutHiveQL processor to create some tables and view into
>>> HIVE. This processor need the SQL to execute into a flowfile.
>>>
>>> I'm using a GenerateFlowFile to build an empty flowfile which is later
>>> filled using the ReplaceText processor.
>>>
>>> It Works but the cluster has 3 host and I'm getting the same query 3 times.
>>>
>>> If I use the "On primany node" scheduling stretegy into the
>>> GenerateFlowFile, anything works but I'm no longer able to use the CRON
>>> scheduling option.
>>>
>>> So.. There is a way to produce a SINGLE FlowFile into a cluster using a
>>> "cron scheduling" option ?
>>>
>> .
>>

Re: GetDynamoDB Processor

[Joe Witt]

Nice.  James - do you plan to file a JIRA for this or was there already one?

Thanks
Joe

On Wed, Oct 26, 2016 at 1:50 AM, Pablo Lopez
 wrote:
> It worked!
> I had to actually delete the Processor and add a new new. Stopping and
> starting the Processor or the Flow didn't actually work.
>
> Thanks,
> Pablo.
>
> On Wed, Oct 26, 2016 at 3:59 PM, James Wing  wrote:
>>
>> Pablo,
>>
>> Did you by any chance start the GetDynamoDB processor first without the
>> credentials, then stop, configure credentials, and restart?  I suspect there
>> might be a bug where GetDynamoDB caches the client even through stopping and
>> starting the processor.
>>
>> To test this theory, you might try copy/pasting the GetDynamoDB processor
>> and configure the access keys on the new one before starting it.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Oct 25, 2016 at 4:35 PM, Pablo Lopez
>>  wrote:
>>>
>>> Hi,
>>>
>>> I've been trying to get GetDynamoDB processor to work. I've configured
>>> the AccessKey and SecretKey but i'm still getting Exception
>>> MissingAuthenticationTokenException. Any ideas?
>>>
>>> This is the Error from the Bulletin Board.
>>>
>>> GetDynamoDB[id=f14db94a-0157-1000-4a66-5f7d9437a744] Could not process
>>> flowFiles due to service exception : Request is missing Authentication Token
>>> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
>>> MissingAuthenticationTokenException; Request ID:
>>> 3SB9O22VV16LE1BV31A79DD7QBVV4KQNSO5AEMVJF66Q9ASUAAJG)
>>>
>>> Thanks,
>>> Pablo.
>>
>>
>
>
>
> --
> Pablo Lopez.
> Integration Architect
> Integrado Pty Ltd
> M: 044 84 52 479
> pablo.lo...@integrado.com.au

Re: How does Nifi ingest large files?

[Joe Witt]

That is correct.

Thanks
Joe

On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota 
wrote:

> Bryan,
>
> If I have the content repo implementation set to
> org.apache.nifi.controller.repository.VolatileContentRepository, it will
> stream the content in memory, correct?
>
> On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende  wrote:
>
>> Monica,
>>
>> Are you asking what does NiFi do when it picks up a large file from the
>> filesystem using a processor like GetFile?
>>
>> If so, it will stream the content of that file into NiFi's content
>> repository, and create a FlowFile pointing to that content. As far as NiFi
>> is concerned the content is just bytes at this point and has not been
>> changed in anyway from the original file.
>>
>> The content is not held in memory, and the FlowFile can move through many
>> processors without ever accessing the content, unless the processor needs
>> to, and then when accessing the content it is typically done in a streaming
>> fashion (when possible) to avoid loading the large content into memory.
>>
>> There are processors that can then split up the content based on specific
>> data formats, for example SplitText, SplitJSON, SplitAvro, etc.. but it is
>> up to the designer of the flow to do that.
>>
>> -Bryan
>>
>>
>> On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini <
>> monica.francesch...@eng.it> wrote:
>>
>>> Hi,
>>> I'm figuring out how does Nifi ingest large files: does it split them
>>> into chunks or is it a massive load?Can you please, explain the behavior?
>>> Kind regards,
>>> Monica
>>> --
>>>
>>> *Monica Franceschini*
>>> Solution Architecture Manager
>>>
>>> *Big Data Competence Center Engineering Group*
>>> Corso Stati Uniti 23/C, 35127 Padova, Italia
>>> Tel: +39 049.8283547
>>> Fax: +39 049.8692566
>>> Twitter: @twittmonique
>>> www.spagobi.org - www.eng.it  *proud
>>> SpagoBI supporter and contributor*
>>> [image: SpagoBI]
>>>   Respect the environment. Please don't print this e-mail unless you
>>> really need to.
>>>
>>> The information transmitted is intended only for the person or entity to
>>> which it is addressed and may contain confidential and/or privileged
>>> material. Any review, retransmission, dissemination or other use of, or
>>> taking of any action in reliance upon, this information by persons or
>>> entities other than the intended recipient is prohibited. If you received
>>> this in error, please contact the sender and delete the material from any
>>> computer.
>>>
>>
>>
>
>
> --
>
> [image: Payoff, Inc.] 
>
> Jeremy Farbota
> Software Engineer, Data
> jfarb...@payoff.com  • (217) 898-8110 <(949)+430-0630>
>
> I'm a Storyteller. Discover your Financial Personality!
> 
>
> [image: Facebook]   [image: Twitter]
>  [image: Linkedin]
> 
>

Re: How does Nifi ingest large files?

[Joe Witt]

I should add that if you're comfortable with that sort of volatile behavior
a better path to consider is to setup a RAM-Disk and just run a persistent
content repository on that.  It will survive process restarts, give better
memory/heap behavior (by a lot), but you'll lose data on system restarts.

Thanks
Joe

On Thu, Oct 27, 2016 at 11:58 AM, Joe Witt  wrote:

> That is correct.
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota 
> wrote:
>
>> Bryan,
>>
>> If I have the content repo implementation set to
>> org.apache.nifi.controller.repository.VolatileContentRepository, it will
>> stream the content in memory, correct?
>>
>> On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende  wrote:
>>
>>> Monica,
>>>
>>> Are you asking what does NiFi do when it picks up a large file from the
>>> filesystem using a processor like GetFile?
>>>
>>> If so, it will stream the content of that file into NiFi's content
>>> repository, and create a FlowFile pointing to that content. As far as NiFi
>>> is concerned the content is just bytes at this point and has not been
>>> changed in anyway from the original file.
>>>
>>> The content is not held in memory, and the FlowFile can move through
>>> many processors without ever accessing the content, unless the processor
>>> needs to, and then when accessing the content it is typically done in a
>>> streaming fashion (when possible) to avoid loading the large content into
>>> memory.
>>>
>>> There are processors that can then split up the content based on
>>> specific data formats, for example SplitText, SplitJSON, SplitAvro, etc..
>>> but it is up to the designer of the flow to do that.
>>>
>>> -Bryan
>>>
>>>
>>> On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini <
>>> monica.francesch...@eng.it> wrote:
>>>
>>>> Hi,
>>>> I'm figuring out how does Nifi ingest large files: does it split them
>>>> into chunks or is it a massive load?Can you please, explain the behavior?
>>>> Kind regards,
>>>> Monica
>>>> --
>>>>
>>>> *Monica Franceschini*
>>>> Solution Architecture Manager
>>>>
>>>> *Big Data Competence Center Engineering Group*
>>>> Corso Stati Uniti 23/C, 35127 Padova, Italia
>>>> Tel: +39 049.8283547
>>>> Fax: +39 049.8692566
>>>> Twitter: @twittmonique
>>>> www.spagobi.org - www.eng.it <http://www.eng.it/web/eng_en/home> *proud
>>>> SpagoBI supporter and contributor*
>>>> [image: SpagoBI]
>>>>   Respect the environment. Please don't print this e-mail unless you
>>>> really need to.
>>>>
>>>> The information transmitted is intended only for the person or entity
>>>> to which it is addressed and may contain confidential and/or privileged
>>>> material. Any review, retransmission, dissemination or other use of, or
>>>> taking of any action in reliance upon, this information by persons or
>>>> entities other than the intended recipient is prohibited. If you received
>>>> this in error, please contact the sender and delete the material from any
>>>> computer.
>>>>
>>>
>>>
>>
>>
>> --
>>
>> [image: Payoff, Inc.] <http://www.payoff.com/>
>>
>> Jeremy Farbota
>> Software Engineer, Data
>> jfarb...@payoff.com  • (217) 898-8110 <(949)+430-0630>
>>
>> I'm a Storyteller. Discover your Financial Personality!
>> <https://www.payoff.com/quiz>
>>
>> [image: Facebook]  <https://www.facebook.com/payoff> [image: Twitter]
>> <https://www.twitter.com/payoff> [image: Linkedin]
>> <https://www.linkedin.com/company/payoff-com>
>>
>
>

Re: IllegalStateException

[Joe Witt]

Conrad,

Am looking into this now.  Could you please file a JIRA.  Given the
error it is showing there is without any doubt a coding error.

Thanks
Joe

On Thu, Oct 27, 2016 at 9:35 AM, Conrad Crampton
 wrote:
> Hi,
>
> I’m getting this exception thrown by most of my processors for a given
> pipeline,
>
>
>
> “ConvertAvroSchema[id=3b590825-5ef3-38d7-8c26-4e99d24f9d69]
> ConvertAvroSchema[id=3b590825-5ef3-38d7-8c26-4e99d24f9d69] failed to process
> session due to java.lang.IllegalStateException: Cannot create Provenance
> Event Record because FlowFile UUID is not set:
> java.lang.IllegalStateException: Cannot create Provenance Event Record
> because FlowFile UUID is not set”
>
> “UpdateAttribute[id=56df18c0-d525-3cd7-893d-5d9e9c322969]
> UpdateAttribute[id=56df18c0-d525-3cd7-893d-5d9e9c322969] failed to process
> session due to java.lang.IllegalStateException: Cannot create Provenance
> Event Record because FlowFile UUID is not set:
> java.lang.IllegalStateException: Cannot create Provenance Event Record
> because FlowFile UUID is not set”
>
>
>
> This is after my upgrade from 0.6.1 to 1.0.0. The upgrade may be a red
> herring, but I have only just noticed it.
>
> Because of this, I believe my processors are not actually processing
> anything anymore.
>
>
>
> Is there a way to resolve this??
>
>
>
> Thanks
>
> Conrad
>
>
>
>
>
> SecureData, combating cyber threats
>
> 
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email are
> those of the individual and not necessarily of SecureData Europe Ltd. Any
> prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT

Re: nifi is running out of memory

[Joe Witt]

moving dev to bcc

Yes I believe the issue here is that FetchS3 doesn't do chunked
transfers and so is loading all into memory.  I've not verified this
in the code yet but it seems quite likely.  Krish if you can verify
that going with a larger heap gets you in the game can you please file
a JIRA.

Thanks
Joe

On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende  wrote:
> Hello,
>
> Are you running with all of the default settings?
>
> If so you would probably want to try increasing the memory settings in
> conf/bootstrap.conf.
>
> They default to 512mb, you may want to try bumping it up to 1024mb.
>
> -Bryan
>
> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr  wrote:
>>
>> Hi All,
>>
>> I have very simple data flow, where I need to move s3 data from one bucket
>> in one account to another bucket under another account. I have attached my
>> processor configuration.
>>
>>
>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2]
>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service
>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
>>
>> I am very new to NiFi and trying ot get few of the use cases going. I need
>> help from the community.
>>
>> Thanks again
>>
>> Rai
>>
>>
>>
>

Re: nifi is running out of memory

[Joe Witt]

Looking at this line [1] makes me think the FetchS3 processor is
properly streaming the bytes directly to the content repository.

Looking at the screenshot showing nothing out of the ListS3 processor
makes me think the bucket has so many things in it that the processor
or associated library isn't handling that well and is just listing
everything with no mechanism of max buffer size.  Krish please try
with the largest heap you can and let us know what you see.

[1] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java#L107

On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt  wrote:
> moving dev to bcc
>
> Yes I believe the issue here is that FetchS3 doesn't do chunked
> transfers and so is loading all into memory.  I've not verified this
> in the code yet but it seems quite likely.  Krish if you can verify
> that going with a larger heap gets you in the game can you please file
> a JIRA.
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende  wrote:
>> Hello,
>>
>> Are you running with all of the default settings?
>>
>> If so you would probably want to try increasing the memory settings in
>> conf/bootstrap.conf.
>>
>> They default to 512mb, you may want to try bumping it up to 1024mb.
>>
>> -Bryan
>>
>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr  wrote:
>>>
>>> Hi All,
>>>
>>> I have very simple data flow, where I need to move s3 data from one bucket
>>> in one account to another bucket under another account. I have attached my
>>> processor configuration.
>>>
>>>
>>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2]
>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service
>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
>>>
>>> I am very new to NiFi and trying ot get few of the use cases going. I need
>>> help from the community.
>>>
>>> Thanks again
>>>
>>> Rai
>>>
>>>
>>>
>>

Re: How I put the cluster down.

[Joe Witt]

Alessio

You have two clusters here potentially.  The NiFi cluster and the
Hadoop cluster.  Which one went down?

If NiFi went down I'd suspect memory exhaustion issues because other
resource exhaustion issues like full file system, exhausted file
handles, pegged CPU, etc.. tend not to cause it to restart.  If memory
related you'll probably see something in the nifi-app.log.  Try going
with a larger heap as can be controlled in conf/bootstrap.conf.

Thanks
Joe

On Fri, Oct 28, 2016 at 5:55 AM, Alessio Palma
 wrote:
> Hello all,
> yesterday, for a mistake, basically I executed " ls -R / " using the
> ListHDFS processor and the whole cluster gone down ( not just a node ).
>
> Something like this also happened when I was playing with some DO WHILE
> / WHILE DO patterns. I have only the nifi logs and they show the
> heartbeat has been lost. About the CPU LOAD, NETWORK TRAFFIC I have no
> info. Any pointers about where do I have look for the problem's root ?
>
> Today I'm trying to repeat the problems I got with DO/WHILE, nothing bad
> is happening although CPU LOAD is enough high and NETWORK  TRAFFIC
> increased up to 282 Kb/sec.
>
> Of course I can redo the "ls -R /" on production, however I like to
> avoid it since there are already some ingestion flows running.
>
> AP

Re: Trouble with PublishKafka10

[Joe Witt]

Daniel

Can you please show the error information you are seeing?  Should be
available via bulletins through the UI, rest API, or in the
logs/nifi-app.log.

Thanks
Joe

On Oct 30, 2016 10:27 AM, "Daniel Einspanjer" <
daniel.einspan...@designet.com> wrote:

> I have been trying to get a simple flow going that reads from a DB table
> and writes to a local Kafka instance.
>
>
> I was able to get it to work with a GenerateFlowFile processor, but
> whenever I try to write real data, it is failing.
>
>
> Here is a link to a template that shows what I'm doing.
>
>
> I'd greatly appreciate a tip if you have one.
>
>
> https://gist.github.com/deinspanjer/5936402bb1ab2eea48fba166791fc26b
>
>
> -Daniel
>

Re: Trouble with PublishKafka10

[Joe Witt]

Daniel

How large is each object you are trying to write to Kafka?

Since it was working based on a different source of data but now
problematic that is the direction I am looking in in terms of changes.
Output of the db stuff could need demarcation for example.

Thanks
Joe

On Oct 30, 2016 10:46 AM, "Daniel Einspanjer" <
daniel.einspan...@designet.com> wrote:

Argh, one more follow-up regarding environment..

I am working with the following components on a MacBook Pro running El
Capitan 10.11.6:

brew install kafka -- installs zk 3.4.8 and kafka 0.10.0.1
brew install nifi -- installs nifi stable 1.0.0

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
kafka-server-start /usr/local/etc/kafka/server.properties  # one change
made to server.properties to enable topic deletion
nifi start

On Sun, Oct 30, 2016 at 10:42 AM Daniel Einspanjer <
daniel.einspan...@designet.com> wrote:

> Sorry, I posted the information in the description of the template, but
> that isn't the most obvious place.
>
> 2016-10-29 23:52:49,629 WARN [Timer-Driven Process Thread-1]
> o.a.n.p.kafka.pubsub.PublishKafka_0_10 
> PublishKafka_0_10[id=12b0a84d-0158-1000-ce29-967712bf5ab6]
> Timed out while waiting for acks from Kafka
>
> On Sun, Oct 30, 2016 at 10:33 AM Joe Witt  wrote:
>
> Daniel
>
> Can you please show the error information you are seeing?  Should be
> available via bulletins through the UI, rest API, or in the
> logs/nifi-app.log.
>
>
>

Re: ExecuteSQL just once

[Joe Witt]

Daniel

We don't have mechanisms provided for one time execution at this point.
Everything is oriented toward continuous execution.  That said this is
starting to come up more and more so seems like we should offer run once
scheduling or something like that.

Would that give you what you are looking for?  Can you talk about your use
case a bit more ?

Thanks
Joe

On Oct 30, 2016 7:07 PM, "Daniel Einspanjer" 
wrote:
>
> If I put a SQL statement into an ExecuteSQL processor along the lines of:
>
> SELECT * FROM foo WHERE thedate >= '2016-01-01'
>
> and I build a flow that does something with that data, I'f found if I
leave the ExecuteSQL processor running, it will continually run the query,
resulting in lots of duplicate data.
>
> What is the proper way to handle something that would be sort of a one
time event?
>
> Right now, I'm just setting a long run schedule time and making sure I
turn the step off after it finishes the first run.
>
> -Daniel

Re: nifi is running out of memory

[Joe Witt]

t;>>
>>>>> at
>>>>> org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.java:106)
>>>>> ~[nifi-aws-processors-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>>> [nifi-api-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> [na:1.8.0_101]
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>>>
>>>>>
>>>>> On Fri, Oct 28, 2016 at 6:31 AM, Pierre Villard
>>>>>  wrote:
>>>>>>
>>>>>> Quick remark: the fix has also been merged in master and will be in
>>>>>> release 1.1.0.
>>>>>>
>>>>>> Pierre
>>>>>>
>>>>>> 2016-10-28 15:22 GMT+02:00 Gop Krr :
>>>>>>>
>>>>>>> Thanks Adam. I will try 0.7.1 and update the community on the
>>>>>>> outcome. If it works then I can create a patch for 1.x
>>>>>>> Thanks
>>>>>>> Rai
>>>>>>>
>>>>>>> On Thu, Oct 27, 2016 at 7:41 PM, Adam Lamar 
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hey All,
>>>>>>>>
>>>>>>>> I believe OP is running into a bug fixed here:
>>>>>>>> https://issues.apache.org/jira/browse/NIFI-2631
>>>>>>>>
>>>>>>>> Basically, ListS3 attempts to commit all the files it finds
>>>>>>>> (potentially 100k+) at once, rather than in batches. NIFI-2631
>>>>>>>> addresses the issue. Looks like the fix is out in 0.7.1 but not yet
>>>>>>>> in
>>>>>>>> a 1.x release.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 27, 2016 at 7:59 PM, Joe Witt 
>>>>>>>> wrote:
>>>>>>>> > Looking at this line [1] makes me think the FetchS3 processor is
>>>>>>>> > properly streaming the bytes directly to the content repository.
>>>>>>>> >
>>>>>>>> > Looking at the screenshot showing nothing out of the ListS3
>>>>>>>> > processor
>>>>>>>> > makes me think the bucket has so many things in it that the
>>>>>>>> > processor
>>>>>>>> > or associated library isn't handling that well and is just listing
>>>>>>>> > everything with no mechanism of max buffer size.  Krish please try
>>>>>>>> > with the largest heap you can and let us know what you see.
>>>>>>>> >
>>>>>>>> > [1]
>>>>>>>> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java#L107
>>>>>>>> >
>>>>>>>> > On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt 
>>>>>>>> > wrote:
>>>>>>>> >> moving dev to bcc
>>>>>>>> >>
>>>>>>>> >> Yes I believe the issue here is that FetchS3 doesn't do chunked
>>>>>>>> >> transfers and so is loading all into memory.  I've not verified
>>>>>>>> >> this
>>>>>>>> >> in the code yet but it seems quite likely.  Krish if you can
>>>>>>>> >> verify
>>>>>>>> >> that going with a larger heap gets you in the game can you please
>>>>>>>> >> file
>>>>>>>> >> a JIRA.
>>>>>>>> >>
>>>>>>>> >> Thanks
>>>>>>>> >> Joe
>>>>>>>> >>
>>>>>>>> >> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende 
>>>>>>>> >> wrote:
>>>>>>>> >>> Hello,
>>>>>>>> >>>
>>>>>>>> >>> Are you running with all of the default settings?
>>>>>>>> >>>
>>>>>>>> >>> If so you would probably want to try increasing the memory
>>>>>>>> >>> settings in
>>>>>>>> >>> conf/bootstrap.conf.
>>>>>>>> >>>
>>>>>>>> >>> They default to 512mb, you may want to try bumping it up to
>>>>>>>> >>> 1024mb.
>>>>>>>> >>>
>>>>>>>> >>> -Bryan
>>>>>>>> >>>
>>>>>>>> >>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr 
>>>>>>>> >>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Hi All,
>>>>>>>> >>>>
>>>>>>>> >>>> I have very simple data flow, where I need to move s3 data from
>>>>>>>> >>>> one bucket
>>>>>>>> >>>> in one account to another bucket under another account. I have
>>>>>>>> >>>> attached my
>>>>>>>> >>>> processor configuration.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2]
>>>>>>>> >>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>>>>>>>> >>>> Thread[Flow Service
>>>>>>>> >>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap
>>>>>>>> >>>> space
>>>>>>>> >>>>
>>>>>>>> >>>> I am very new to NiFi and trying ot get few of the use cases
>>>>>>>> >>>> going. I need
>>>>>>>> >>>> help from the community.
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks again
>>>>>>>> >>>>
>>>>>>>> >>>> Rai
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: NiFi Real World Usage Queries

[Joe Witt]

Responses provided in-line.  Apologies for the delay and brevity on
some of these.  Great questions all worth discsussion.

On Sat, Aug 20, 2016 at 1:46 AM, Gunjan Dave  wrote:
> Hello NiFi Team,
> Firstly, let me congratulate you on the fantastic work that your team is
> doing and also on the release of 1.0 (beta) version. Your team is doing
> great and NiFi shows many possibilites and  use cases, currently not
> envisioned nor currently marketed for.
>
> To be able to use NiFi to fullest capacities can I request your help in
> addressing some of my queries?
>
> 1) Parallel Development
>   How would a team of 50 developers work parallely on designing and
> implementing common data flow? How would version control work? How can we
> ensure that work of one developer in not overridden by another? Can we
> version control the flow.xml file outside of nifi?
> From microsoft biztalk background, where orchestration data flow design xml
> is version controlled in TFS, can we do the same for flow.xml in git?
> Can developers merge the local flow.xml externally and deploy the merged
> flow.xml.gz into UAT?

The development team would generally build flows locally and produce
templates.  Those templates then can be deployed to production.  In
production the expectation would be that the flow can be tweaked
directly or can be replaced by a new version of the template.  There
is no worry about developers overriding eachother unless they are
changing the same component at the same time and in this case the
system will reject the second editor but that would be a pretty rare
case.

> 2) User Authorization
>   I understand from docs that user authorization needs to be managed in
> authorization xml file? But is this manual process? Like I would have some
> 50 to 100 developers using it, i would be pain to manage the authorization
> manually via xml file?
> Is there some other way like ldap for authorization?

Right this would be cumbersome if using our built-in file-based
mechanism.  You could also leverage integration with an external
authorization system such as Apache Ranger (incubating) or implement
an alternative authorization provider which leverages some existing
authorization system you have.

>
> 3) sensitive data like dbcp connection passwords in external file in
> encrypted form, is it possible and how?

Sensitive values like dbcp connection passwords are encrypted by nifi
whenever/wherever the flow configuration is serialized and are not
sent back to the client again as they would not need those values.  So
it should already be well handled.

>
> 4) Functional testing the entire flow
>   I understand each processor has run processor mock, but is there a way
> i can test the entire flow in automated manner before going live?
> Can there be an option of reading flow xml file into a separate test harness
> and execute specific portions of the flow end to end?
> Can minifi used for this purpose? I do understand that minifi is created for
> different purpose but can it also be used for some form of automated
> testing.

Generally this is what local testing and/or templates are helpful for.
We should do more though and offer a programmatic way perhaps.

> 5) how does the debug flow processor work? Could not find enough
> documentation there?

This is really for debugging framework behavior.

>
> 7) Is there a detailed infrastructure recommendation and setup guide for
> NiFi cluster? Like best practises and some sample setup patterns.

Take a look at the admin guide, best practices/install guide, and user
guide.  It should help you get on your way.

>
> Thanking in advance.
>
> Thanks
> Gunjan Dave
>
>
>
>
>
>

Re: Nifi partition data by date

[Joe Witt]

I agree with James.  The general pattern here is

Split with Grouping:
  Take a look at RouteText.  This allows you to efficiently split up
line oriented data into groups based on matching values rather than
spilt text which will be a line for line split.

Merge Grouped Data:
  MergeContent processor will do the trick and you can use correlation
feature to align only those which are from the same group/pattern.

Write to destination:
  You can write directly to HDFS using PutHDFS or you can prepare the
data and write to Hive.

Thanks
Joe

On Wed, Nov 2, 2016 at 9:01 PM, James Wing  wrote:
> This is absolutely possible.  A sample sequence of processors might include:
>
> 1. UpdateAttribute - to extract a record date from the flowfile content into
> an attribute, 'recordgroup' for example
> 2. MergeContent - to group related records together, setting the Correlation
> Attribute Name property to use 'recordgroup'
> 3. UpdateAttribute - (optional) to apply the 'recordgroup' attribute to the
> 'path' and/or 'filename' attributes, depending on how you do #4.  May be
> useful to get customized filenames with extensions.
> 4. Put* - to write the grouped file to storage (PutFile, PutHDFS,
> PutS3Object, etc.).  With PutHDFS for example, use Expression Language in
> the Directory property to apply your grouping - like
> '/tmp/hive/records/${recordgroup}' to get '/tmp/hive/records/2016-01-01'.
>
> In concept, it's that simple.  The #2 MergeContent step can be more
> complicated as you consider how many files should be output from the stream,
> how big they should be, how frequently, and how many bins are likely to be
> open collecting files at any one time.  You might also consider compressing
> the files.
>
> Thanks,
>
> James
>
> On Wed, Nov 2, 2016 at 5:34 PM, Santiago Ciciliani
>  wrote:
>>
>> I'm trying to split a stream of data into multiple different files based
>> on the content date.
>>
>> So imagine that you are receiving streams of logs and you want to save as
>> a Hive partitioned table so for example all records with date 2016-01-01
>> into directory dt=2016-01-01.
>>
>> Is this even possible?
>>
>> Thanks
>>
>

Re: PutHiveStreaming has only one concurrent task

[Joe Witt]

Olav,

I'll let someone more familiar with that processor comment on the best
usage pattern but will say that for any processor which does not offer
controls on the number of concurrent tasks then it means that processor is
meant to run serially by design.

Thanks
Joe

On Sun, Nov 6, 2016 at 7:06 PM, Olav Jordens <
olav.jord...@2degreesmobile.co.nz> wrote:

> Hi,
>
>
>
> I have noticed that it is not possible to change the number of concurrent
> tasks on the Scheduling tab of the PutHiveStreaming processor from the
> default value of 1. Is there a reason for this, or is it a bug? What would
> be the consequence if I try to get around this by using a DistributeLoad
> processor followed by a sequence of PutHiveStreaming processors, which
> would all be streaming into the same Hive table?
>
>
>
> Thanks,
>
> Olav
>
>
>
>
> Olav Jordens
> Senior ETL Developer
> Two Degrees Mobile Limited
> ===
> (M) 022 620 2429
> (P) 09 919 7000
> www.2degreesmobile.co.nz
>
> Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland |
> New Zealand |
> PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9
> 919 7001
>
>
>
>
> --
>
> Disclaimer
> The e-mail and any files transmitted with it are confidential and may
> contain privileged or copyright information. If you are not the intended
> recipient you must not copy, distribute, or use this e-mail or the
> information contained in it for any purpose other than to notify us of the
> error. If you have received this message in error, please notify the sender
> immediately, by email or phone (+64 9 919 7000) and delete this email
> from your system. Any views expressed in this message are those of the
> individual sender, except where the sender specifically states them to be
> the views of Two Degrees Mobile Limited. We do not guarantee that this
> material is free from viruses or any other defects although due care has
> been taken to minimize the risk
>
>
>
>

Re: PutHiveStreaming error after implementing HA

[Joe Witt]

Olav

It is definitely time for the 1.1.0 release to begin.  We've got quite a
queue of PRs and reviews underway.  As they burn down we'll initiate the
release process.  Based on past release cycles I'd say we're a week or two
from the vote.

Thanks
Joe

On Sun, Nov 6, 2016 at 7:03 PM, Olav Jordens <
olav.jord...@2degreesmobile.co.nz> wrote:

> Hi,
>
> I have bumped into the error in  https://issues.apache.org/
> jira/browse/NIFI-2873 after implementing HA on my cluster and running the
> PutHiveStreaming processor. The bug is fixed in 1.1.0. When will 1.1.0 be
> released? Alternatively, is there a work-around for this problem? (The
> PutHiveStreaming processor worked perfectly on my dev cluster which does
> not have HA, but is also different in other ways unfortunately. )
>
> Thanks,
>
> Olav
>
>
>
>
> Olav Jordens
> Senior ETL Developer
> Two Degrees Mobile Limited
> ===
> (M) 022 620 2429
> (P) 09 919 7000
> www.2degreesmobile.co.nz
>
> Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland |
> New Zealand |
> PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9
> 919 7001
>
>
>
>
> --
>
> Disclaimer
> The e-mail and any files transmitted with it are confidential and may
> contain privileged or copyright information. If you are not the intended
> recipient you must not copy, distribute, or use this e-mail or the
> information contained in it for any purpose other than to notify us of the
> error. If you have received this message in error, please notify the sender
> immediately, by email or phone (+64 9 919 7000) and delete this email
> from your system. Any views expressed in this message are those of the
> individual sender, except where the sender specifically states them to be
> the views of Two Degrees Mobile Limited. We do not guarantee that this
> material is free from viruses or any other defects although due care has
> been taken to minimize the risk
>
>
>
>

Re: NPE MergeContent processor

[Joe Witt]

that said even if it is due to crashes or even disk full cases we
should figure out what happened and make it not possible.  We must
always work to eliminate the possibility of corruption causing events
and work to recover well in the face of corruption...

On Fri, Nov 11, 2016 at 2:57 PM, Oleg Zhurakousky
 wrote:
> Conrad
>
> Is it possible that you may be dealing with corrupted repositories (swap,
> flow file etc.) due to your upgrades or may be even possible crashes?
>
> Cheers
> Oleg
>
> On Nov 11, 2016, at 3:11 AM, Conrad Crampton 
> wrote:
>
> Hi,
> This is the flow. The incoming flow is basically a syslog message which is
> parsed, enriched then saved to HDFS
> 1.   Parse (extracttext)
> 2.   Assign matching parts to attributes
> 3.   Enrich ip address location
> 4.   Assign attributes with geoenrichment
> 5.   Execute python script to parse useragent
> 6.   Create json from attributes
> 7.   Convert to avro (all strings)
> 8.   Convert to target avro schema (had to do 7 & 8 due to bug(?) where
> couldn’t go directly from json to avro with integers/longs)
> 9.   Merge into bins (see props below)
> 10.   Append ‘.avro’ to filenames (for reading in Spark subsequently)
> 11.   Save to HDFS
>
> Does this help at all?
> If you need anything else just shout.
> Regards
> Conrad
>
> 
>
>
> 
> additional out of shot
> · compression level : 1
> · Keep Path : false
>
>
> From: Oleg Zhurakousky 
> Reply-To: "users@nifi.apache.org" 
> Date: Thursday, 10 November 2016 at 18:40
> To: "users@nifi.apache.org" 
> Subject: Re: NPE MergeContent processor
>
> Conrad
>
> Any chance you an provide a bit more info about your flow?
> I was able to find a condition when something like this can happen, but it
> would have to be with some legacy NiFi distribution, so it’s a bit puzzling,
> but i really want o see if we can close the loop on this.
> In any event I think it is safe to raise JIRA on this one
>
> Cheers
> Oleg
>
>
> On Nov 10, 2016, at 10:06 AM, Conrad Crampton 
> wrote:
>
> Hi,
> The processor continues to write (to HDFS – the next processor in flow) and
> doesn’t block any others coming into this processor (MergeContent), so not
> quite the same observed behaviour as NIFI-2015.
> If there is anything else you would like me to do to help with this more
> than happy to help.
> Regards
> Conrad
>
> From: Bryan Bende 
> Reply-To: "users@nifi.apache.org" 
> Date: Thursday, 10 November 2016 at 14:59
> To: "users@nifi.apache.org" 
> Subject: Re: NPE MergeContent processor
>
> Conrad,
>
> Thanks for reporting this. I wonder if this is also related to:
>
> https://issues.apache.org/jira/browse/NIFI-2015
>
> Seems like there is some case where the UUID is ending up as null.
>
> -Bryan
>
>
> On Wed, Nov 9, 2016 at 11:57 AM, Conrad Crampton
>  wrote:
>
> Hi,
> I saw this error after I upgraded to 1.0.0 but thought it was maybe due to
> the issues I had with that upgrade (entirely my fault it turns out!), but I
> have seen it a number of times since so I turned debugging on to get a
> better stacktrace. Relevant log section as below.
> Nothing out of the ordinary, and I never saw this in v0.6.1 or below.
> I would have raised a Jira issue, but after logging in to Jira it only let
> me create a service desk request (which didn’t sound right).
> Regards
> Conrad
>
> 2016-11-09 16:43:46,413 DEBUG [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=12c0bec7-68b7-3b60-a020-afcc7b4599e7] has chosen to yield
> its resources; will not be scheduled to run again for 1000 milliseconds
> 2016-11-09 16:43:46,414 DEBUG [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] Binned 42 FlowFiles
> 2016-11-09 16:43:46,418 INFO [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] Merged
> [StandardFlowFileRecord[uuid=5e846136-0a7a-46fb-be96-8200d5cdd33d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1475059643340-275849,
> container=default, section=393], offset=567158,
> length=2337],offset=0,name=17453303363322987,size=2337],
> StandardFlowFileRecord[uuid=a5f4bd55-82e3-40cb-9fa9-86b9e6816f67,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1475059643340-275849,
> container=default, section=393], offset=573643,
> length=2279],offset=0,name=17453303351196175,size=2279],
> StandardFlowFileRecord[uuid=c1ca745b-660a-49cd-82e5-fa8b9a2f4165,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1475059643340-275849,
> container=default, section=393], offset=583957,
> length=2223],offset=0,name=17453303531879367,size=2223],
> StandardFlowFileRecord[uuid=,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1475059643340-275849,
> container=default, section=393], offset=595617,
> length=2356],offset=0,name=,size=2356],
> StandardFlowFileRecord

Re: stream one large file, only once

[Joe Witt]

The pattern you want for this is

1) GetFile or (ListFile + FetchFile)
2) RouteText
3) PublishKafka

As Andrew points out GetFile and FetchFile do *not* read the file
contents into memory.  The whole point of NiFi's design in general is
to take advantage of the content repository rather than forcing
components to hold things in memory.  While they can elect to hold
things in memory they don't have to and the repository allows reading
from and writing to streams all within a unit of work pattern
transactional model.  There is a lot more to say on that topic but you
can see a good bit about it in the docs.

RouteText is the way to avoid the SplitText memory scenario where
there are so many lines that even holding pointers/metadata about
those lines itself becomes problematic.  You can also do as Andrew
points out and split in chunks which also works well.  RouteText will
likely yield higher performance though overall if it works for your
case.

Thanks
Joe

On Mon, Nov 14, 2016 at 8:11 AM, Andrew Grande  wrote:
> Neither GetFile nor FetchFile read the file into memory, they only deal with
> the file handle and pass the contents via a handle to the content repository
> (NiFi streams data into and reads as a stream).
>
> What you will face, however, is an issue with a SplitText when you try to
> split it in 1 transaction. This might fail based on the JVM heap allocated
> and file size. A recommended best practice in this case is to introduce a
> series of 2 SplitText processors. 1st pass would split into e.g. 10 000 row
> chunks, 2nd - into individual. Adjust for your expected file sizes and
> available memory.
>
> HTH,
> Andrew
>
> On Mon, Nov 14, 2016 at 7:23 AM Raf Huys  wrote:
>>
>> I would like to read in a large (several gigs) of logdata, and route every
>> line to a (potentially different) Kafka topic.
>>
>> - I don't want this file to be in memory
>> - I want it to be read once, not more
>>
>> using `GetFile` takes the whole file in memory. Same with `FetchFile` as
>> far as I can see.
>>
>> I also used a `ExecuteProcess` processor in which the file is `cat` and
>> which splits off a flowfile every millisecond. This looked to be a somewhat
>> streaming approach to the problem, but this processor runs continuously (or
>> cron based) and by consequence the logfile is re-injected all the time.
>>
>> What's the typical Nifi for this? Tx
>>
>> Raf Huys

Re: stream one large file, only once

[Joe Witt]

OOM errors can often show you the symptom more readily than the cause.

If you have SplitText after it then what Andrew mentioned is almost
certainly the cause.  If RouteText will meet the need I think you'll
find that yields far better behavior.  The way I'd do what it sounds
like you're doing is:

ListFile
FetchFile
RouteText
PublishKafka (using demarcation strategy based on whatever the end of
line bytes you have are)

This will be very efficient and low memory.

Thanks
Joe

On Mon, Nov 14, 2016 at 8:32 AM, Raf Huys  wrote:
> Thanks for making this clear!
>
> I was distracted because I do have a `java.lang.OutOfMemoryError` on the
> GetFile processor itself (and a matching `bytes read` spike corresponding to
> the file size).
>
> On Mon, Nov 14, 2016 at 2:23 PM, Joe Witt  wrote:
>>
>> The pattern you want for this is
>>
>> 1) GetFile or (ListFile + FetchFile)
>> 2) RouteText
>> 3) PublishKafka
>>
>> As Andrew points out GetFile and FetchFile do *not* read the file
>> contents into memory.  The whole point of NiFi's design in general is
>> to take advantage of the content repository rather than forcing
>> components to hold things in memory.  While they can elect to hold
>> things in memory they don't have to and the repository allows reading
>> from and writing to streams all within a unit of work pattern
>> transactional model.  There is a lot more to say on that topic but you
>> can see a good bit about it in the docs.
>>
>> RouteText is the way to avoid the SplitText memory scenario where
>> there are so many lines that even holding pointers/metadata about
>> those lines itself becomes problematic.  You can also do as Andrew
>> points out and split in chunks which also works well.  RouteText will
>> likely yield higher performance though overall if it works for your
>> case.
>>
>> Thanks
>> Joe
>>
>> On Mon, Nov 14, 2016 at 8:11 AM, Andrew Grande  wrote:
>> > Neither GetFile nor FetchFile read the file into memory, they only deal
>> > with
>> > the file handle and pass the contents via a handle to the content
>> > repository
>> > (NiFi streams data into and reads as a stream).
>> >
>> > What you will face, however, is an issue with a SplitText when you try
>> > to
>> > split it in 1 transaction. This might fail based on the JVM heap
>> > allocated
>> > and file size. A recommended best practice in this case is to introduce
>> > a
>> > series of 2 SplitText processors. 1st pass would split into e.g. 10 000
>> > row
>> > chunks, 2nd - into individual. Adjust for your expected file sizes and
>> > available memory.
>> >
>> > HTH,
>> > Andrew
>> >
>> > On Mon, Nov 14, 2016 at 7:23 AM Raf Huys  wrote:
>> >>
>> >> I would like to read in a large (several gigs) of logdata, and route
>> >> every
>> >> line to a (potentially different) Kafka topic.
>> >>
>> >> - I don't want this file to be in memory
>> >> - I want it to be read once, not more
>> >>
>> >> using `GetFile` takes the whole file in memory. Same with `FetchFile`
>> >> as
>> >> far as I can see.
>> >>
>> >> I also used a `ExecuteProcess` processor in which the file is `cat` and
>> >> which splits off a flowfile every millisecond. This looked to be a
>> >> somewhat
>> >> streaming approach to the problem, but this processor runs continuously
>> >> (or
>> >> cron based) and by consequence the logfile is re-injected all the time.
>> >>
>> >> What's the typical Nifi for this? Tx
>> >>
>> >> Raf Huys
>
>
>
>
> --
> Mvg,
>
> Raf Huys

Re: NiFi w/ Ranger and AD

[Joe Witt]

Nick - there does appear to be agreement with your finding.  Take a
look here https://issues.apache.org/jira/browse/NIFI-3020

On Mon, Nov 14, 2016 at 10:57 AM, Nicholas Hughes
 wrote:
> Has anyone implemented Apache NiFi with authentication against Microsoft
> Active Directory and Apache Ranger for authorization (also using AD
> accounts)?
>
> The authentication works as expected and UserSync works properly in Ranger,
> but I think NiFi and Ranger might not be on the same page page when it comes
> to the expected username format.
>
> I can type in my AD sAMAccountName and password at the NiFi login screen,
> and authentication is successful. Additionally, Ranger is set to sync users
> from AD using the sAMAccountName and that seems to work fine. However,
> authorization fails with a "Unable to perform the desired action due to
> insufficient permissions. Contact the system administrator." error. I
> decoded the JWT from the user log, and the payload looks like:
>
> {
>   "sub": "cn=Nick Hughes,ou=Users,ou=Accounts,dc=example,dc=com",
>   "iss": "LdapProvider",
>   "aud": "LdapProvider",
>   "preferred_username": "Nick Hughes",
>   "kid": 1,
>   "exp": 1479180675,
>   "iat": 1479137475
> }
>
> I suspect that authorization isn't working since the usernames in Ranger are
> the short sAMAccountName (nhughes for example) while the JWT has the CN and
> DN in the token. Totally guessing, so feel free to set me straight...
>
> Anyone have any experience here? I saw some posts on the Internet regarding
> Ranger with LDAP, but there may be some idiosyncrasies with AD.
>
> Thanks!
>
> -Nick
>

Re: Nifi- PutEmail processor issue

[Joe Witt]

Sravani,

The flow you describe makes sense.  So now lets focus on the PutEmail
processor.  Can you please clarify if when it fails to connect the
flowfile is transferred to the 'failure' relationship or the 'success'
relationship?  If it flows to the 'failure' relationship then as
Conrad points out you can just simply have failure loop back to that
processor so it will continue to retry until it actually succeeds in
transferring.  If it flows to the 'success' relationship even though
the email send fails then we need to look into it further and the
stack trace oleg requests will be helpful.

Thanks
Joe

On Tue, Nov 15, 2016 at 7:47 AM, Oleg Zhurakousky
 wrote:
> Sravani
>
> Would you be able to provide a full stack trace of the connection exception.
> Also, while I assume you are providing the correct connection properties
> (i.e., host,port etc) I would still recommend to check the they are correct,
> but in any event the full stack trace would definitely help and you cn find
> it the the wifi app logs.
>
> Cheers
> Oleg
>
> On Nov 15, 2016, at 4:07 AM, Gadiputi, Sravani
>  wrote:
>
> Thank you for your reply.
>
> My requirement is , I just try to send/copy the 3 different files from
> source to destination through Nifi, and these jobs runs weekly once.
> So I wanted to know which file is successfully moved through email.
> In this process, I have configured putemail for each flow. There are hardly
> 3 notifications only.
> Though files have been moved to destination, we could not receive the
> notifications properly and giving the below error.
>
> Please suggest.
>
> Thanks,
> Sravani
>
>
> From: Jeff [mailto:jtsw...@gmail.com]
> Sent: Tuesday, November 15, 2016 1:25 PM
> To: users@nifi.apache.org
> Subject: Re: Nifi- PutEmail processor issue
>
> Hello Sravani,
>
> Could it be possible that the SMTP server you're using is denying
> connections due to the volume of emails your flow might be sending?  How
> many emails are sent per flow file, and how many emails do you estimate are
> sent per minute?
>
> If this is the case, you can modify your flow to aggregate flowfiles with a
> processor like MergeContent so that you can send emails that resemble a
> digest, rather than a separate email for each flowfile that moves through
> your flow.
>
> On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani
>  wrote:
>
>
> Hi,
>
> I have used PutEmail processor in my project to send email notification for
> successful/failure copying of a files.
> Each file flow having corresponding PutEmail to send  email notification to
> respective recipients.
>
> Here the issue is, sometimes email notification will send to respective
> recipients successfully  for successful/failure job.
> But sometimes for any one specific job email notification will not be send
> to recipients though job is successful, due to  below error.
>
> Error:
>
> Could not connect to SMTP host
> Java.net.ConnectException: Connection timed out
>
> Could you please suggest me how we can overcome this error.
>
>
> Thanks,
> Sravani
>
> This message contains information that may be privileged or confidential and
> is the property of the Capgemini Group. It is intended only for the person
> to whom it is addressed. If you are not the intended recipient, you are not
> authorized to read, print, retain, copy, disseminate, distribute, or use
> this message or any part thereof. If you receive this message in error,
> please notify the sender immediately and delete all copies of this message.
>
>

Re: Delay Processor

[Joe Witt]

The concept of flowfile penalization exists to support many of the
requirements for delay.  This is something which can be set, by a
component (processor), on a flow file for some given period of time
and which will be honored on the connection it is placed on such that
things without this penalization will be higher priority.  It was
built for the typical failure loops.  This is appropriate when it is
something about that 'flowfile' and the thing the processor it trying
to interact with that is the problem and therefore simply marking that
flowfile as problematic for a bit is likely to allow whatever the
condition is to go away without just making the problem worse.

There is also the concept of process yielding.  A component
(processor) can signal that it should yield which means it will not be
triggered again at its normally scheduled time but rather will yield
until its yield period is up.  This makes sense when there is
something about the state of that processor and its configuration
relative to whatever it is trying to do/interacting with which is
expected to be able to go away or be resolved on its own.

Mark Payne and I have talked in the past about the notion of the
framework automatically tracking flowfiles that appear to be looping
and doing exponential backoff on them.  This would be done just as you
suggested which is via flow file attributes.

Is your case/concern more like the 'something is wrong with this flow
file vs environment' or more like 'something is wrong with this
process vs environment'?



On Tue, Nov 15, 2016 at 8:50 AM, Andrew Grande  wrote:
> Hi,
>
> I'd lime to check where discussions are on this, ir propose the new
> component otherwise.
>
> Use case: make delay strategies explicit, easier to use. E.g. think of a
> failure retry loop.
>
> Currently, ControlRate is somewhat related, but can be improved. E.g.
> introduce delay strategies a la prioritizers on the connection?
>
> Thinking out loud, something like a exponential backoff strategy could be
> kept stateless by adding a number of housekeeping attributes to the FF,
> which eliminates the need for any state in the processor itself.
>
> I'll stop here to see if any ideas were captured prior abd what community
> thinks of it.
>
> Andrew

Re: puttcp

[Joe Witt]

In 1.x back pressure happens by default so if that is what is being used it
is probably why the flow appears stopped.

In the 1.1 release this will he visually more obvious.

Thanks
Joe

On Nov 16, 2016 9:29 AM, "Bryan Bende"  wrote:

> Hello Raf,
>
> The message about attempting to set the socket buffer size is not really
> an error that would prevent anything from working, it is just a warning so
> that the user knows that NiFI created a new connection and tried to set the
> Socket's receive buffer to some value that was specified in the properties
> (1 MB in your case), and the OS only let it get set to some smaller value,
> which could be important for someone trying to tune various settings.
>
> This happens whenever a new connection is created, which likely happens
> your TCP servers goes down and comes back up. It can also happen during
> normal operation of the processor, PutTCP will create connections on the
> fly as needed and then close them if they have been used in the amount of
> time greater than "Idle Connection Expiration".
>
> I definitely agree it would be nice for that message to not print all the
> time though, one way to get rid of it would be to reduce the value of "Max
> Size of Socket Send Buffer" to meet what the OS is allowing, another way
> would be to configure logback.xml so that org.apache.nifi.processor.
> util.put.sender.SocketChannelSender only logged at the ERROR level, since
> this message is logged at the WARN level, but this means you could miss
> other valuable warnings.
>
> When you say "some time later it becomes worse... no flowfiles are
> generated", are you saying GenerateFlowFile is no longer generating flow
> files? when this happens do you have a lot of flow files in queues, and do
> you have back-pressure configured?
>
> Thanks,
>
> Bryan
>
>
> On Wed, Nov 16, 2016 at 5:59 AM, Raf Huys  wrote:
>
>> I'm having a simple flow GenerateFlowfile -> ReplaceText -> PutTCP which
>> should establish a TCP connection and send a small piece of text over that
>> connection every 60 seconds. This is established bij scheduling the first
>> processor as a cron job.
>>
>> This pipeline works, until I start restarting our TCP server upstream.
>>
>>
>> What can happen is that the TCP server is unavailable (due to
>> reasons...). Nevertheless, when the TCP server becomes available again, the
>> Nifi pipeline should continue doing it's job.
>>
>> However, what I observe is that the PutTCP processor start throwing the
>> following errors after the TCP became unavailable, and then available again:
>>
>> Attempted to set Socket Buffer Size to ... bytes but could only set to
>> ... bytes. You may want to consider changing the Operating System's maximum
>> receive buffer
>>
>> I find this message confusing because the messages we are sending are
>> about 10 characters wide. Also, the time between between stopping/starting
>> the TCP server is a couple of seconds, which means there is no backpressure
>> of unsent flowfiles.
>>
>> Properties of the PutTCP processor
>>
>>
>>- Hostname localhost
>>- Port 4001
>>- Max Size of Socket Send Buffer 1 MB
>>- Idle Connection Expiration 5 seconds
>>- Connection Per FlowFile true
>>- Outgoing Message Delimiter \r\n
>>- Timeout 10 seconds
>>- SSL Context Service No value set
>>- Character Set UTF-8
>>
>> I would love some help here.
>>
>> PS: Some time later it becomes worse, as every processor in the above
>> pipeline actually stops doing anything...no flowfiles are generated, no
>> errors are thrown...
>>
>> Thanks,
>>
>> Raf Huys
>>
>
>

Re: Problems with frequent Nifi core dumping

[Joe Witt]

Hello Cheryl

Nope - not something that I've seen.  But could you try doing what it
shows in that log

"Failed to write core dump. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again"

There *might* be some useful data in the core dump itself.

Thanks
Joe

On Wed, Nov 16, 2016 at 10:12 PM, Cheryl Jennings  wrote:
> Hi Everyone,
>
> I'm running Nifi 0.7.0 and I have started to see problems where nifi will
> crash repeatedly and often won't be stable until I've restarted the machine.
> Here's a relevant part from nifi-bootstrap.log:
>
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> A fatal error has been detected by the Java Runtime Environment:
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> SIGSEGV (0xb) at pc=0x0033ada09470, pid=27979, tid=140067910633216
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build
> 1.8.0_74-b02)
> 2016-11-15 14:55:54,272 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64
> compressed oops)
> 2016-11-15 14:55:54,273 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> Problematic frame:
> 2016-11-15 14:55:54,273 INFO [NiFi logging handler] org.apache.nifi.StdOut #
> C  [libpthread.so.0+0x9470]  pthread_mutex_lock+0x0
>
>
> A bit more detail can be found here: http://paste.ubuntu.com/23488485/
>
> Has anyone seen something like this before?
>
> Thanks!
> -Cheryl

Re: Status of Running Process Group for completion of processing data

[Joe Witt]

Nirmal,

NiFi's orientation is around continuous streams of data rather than
starting and completing jobs.

When you look at processors via the UI and you see the green started
arrow or the red stopped square it is best to think about that as
"this component is (or is not) scheduled to execute" rather than "this
is running or not running".  It is ok that they're scheduled to run.
They're still not going to actually get triggered unless they need to
be.  If PutHDFS has not data to send then it won't actually be
triggered to execute.

The status information you mention, particularly the rolling 5 min
window information, is to help operators and flow managers understand
behavior in the recent past and is not likely to be useful for an
external process to decide when to schedule and unschedule components
to be eligible to run.

Does this make sense?  Does this approach create some challenge for
your use case?  If so please describe a bit more on what you're trying
to accomplish so we can see if we can help.

Thanks
Joe

On Tue, Nov 22, 2016 at 7:12 AM, Nirmal Kumar
 wrote:
> Hi All,
>
>
>
> How do I know when exactly the Process Group(and its processors) has
> actually finished the processing?
>
>
>
> Consider a simple Nifi use case:
>
> GetFile -> PutHDFS
>
>
>
> I want to execute the above use case and on completion want to stop it.
>
>
>
> From the Rest API endpoint
> http://nifi-host:port/nifi-api/process-groups/$processGroupId I can get the
> below statistics(aggregateSnapshot).
>
> Currently I check these aggregateSnapshot 25 metrics for zeros and
> runningCount as 2.
>
> But somehow seems like these statistics by design shows last 5 mins metrics
> that means I will need to wait at least for 5 mins.
>
> My requirement is to get the status of Nifi graph / Process Group as soon as
> the use case is done processing.
>
>
>
> I also tried looking if this 5 min time period can be tweaked/changed
> somewhere but seems it CANNOT.
>
>
>
> {
>
> revision: + {... },
>
> id: "4e26e718-0158-1000-90e2-7b10528dc41d",
>
> uri:
> "http://xxx.xxx.xxx.xxx:8080/nifi-api/process-groups/4e26e718-0158-1000-90e2-7b10528dc41d";,
>
> position: + {... },
>
> permissions: + {... },
>
> bulletins:[],
>
> component: + {... },
>
> status: - {
>
> id:
> "4e26e718-0158-1000-90e2-7b10528dc41d",
>
> name: "TestGetFileToHDFS",
>
> statsLastRefreshed:
> "17:20:49 IST",
>
>aggregateSnapshot: - {
>
> id:
> "4e26e718-0158-1000-90e2-7b10528dc41d",
>
> name:
> "TestGetFileToHDFS",
>
> flowFilesIn:
> 0,
>
> bytesIn: 0,
>
> input: "0 (0
> bytes)",
>
>
> flowFilesQueued: 0,
>
> bytesQueued:
> 0,
>
> queued: "0
> (0 bytes)",
>
> queuedCount:
> "0",
>
> queuedSize:
> "0 bytes",
>
> bytesRead:
> 0,
>
> read: "0
> bytes",
>
>
> bytesWritten: 0,
>
> written: "0
> bytes",
>
>
> flowFilesOut: 0,
>
> bytesOut: 0,
>
> output: "0
> (0 bytes)",
>
>
> flowFilesTransferred: 0,
>
>
> bytesTransferred: 0,
>
> transferred:
> "0 (0 bytes)",
>
>
> bytesReceived: 0,
>
>
> flowFilesReceived: 0,
>
> received: "0
> (0 bytes)",
>
> bytesSent:
> 0,
>
>
> flowFilesSent: 0,
>
> sent: "0 (0
> bytes)",
>
>
> activeThreadCount: 0
>
>  }
>
> },
>
> runningCount: 2,
>
> stoppedCount: 0,
>
> invalidCount: 0,
>
> disabledCount: 0,
>
> activeRemotePortCount: 0,
>
>   

Re: Kafka (Avro) --> HDFS (ORC) in proper partition

[Joe Witt]

Hello

Collecting the data:
- Use Consume Kafka

Routing each event based on its content:
- There is no out of the box processor to route Avro just yet.  We
will hopefully tackle that soon.  In the mean time you could covert
the Avro to JSON and use RouteJson.  If you want the data back in avro
you can convert back to avro. So it would be something like
 - ConvertAvroToJson
 - EvaluateJsonPath
 - ConvertJsonToAvro
 At this point you'll have a flow file attribute which tells you about
the binning you want.  This approach will have the performance hit of
the conversions so if you can avoid going back to Avro that will help
or if you use the ExecuteScript or related processors you can whip up
some Groovy to deal with the Avro directly.  You have lots of options
there.

Merging events together to sympathize with HDFS block sizes:
- Use MergeContent.  It supports binning data together using a flow
file attribute such as the one you would have pulled in the previous
section above.

Delivery to HDFS
- Use PutHDFS

Thanks
Joe

On Tue, Nov 22, 2016 at 7:59 AM, E Worthy  wrote:
> Is this not possible with this product?
>
>
> On Sunday, November 20, 2016 8:50 PM, E Worthy 
> wrote:
>
>
> Hello,
>
> I have Avro data coming in from Kafka and I need to place this data into the
> proper HDFS partition based on values in each row of data. Is this possible
> in Nifi? I'm looking at the PutHDFS, ConvertAvroToORC, and expressions. Do I
> somehow set a variable using an expression for each row and use that in the
> PutHDFS?
>
> Thanks for any help,
> Eric
>
>

Re: How to determine NiFi version number from NiFi API call

[Joe Witt]

Nick,

The upcoming release of NiFi has a cool new diagnostics feature which
lists these sort of details.

Thanks
Joe

On Tue, Nov 22, 2016 at 9:28 AM, Piper, Nicholas C  wrote:
> Hi,
>
> I'm looking through the API https://nifi.apache.org/docs/nifi-docs/rest-api/ 
> to see how to query the NiFi version number (or API version number.) I'd like 
> to do this to check that an upgrade worked, and that NiFi is the version I 
> expect.
>
> I hoped maybe /system-diagnostics, or /controller/config , but those do not 
> contain this information.
>
> Is anyone able to point me at the right place?
>
> Many thanks,
>
>  Nick
>
> --
> CGI IT UK Limited. A CGI Group Inc. Company
> Registered Office 250 Brook Drive, Green Park, Reading RG2 6UA, United 
> Kingdom. Registered in England & Wales - Number 947968
>
>
>

Re: Flow file replication

[Joe Witt]

Hello Nicolas

So today if a NiFi node dies the data should still be recoverable unless
the disks/storage itself is dead.  So losing a node should not cause data
loss but it does impact availability.

Currently the way to achieve rapid availability in the presence of node
failure would be through having the key repositories (content, flowfile,
provenance) all mounted in such a way that you could remount them to
another host.  This is a typical infrastructure/OS level approach that can
be managed outside of NiFi itself.  It is admittedly non-trivial to get
right.

However, there have been discussions on making this a first class feature
of nifi itself.
https://cwiki.apache.org/confluence/display/NIFI/Data+Replication

Thanks
Joe



On Tue, Nov 22, 2016 at 12:30 PM, Provenzano Nicolas <
nicolas.provenz...@gfi.fr> wrote:

> Hi all,
>
>
>
> Is there a way to replicate the repositories in order to be able to resume
> the processing of data when a nodes crashes ?
>
>
>
> What is the recommendation to avoid losing data in case a node crashes ?
>
>
>
> Thanks in advance
>
>
>
> BR
>
>
>
> Nicolas
>
>
>
> —
> Gfi Informatique Telecom
> *Nicolas Provenzano*
>
> BU Telecom / Customer Experience
>
> *nicolas.provenz...@gfi.fr *
> —
> 2 rue Andras Beck – 92360 Meudon La Forêt
> Tél. : +33 (0)1 46 83 35 06
> Mob. : +33 (0)6 82 07 19 21
> www.gfi.world
> —
> [image: signature-img]
>
>
>

Re: Code index question about PutElasticSearchHttp.java#L333

[Joe Witt]

Hello

Matt put in the JIRA/pr last night and it was merged to master this
morning.  If you want to give that a go that would help too.

Thanks
Joe

On Wed, Nov 23, 2016 at 9:11 AM, Andreas Petter (External)
 wrote:
> Hi Matt & James,
>
> thank you very much for your prompt help. @James: We accidentially 
> encountered an IndexOutOfBounds so I read the code; and yes, changing the 
> flow and the hierarchy of the data seemingly fixed the problem.
>
> Thanks again, regards,
> Andreas
>
> -Ursprüngliche Nachricht-
> Von: Matt Burgess [mailto:mattyb...@apache.org]
> Gesendet: Mittwoch, 23. November 2016 02:04
> An: users@nifi.apache.org
> Betreff: Re: Code index question about PutElasticSearchHttp.java#L333
>
> This is definitely a bug, classic reverse logic/iteration error, not sure 
> what I was thinking there :(
>
> I took the liberty of filing a JIRA ticket [1], and will take the liberty of 
> fixing it shortly :P
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-3087
>
> On Tue, Nov 22, 2016 at 5:33 PM, James Wing  wrote:
>> Andreas,
>>
>> I'm not familiar with PutElasticSearch, but the code you point to does
>> appear strange.  It doesn't look like there are any unit tests where
>> multiple results are returned from ElasticSearch with one or more
>> failures that would exercise this case.
>>
>> L329...
>> // All items are returned whether they succeeded or failed, so
>> iterate through the item array
>> // at the same time as the flow file list, moving each to success
>> or failure accordingly
>> for (int i = 0; i < itemNodeArray.size(); i++) {
>> JsonNode itemNode = itemNodeArray.get(i);
>> FlowFile flowFile = flowFilesToTransfer.remove(i); ...
>>
>>
>> Have you had problems with PutElasticSearch?  Again, I'm not very
>> familiar with it, but a workaround might be to keep the batch size at 1 (not 
>> ideal).
>>
>> Would you be willing to create a JIRA ticket for this issue at
>> https://issues.apache.org/jira/browse/NIFI?
>>
>>
>> Thanks,
>>
>> James
>>
>> On Mon, Nov 21, 2016 at 3:28 AM, Andreas Petter (External)
>>  wrote:
>>>
>>> Hello everybody,
>>>
>>>
>>>
>>> I have a small question concerning the putElasticSearch.java code…
>>>
>>> While reading through the code I found the following line…
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elas
>>> ticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apac
>>> he/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L333
>>>
>>>
>>>
>>> Maybe it is a dumb question, but is the index in the remove (i.e.
>>> “i”) the index it was intended for (think about what happens if run
>>> on at least two items)?
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>> Regards,
>>>
>>> Andreas
>>>
>>>
>>>
>>>
>>> 
>>>
>>> Este mensaje y sus adjuntos se dirigen exclusivamente a su
>>> destinatario, puede contener información privilegiada o confidencial
>>> y es para uso exclusivo de la persona o entidad de destino. Si no es
>>> usted. el destinatario indicado, queda notificado de que la lectura,
>>> utilización, divulgación y/o copia sin autorización puede estar
>>> prohibida en virtud de la legislación vigente. Si ha recibido este
>>> mensaje por error, le rogamos que nos lo comunique inmediatamente por
>>> esta misma vía y proceda a su destrucción.
>>>
>>> The information contained in this transmission is privileged and
>>> confidential information intended only for the use of the individual
>>> or entity named above. If the reader of this message is not the
>>> intended recipient, you are hereby notified that any dissemination,
>>> distribution or copying of this communication is strictly prohibited.
>>> If you have received this transmission in error, do not read it.
>>> Please immediately reply to the sender that you have received this
>>> communication in error and then delete it.
>>>
>>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
>>> destinatário, pode conter informação privilegiada ou confidencial e é
>>> para uso exclusivo da pessoa ou entidade de destino. Se não é vossa
>>> senhoria o destinatário indicado, fica notificado de que a leitura,
>>> utilização, divulgação e/ou cópia sem autorização pode estar proibida em 
>>> virtude da legislação vigente.
>>> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
>>> imediatamente por esta mesma via e proceda a sua destruição
>>
>>
>
> 
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, 
> puede contener información privilegiada o confidencial y es para uso 
> exclusivo de la persona o entidad de destino. Si no es usted. el destinatario 
> indicado, queda notificado de que la lectura, utilización, divulgación y/o 
> copia sin autorización puede estar prohibida en virtud de la legislación 
> vigente. Si ha recibido este mensaje por error, le rogamos que nos lo 
> comunique inmediatamente por esta misma vía y proceda

Re: Delimiter splitting in ExtractText possible?

[Joe Witt]

Jason

That was an excellent response.

Prabhu - i think the question is what would you like to do with the
data?  Are you going to transform it then send it somewhere?  Do you
want to be able to filter some rows out then send the rest?  Can you
describe that part more?

The general pattern here is

It is certainly easy enough to do the two-phase split to maintain efficiency

SplitText (500 line chunks for example)
SplitText (single line chunks)
?? - what do you want to accomplish per line?
?? - where is the data going?

Thanks
Joe

On Wed, Nov 23, 2016 at 9:41 AM, Jason Tarasovic
 wrote:
> Prabhu,
>
> It's possible to do what you're asking but not especially efficient. You can
> SplitText twice (10,000 and then 1) outputting the header on each and then
> running the result through ExtractText. Your regex would be something like
> ([^,]*?),([^,]*), so match 0 or more non-comma characters followed by a
> comma. ExtractText will place the matched capture groups into attributes
> like you mentioned (date.1->the_captured_text)
>
> However, it's not super efficient or at least it hasn't been in my case as
> you're moving the FlowFile contents into attributes and the attributes are
> stored in memory so, depending on how large the file is, you *may*
> experience excessive GC activity or OOM errors.
>
> Using InferAvroSchema (if you don't know the schema in advance) and then
> using ConvertCSVtoAvro may be better option depending on where the data is
> ultimately going. One caveat though is that ConvertCSVtoAvro seems to only
> work with properly quoted and escaped CSV that conforms to RFC 4180.
>
> I'm just getting started with NiFi myself so not an expert or anything but I
> hope that helps.
>
> -Jason
>
> On Tue, Nov 22, 2016 at 3:34 AM, prabhu Mahendran 
> wrote:
>>
>> Hi All,
>>
>> I have CSV unstructured data with comma as delimiter which contains 100
>> rows.
>>
>> Is it possible to extract the data's in csv file using comma as seperator
>> in nifi processors.
>>
>>
>> See my Sample data 3 from 100 rows.
>>
>> No,Name,Age,PAN,City
>> 1,Siva,22,91230,Londan,
>> 2,,23,91231,UK
>> 3,Greck,22,,US
>>
>>
>> In 1st row having all values which can be seperated by "data" attribute
>> having regex (.+),(.+),(.+),(.+),(.+) then row will be split like below..,
>>
>> data.1-->1
>> data.2-->Siva
>> data.3-->22
>> data.4-->91230
>> data.5-->Londan
>>
>> But in Second row which having Empty values in Name column can using regex
>> (.+),,(.+),(.+),(.+) then row will be split like below..,
>>
>>data.1-->2
>>data.2-->23
>>data.3-->91231
>>data.4-->UK
>>
>> Third row same as PAN Column empty it can able to split using another
>> regex attribute.
>>
>> But my problem is now data having 100 rows.In future this may having
>> another 100 rows.So again need to write more regex attributes to capture
>> group wise .
>>
>>
>> So I think  i have given comma(,) as common regex for all rows in csv file
>> then it will split data as into data.1,data.2,...data.5
>>
>> But i gets an validation failed error in Bulletins Indicator in
>> ExtractTextProcessor.
>>
>> So is this possible to write delimiter wise splitting of rows in CSV File?
>>
>> Is this possible to write common regex for all csv data in ExtractText or
>> any other processor?
>>
>

Re: NiFi w/ Ranger and AD

[Joe Witt]

Nick

Thanks for documenting those findings.  Will definitely help in
resolving it in a more consistent manner.

As for the release of any vendor specific distributions of NiFi you'll
want to connect directly with that vendor for those sorts of
questions.

Thanks
Joe

On Sun, Nov 27, 2016 at 7:40 PM, Nicholas Hughes
 wrote:
> This won't matter very soon with the impending release that will fix this
> issue, but just so it's recorded somewhere...
>
> I tried setting up UserSync in Ranger to pull in the distinguishedName from
> Active Directory since that is what was passed by NiFi. Unfortunately, there
> are two problems with that...
>
> First, UserSync brings the DN in from AD with capital letters in the field
> names (CN=,OU=,DC=) while NiFi is passing them as lowercase (cn=,ou=,dc=).
> This causes the access request to fail to match any policies due to the case
> mismatch. I manually converted the case in the Ranger MySQL DB as a
> workaround.
>
> Second, any policies created which contain the DN user names (or any other
> names containing commas) will not be able to be edited after the initial
> creation. This is due to how Ranger is encoding the URL when accessing the
> Ranger API. A ticket has been created for this issue [1], but I don't think
> it'll get much priority since there was a comment about not having commas in
> user names.
>
> Does anybody know how long it's likely to take for the 1.1.0 release (once
> completed) to get rolled into HDF? I'd rather not have to hack up NiFi in an
> HDF install in order to take advantage of the fix for sAMAccountNames [2]
> that's in that release.
>
> -Nick
>
> [1] - https://issues.apache.org/jira/browse/RANGER-1224
> [2] - https://issues.apache.org/jira/browse/NIFI-3020
>
> On Mon, Nov 14, 2016 at 10:59 AM, Joe Witt  wrote:
>>
>> Nick - there does appear to be agreement with your finding.  Take a
>> look here https://issues.apache.org/jira/browse/NIFI-3020
>>
>> On Mon, Nov 14, 2016 at 10:57 AM, Nicholas Hughes
>>  wrote:
>> > Has anyone implemented Apache NiFi with authentication against Microsoft
>> > Active Directory and Apache Ranger for authorization (also using AD
>> > accounts)?
>> >
>> > The authentication works as expected and UserSync works properly in
>> > Ranger,
>> > but I think NiFi and Ranger might not be on the same page page when it
>> > comes
>> > to the expected username format.
>> >
>> > I can type in my AD sAMAccountName and password at the NiFi login
>> > screen,
>> > and authentication is successful. Additionally, Ranger is set to sync
>> > users
>> > from AD using the sAMAccountName and that seems to work fine. However,
>> > authorization fails with a "Unable to perform the desired action due to
>> > insufficient permissions. Contact the system administrator." error. I
>> > decoded the JWT from the user log, and the payload looks like:
>> >
>> > {
>> >   "sub": "cn=Nick Hughes,ou=Users,ou=Accounts,dc=example,dc=com",
>> >   "iss": "LdapProvider",
>> >   "aud": "LdapProvider",
>> >   "preferred_username": "Nick Hughes",
>> >   "kid": 1,
>> >   "exp": 1479180675,
>> >   "iat": 1479137475
>> > }
>> >
>> > I suspect that authorization isn't working since the usernames in Ranger
>> > are
>> > the short sAMAccountName (nhughes for example) while the JWT has the CN
>> > and
>> > DN in the token. Totally guessing, so feel free to set me straight...
>> >
>> > Anyone have any experience here? I saw some posts on the Internet
>> > regarding
>> > Ranger with LDAP, but there may be some idiosyncrasies with AD.
>> >
>> > Thanks!
>> >
>> > -Nick
>> >
>
>

Re: Keep attributes when merging

[Joe Witt]

Giovanni

You can definitely do this.  The file pulling should be retaining the
key path information as flow file attributes.

The merge process has a property to control what happens with
attributes.  The default is to only copy over matching attributes and
is likely what you'll want.  Take a look at "Attribute Strategy".  Now
you want to retain some key values of course and that would be the
parts of the timestamp you'd want to group on.  You could do this with
an UpdateAttribute processor before MergeContent.  Use that to create
an attribute such as "base-timestamp" or something which just pulls
out the common part of the timestamp you want.  In MergeContent then
you can correlate on this value and since it will be the same it will
also be there for you afterwards.  You can then use this when writing
to HDFS.

This is a pretty common use case so we can definitely help you get
where you want to go with this.

Thanks
Joe

On Tue, Nov 29, 2016 at 9:14 AM, Giovanni Lanzani
 wrote:
> Hi all,
>
> I have the following use case:
>
> I'm reading xml from a folder with subfolders using the following schema:
>
> /my_folder/20161120/many xml's inside
> /my_folder/20161121/many xml's inside
> /my_folder/201611.../many xml's inside
>
> The current pipeline involves: XML -> JSON -> Avro -> HDFS
>
> where the HDFS folder structure is
>
> /my_folder/column=20161120/many avro's inside
> /my_folder/column=20161121/many avro's inside
> /my_folder/column=201611.../many avro's inside
>
> (each column= subfolder is a Hive partition)
>
> In order to reduce the number of avro's in HDFS, I'd love to merge 'em all.
>
> However, as NiFi just reads files from the source folders without any 
> assumption on from which folders they're taken, even if I extract the date 
> from the folder name (or file), this gets lost when using MergeContent. Using 
> the Defragment strategy does not seems like an option, as I don't know in 
> advance how many files I'll see.
>
> That said: isn't there any way to accomplish what I want to do?
>
> Current strategy is to simply merge the files "manually" using avro-tools and 
> bash scripting.
>
> An alternative (although this is forcing what we want to do), is to partition 
> by import date. Then I'd only need to take care of the midnight issue, for 
> example by scheduling NiFi to fetch from the source every 10 minutes, but by 
> doing a MergeContent every 5.
>
> If something isn't clear, please let me know.
>
> Thanks,
>
> Giovanni
>
> Thanks,
>
> Giovanni

Re: Hanging on SplitJSON

[Joe Witt]

Olav

We want you to be able to split your 36MB file into 400,000 things and not
have to stress about this.  Do you mind please filing a JIRA for this to be
followed up on?  We can definitely do better.

Thanks
Joe

On Tue, Nov 29, 2016 at 8:09 PM, Olav Jordens <
olav.jord...@2degreesmobile.co.nz> wrote:

> Hi,
>
>
>
> My bad – the problem appears to be that the 36MB JSON file would be split
> into > 400 000 individual records, each carrying a substantial load of
> attributes. This must be causing an out of memory although I could not find
> such an error in the logs – perhaps even the logs were no longer being
> written to properly!
>
>
>
> Thanks,
>
> Olav
>
>
>
>
>
> *From:* Olav Jordens [mailto:olav.jord...@2degreesmobile.co.nz]
> *Sent:* Wednesday, 30 November 2016 1:25 p.m.
> *To:* users@nifi.apache.org
> *Subject:* Hanging on SplitJSON
>
>
>
> Hi,
>
>
>
> I have a JSON file of about 36MB which is passed to a SplitJSON processor.
> This processor runs for a while and then my UI hangs. In the app-log the
> following ERRORs pop up:
>
>
>
> 2016-11-30 13:03:30,999 ERROR [Site-to-Site Worker Thread-393]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
> remote instance Peer[url=nifi://localhost:42758] due to 
> java.net.SocketTimeoutException:
> Timed out reading from socket; closing connection
>
>
>
> However, I suspect that this has nothing to do with Site-to-Site (from my
> single nifi instance to itself) as there are no ERRORs prior to my flowfile
> hitting the SplitJSON processor, and every time I re-run, it is at this
> point that it hangs. My java Xmx=1024m and Xms=1024m. When I do a nifi dump:
>
>
>
> bin/nifi.sh dump
>
> nifi.sh: JAVA_HOME not set; results may vary
>
>
>
> Java home:
>
> NiFi home: /app/HDF-2.0.1.0/nifi
>
>
>
> Bootstrap Config File: /app/HDF-2.0.1.0/nifi/conf/bootstrap.conf
>
>
>
> Exception in thread "main" java.net.SocketTimeoutException: Read timed out
>
> at java.net.SocketInputStream.socketRead0(Native Method)
>
> at java.net.SocketInputStream.socketRead(SocketInputStream.
> java:116)
>
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
>
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
>
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
>
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
>
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
>
> at java.io.BufferedReader.fill(BufferedReader.java:161)
>
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
>
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
>
> at org.apache.nifi.bootstrap.RunNiFi.dump(RunNiFi.java:695)
>
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:225)
>
>
>
> This again points at a socket issue, but my main confusion is why this
> error occurs every time the flowfile hits the SplitJSON processor?
>
>
>
> The status indicates that it is hanging and not responding to ping
> requests:
>
>
>
> service nifi status
>
> nifi.sh: JAVA_HOME not set; results may vary
>
>
>
> Java home:
>
> NiFi home: /app/HDF-2.0.1.0/nifi
>
>
>
> Bootstrap Config File: /app/HDF-2.0.1.0/nifi/conf/bootstrap.conf
>
>
>
> 2016-11-30 13:23:31,786 INFO [main] org.apache.nifi.bootstrap.Command
> Apache NiFi is running at PID 23080 but is not responding to ping requests
>
>
>
> Any ideas?
>
>
>
> Thanks,
>
> Olav
>
>
>
>
> *Olav Jordens *
>
>
>
>
>
> *Senior ETL Developer Two Degrees Mobile Limited
> === (M) 022 620 2429 (P) 09 919 7000
> www.2degreesmobile.co.nz  *
> Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland |
> New Zealand |
> PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9
> 919 7001 <+64%209-919%207001>
>
>
>
> --
>
> Disclaimer
> The e-mail and any files transmitted with it are confidential and may
> contain privileged or copyright information. If you are not the intended
> recipient you must not copy, distribute, or use this e-mail or the
> information contained in it for any purpose other than to notify us of the
> error. If you have received this message in error, please notify the sender
> immediately, by email or phone (+64 9 919 7000 <+64%209-919%207000>) and
> delete this email from your system. Any views expressed in this message are
> those of the individual sender, except where the sender specifically states
> them to be the views of Two Degrees Mobile Limited. We do not guarantee
> that this material is free from viruses or any other defects although due
> care has been taken to minimize the risk
>
>
>
>

Re: Hanging on SplitJSON

[Joe Witt]

y other defects although due
> care has been taken to minimize the risk
>
>
> *From:* Andy LoPresto [mailto:alopre...@apache.org ]
>
> *Sent:* Wednesday, 30 November 2016 2:48 p.m.
> *To:* users@nifi.apache.org
> *Subject:* Re: Hanging on SplitJSON
>
> Olav,
>
> Have you tried “stacking” these processors so the initial split breaks the
> complete input into smaller chunks and then each of those are split again?
> This is a common pattern we recommend with splitting or merging from/to
> large files. I don’t know what the overall structure of your original file
> is, but you should be able to use the SplitContent processor to split on
> boundaries (for example, if you know each distinct JSON block starts with
> the same key (I know order is not enforced, but you may have this scenario
> because all of the blocks are in the same file)), and take each flowfile
> containing 100-1000 JSON objects and then route them to the SplitJSON
> processor.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
> On Nov 29, 2016, at 5:34 PM, Olav Jordens  co.nz> wrote:
>
> Joe,
>
> Thanks so much – certainly if it tries to batch this job, I will not have
> enough RAM on my small system, but if the processor would understand that
> and push out batches of splits at a time, then it would work for me. I’ll
> log the JIRA.
> Cheers,
> Olav
>
>
> *From:* Joe Witt [mailto:joe.w...@gmail.com ]
> *Sent:* Wednesday, 30 November 2016 2:22 p.m.
> *To:* users@nifi.apache.org
> *Subject:* Re: Hanging on SplitJSON
>
> Olav
>
> We want you to be able to split your 36MB file into 400,000 things and not
> have to stress about this.  Do you mind please filing a JIRA for this to be
> followed up on?  We can definitely do better.
>
> Thanks
> Joe
>
> On Tue, Nov 29, 2016 at 8:09 PM, Olav Jordens <
> olav.jord...@2degreesmobile.co.nz> wrote:
>
> Hi,
>
> My bad – the problem appears to be that the 36MB JSON file would be split
> into > 400 000 individual records, each carrying a substantial load of
> attributes. This must be causing an out of memory although I could not find
> such an error in the logs – perhaps even the logs were no longer being
> written to properly!
>
> Thanks,
> Olav
>
>
> *From:* Olav Jordens [mailto:olav.jord...@2degreesmobile.co.nz]
> *Sent:* Wednesday, 30 November 2016 1:25 p.m.
> *To:* users@nifi.apache.org
> *Subject:* Hanging on SplitJSON
>
> Hi,
>
> I have a JSON file of about 36MB which is passed to a SplitJSON processor.
> This processor runs for a while and then my UI hangs. In the app-log the
> following ERRORs pop up:
>
> 2016-11-30 13:03:30,999 ERROR [Site-to-Site Worker Thread-393]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
> remote instance Peer[url=nifi://localhost:42758] due to 
> java.net.SocketTimeoutException:
> Timed out reading from socket; closing connection
>
> However, I suspect that this has nothing to do with Site-to-Site (from my
> single nifi instance to itself) as there are no ERRORs prior to my flowfile
> hitting the SplitJSON processor, and every time I re-run, it is at this
> point that it hangs. My java Xmx=1024m and Xms=1024m. When I do a nifi dump:
>
> bin/nifi.sh dump
> nifi.sh: JAVA_HOME not set; results may vary
>
> Java home:
> NiFi home: /app/HDF-2.0.1.0/nifi
>
> Bootstrap Config File: /app/HDF-2.0.1.0/nifi/conf/bootstrap.conf
>
> Exception in thread "main" java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.
> java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at org.apache.nifi.bootstrap.RunNiFi.dump(RunNiFi.java:695)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:225)
>
> This again points at a socket issue, but my main confusion is why this
> error occurs every time the flowfile hits the SplitJSON processor?
>
> The status indicates that it is hanging an

Re: Merge Content : triggering merge according to a field

[Joe Witt]

Nicolas,

I don't believe there is support for this in MergeContent.  It does
support a similar pattern for segmented data such that it knows how to
recombine it.  It handles out of order alignment and understanding
start/end/indexes.  However, it requires specific metadata be made
available.  You'd want to build/have something that support your
particular case.

Thanks
Joe

On Wed, Nov 30, 2016 at 4:07 AM, Provenzano Nicolas
 wrote:
> Hi all,
>
>
>
> Yet another question…
>
>
>
> I defined a flow to process session information coming from CSV files.
>
>
>
> Each record contains a session ID, a session state and some session
> counters.
>
>
>
> The merge content processor allows merging flowfiles according to an
> attribute (the correlation attribute name) so I was able to merge flowfiles
> according to the session ID.
>
>
>
> However, I would like to trigger the merging only when the session state
> reaches a specific value (for example, ended).
>
>
>
> Please note that session info can be distributed over several input flows
> but always end with state = ended.
>
>
>
> So for example, a first CSV file contains :
>
>
>
> 100, started, 1, 2
>
>
>
> A second CSV file contains
>
>
>
> 100, inProgress, 2, 4
>
>
>
> And a third CSV file contains
>
>
>
> 100, ended, 4,6
>
>
>
> The merge content processor should then produce the following flowfile :
>
>
>
> 100, started, 1, 2
>
> 100, inProgress, 2, 4
>
> 100, ended, 4,6
>
>
>
> Meaning that the two first records should be kept as long as the “ended” one
> is not received.
>
>
>
> Is there anyway of doing it directly with this processor or with any others
> ?
>
>
>
> Thanks
>
>
>
> Nicolas

[ANNOUNCE] Apache NiFi 1.1.0 Release

[Joe Witt]

Hello

The Apache NiFi team would like to announce the release of Apache NiFi 1.1.0.

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute data.  Apache NiFi was made for dataflow.  It
supports highly configurable directed graphs of data routing,
transformation, and system mediation logic.

This release is the result of fantastic community contribution across
feature requests, documentation, bug reports, code contributions,
reviews, and release validation.

The release highlights:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.1.0

More details on Apache NiFi can be found here:
  http://nifi.apache.org/

The release artifacts can be downloaded from here:
  http://nifi.apache.org/download.html

Maven artifacts have been made available here:
  https://repository.apache.org/content/repositories/releases/org/apache/nifi/

Issues closed/resolved for this list can be found here:
  
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12337875

Thank you
The Apache NiFi team

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

When it is stuck do you see a number in the top right corner of the
processor?  When you stop it does the number remain?  That number is
telling you how many threads are still executing.  Which processor are
we talking about?  When it is in the stuck state can you please run
bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.

Thanks
Joe

On Wed, Nov 30, 2016 at 7:02 PM, Manish G  wrote:
>
> Hi,
>
> I have noticed that sometime a flow file gets stuck on a processor for a
> very long time for no reason and then I can not even stop the processor to
> look at the flow flow file from queue. If I click on stop, then processor
> goes into a state where I cannot start/stop the processor.
>
> On restarting the NiFi, the file gets processed successfully and routed to
> success queue. I checked in App log, but everything seems to be normal for
> the flow file. I don't see anything mysterious in provenance too (except
> that queue time is in hours).
>
> Has anyone else faced a similar issue? What else should I check to identify
> the root cause for this?
>
> Thanks,
> Manish

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

Please produce and share the thread dump I mentioned.

Thanks
Joe

On Dec 1, 2016 7:23 AM, "Manish G"  wrote:

> Hi,
>
> I don't know why, but this is happening now more frequently. Where should
> I look into to find the root cause?
>
> Thanks,
> Manish
>
> On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:
>
>> Hi Joe,
>>
>> Thanks for the quick reply. Yes, the processor keeps running on a single
>> thread (even after stopping). And the number remains there even after
>> stopping.
>> Today, it happened on my customized putHDFS processor. Only thing
>> different in this processor is - I have added an additional attribute that
>> tells if the processor created the directory while loading the file on
>> HDFS. I don't think this should be the issue though.
>>
>> Regards,
>> Manish
>>
>>
>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt  wrote:
>>
>>> Manish
>>>
>>> When it is stuck do you see a number in the top right corner of the
>>> processor?  When you stop it does the number remain?  That number is
>>> telling you how many threads are still executing.  Which processor are
>>> we talking about?  When it is in the stuck state can you please run
>>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> would aid us in narrowing in on a possible cause.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G 
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have noticed that sometime a flow file gets stuck on a processor for
>>> a
>>> > very long time for no reason and then I can not even stop the
>>> processor to
>>> > look at the flow flow file from queue. If I click on stop, then
>>> processor
>>> > goes into a state where I cannot start/stop the processor.
>>> >
>>> > On restarting the NiFi, the file gets processed successfully and
>>> routed to
>>> > success queue. I checked in App log, but everything seems to be normal
>>> for
>>> > the flow file. I don't see anything mysterious in provenance too
>>> (except
>>> > that queue time is in hours).
>>> >
>>> > Has anyone else faced a similar issue? What else should I check to
>>> identify
>>> > the root cause for this?
>>> >
>>> > Thanks,
>>> > Manish
>>>
>>
>>
>>
>> --
>>
>>
>> *With Warm Regards,*
>> *Manish*
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

When it is in the stuck state can you please run
bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.

Thanks
Joe

On Thu, Dec 1, 2016 at 8:44 AM, Manish G  wrote:
> Hi Joe,
>
> Here is what I can see in the App Log:
>
> 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> testing.nifi.processor.hdfs.testingPutHDFS ""
> org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException:
> An error occurred while enumerating the result, check the original exception
> for details.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> ~[hadoop-azure-2.7.2.jar:na]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[hadoop-common-2.7.2.jar:na]
> at
> testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
> ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_101]
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> Caused by: java.util.NoSuchElementException: An error occurred while
> enumerating the result, check the original exception for details.
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
> ~[azure-storage-2.0.0.jar:na]
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> ~[hadoop-azure-2.7.2.jar:na]
> ... 15 common frames omitted
> Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> at
> com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> ~[azure-storage-2.0.0.jar:na]
> ... 17 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
> ~[azure-storage-2.0.0.jar:na]
> ... 18 common frames omitted
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt  wrote:
>>
>> Manish
>>
>> Please produce and share the thread dump I mentioned.
>>
>> Thanks
>> Joe
>>
>> On Dec 1, 2016 7:23 AM, "Manish G"  wrote:
>>>
>>> Hi,
>>>
>>> I don't know why, but this is happening now more frequently. Where should
>>> I look into to find the root cause?
>>>
>>> Thanks,
>>> Manish
>>>
>>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:
>>>>
>>>> Hi Joe,
>>>>
>>>> Thanks for the quick reply. Yes, the processor keeps running on a single
>>>> thread (even after stopping). And the number remains there even after
>>>&

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

It also appears from the log output you provided so far that this is a
combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
and it is not likely to work.

Please try the flow from a single release line.  Apache NiFi 1.1.0 is
available for use now.

Thanks
Joe

On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
> the issue again.
>
> Thanks,
> Manish
>
> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>>
>> Manish
>>
>> When it is in the stuck state can you please run
>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G  wrote:
>> > Hi Joe,
>> >
>> > Here is what I can see in the App Log:
>> >
>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> > org.apache.hadoop.fs.azure.AzureException:
>> > java.util.NoSuchElementException:
>> > An error occurred while enumerating the result, check the original
>> > exception
>> > for details.
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> > ~[hadoop-common-2.7.2.jar:na]
>> > at
>> >
>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> > at
>> >
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> > [nifi-api-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> > [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> > Source) [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> > Source) [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> > Caused by: java.util.NoSuchElementException: An error occurred while
>> > enumerating the result, check the original exception for details.
>> > at
>> >
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > ... 15 common frames omitted
>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.transla

Re: Flow File Stuck for no reason

[Joe Witt]

manish

my apologies now I see what you mean.  It is 0.7.0 nifi and you have a
custom nar that is 1.0.0 which is totally fine.

Bryan's response though is a good one to follow up on.

Thanks
joe

On Thu, Dec 1, 2016 at 9:16 AM, Bryan Bende  wrote:
> Manish,
>
> Was the reason for your custom version of PutHDFS just to include the Azure
> libraries? or was there other functionality you needed to introduce?
>
> The reason I'm asking is because in the Apache NiFi 1.1 release (just
> released) we added an "Additional Classpath Resources" property to all HDFS
> processors, and I have been able to use that to connect to an Azure WASB
> filesystem.
>
> I can provide the list of JARs I added to the classpath to make it work if
> that would help.
>
> Thanks,
>
> Bryan
>
>
> On Thu, Dec 1, 2016 at 12:11 PM, Manish G  wrote:
>>
>> Hi Joe,
>>
>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
>> use of 1.0?
>>
>> Regards,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt  wrote:
>>>
>>> Manish
>>>
>>> It also appears from the log output you provided so far that this is a
>>> combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
>>> and it is not likely to work.
>>>
>>> Please try the flow from a single release line.  Apache NiFi 1.1.0 is
>>> available for use now.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>> > see
>>> > the issue again.
>>> >
>>> > Thanks,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> When it is in the stuck state can you please run
>>> >> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> >> would aid us in narrowing in on a possible cause.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G 
>>> >> wrote:
>>> >> > Hi Joe,
>>> >> >
>>> >> > Here is what I can see in the App Log:
>>> >> >
>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>> >> > java.util.NoSuchElementException:
>>> >> > An error occurred while enumerating the result, check the original
>>> >> > exception
>>> >> > for details.
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.contro

Re: Flow File Stuck for no reason

[Joe Witt]

Hello

This part of the stack dump is the only thing that appears out of
place to me.  it appears to be attempting to read from the socket or
stuck there.  Perhaps you can use a more recent client library for
that custom HD processor?

"Timer-Driven Process Thread-8" Id=78 RUNNABLE  (in native code)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- waiting on java.io.BufferedInputStream@7b23ac8
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at 
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
at 
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
at 
org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
at 
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
at 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
at 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
at 
custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78

Thanks
Joe

On Tue, Dec 6, 2016 at 9:05 AM, Manish G  wrote:
> Hi,
>
> The issue happened again today. I restarted the NiFi and it started working
> again. Below are the rows from bootstrap.log after nifi.sh dump.
> Could this be because of some other process like diagnostic monitor on Azure
> storage blob?
>
> Thanks,
> Manish
>
>
> 2016-12-06 08:41:47,079 INFO [main] org.apache.nifi.bootstrap.RunNiFi "
> "Attach Listener" Id=6 RUNNABLE
>
> "AzureNativeFilesystemStore-UploadBandwidthUpdater" Id=94 TIMED_WAITING  on
> null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.azure.metrics.BandwidthGaugeUpdater$UploadBandwidthUpdater.run(BandwidthGaugeUpdater.java:266)
> at java.lang.Thread.run(Unknown Source)
>
> "Cleanup Archive for default" Id=54 RUNNABLE
> at sun.nio.fs.WindowsNativeDispatcher.FindClose(Native Method)
> at sun.nio.fs.WindowsDirectoryStream.close(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkAccess(Unknown Source)
> at java.nio.file.Files.exists(Unknown Source)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1307)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1546)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.co

Re: How to integrate a custom protocol with Apache Nifi

[Joe Witt]

Kant

Typically this is as simple as creating your processor to utilize that protocol.

Take a look at the developer guide [1] and of course you can always
look at the many example processors and components that come in NiFi
out of the box [2].

[1] https://nifi.apache.org/docs.html
[2] https://git-wip-us.apache.org/repos/asf/nifi.git

Thanks
Joe

On Tue, Dec 6, 2016 at 6:46 PM, kant kodali  wrote:
> HI All,
>
> I understand that Apache Nifi has integration with many systems but what If
> I have an application that talks a custom protocol ? How do I integrate
> Apache Nifi with the custom protocol?
>
> Thanks,
> kant

Re: Nifi Vs Messaging Brokers (Kafka/Rabbitmq)

[Joe Witt]

Raj,

You can use NiFi to replace aspects of these things for general
purpose cases.  In some narrow but important cases it is a reasonable
thing to do but I do not recommend it generally.

Kafka will be far faster and better than NiFi at acting like Kafka.

RabbitMQ will be too.

I recommend thinking about your architecture in such a way that has
both messaging and flow management.  You can see more about my
thoughts on this subject here [1].

[1] https://www.youtube.com/watch?v=sQCgtCoZyFQ&t=1902s

Thanks
Joe

On Fri, Dec 9, 2016 at 12:46 PM, Raj A  wrote:
> Still new to Nifi and was wondering if Nifi can be used as a replacement for
> messaging brokers like Kafka or RabbitMQ in an enterprise application?
>
> I am thinking if we have NiFi processors act as Publishers and Consumers
> would Nifi framework provide all the features (like guaranteed delivery,
> reliability, routing, clustering, HA etc)
> If yes, how comparable is the performance compared to Kafka (100k msg/s) or
> Rabbit (20k msg/sec) ?
>
> Thanks
> -Raj

Re: Failing Tests during Build

[Joe Witt]

Hello

Those tests rely on DNS which appears to be in a funky state.
However, to get the build working otherwise you can run

mvn -T C2 clean install -DskipTests

Thanks
joe

On Wed, Dec 14, 2016 at 10:07 AM, Pushkar  wrote:
> Hi,
>
> I tried to build NiFi on my linux machine using 'mvn -T 2.0C clean install'.
> However there seems to be a few tests failing which I don't seem to be able
> to fix. I have pasted the stacktrace below. Please let me know how I can fix
> this and get the build running.
>
> Please note, I haven't made any changes to the code. I pulled the master
> branch and attempted to build.
>
> Pushkar
>
> -
> nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/target/surefire-reports/org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest-output.txt
>
> [main] INFO org.apache.nifi.io.socket.SocketListener - Now listening for
> connections from nodes on port 0
> [Process Cluster Protocol Request-1] WARN
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener - Failed
> processing protocol message from localhost.localdomain due to
> java.io.IOException: Failed reading protocol message due to malformed header
> java.io.IOException: Failed reading protocol message due to malformed header
> at
> org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:115)
> at
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142)
> at
> org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [main] INFO org.apache.nifi.io.socket.SocketListener - Socket Listener has
> been terminated successfully.
> [Cluster Socket Listener] WARN org.apache.nifi.io.socket.SocketListener -
> Failed to communicate with Unknown Host due to java.net.SocketException:
> Socket closed
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> at java.lang.Thread.run(Thread.java:745)
> [main] INFO org.apache.nifi.io.socket.SocketListener - Now listening for
> connections from nodes on port 0
> [Process Cluster Protocol Request-1] INFO
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener - Finished
> processing request a3e83b60-05ee-47a0-a636-d528bea91cd3 (type=PING,
> length=129 bytes) from localhost.localdomain in 3935 millis
> [main] INFO org.apache.nifi.io.socket.SocketListener - Socket Listener has
> been terminated successfully.
> [main] INFO org.apache.nifi.io.socket.SocketListener - Now listening for
> connections from nodes on port 0
> [Process Cluster Protocol Request-1] INFO
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener - Finished
> processing request 85ecda07-1d72-4109-84d5-f5f882194b76 (type=PING,
> length=129 bytes) from localhost.localdomain in 1 millis
> [main] INFO org.apache.nifi.io.socket.SocketListener - Socket Listener has
> been terminated successfully.
>
>
>
> -
> nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/target/surefire-reports/org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest.txt
>
> ---
> Test set: org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest
> ---
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.767 sec
> <<< FAILURE! - in
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest
> testDelayedRequest(org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest)
> Time elapsed: 4.358 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListenerTest.testDelayedRequest(SocketProtocolListenerTest.java:128)
>
>

Re: File causes unexpected stoppage in play

[Joe Witt]

James,

Can you please share the full log entry for that failure.  It is
possible the processor is not catching the exception properly and
routing the data to failure.  It might simply be letting the exception
loose thus the framework detects the issue and rollsback the session
and yields the processor.

Likely an easy thing to fix in the processor but please do share the
full nifi-app.log entry for this.

Thanks
Joe

On Wed, Dec 14, 2016 at 10:47 AM, James McMahon  wrote:
> Hello. Am wondering if anyone knows how to overcome a challenge with
> unmappable file characters? I have used a GetFile processor to bring a large
> number of zip files into a NiFi flow. All read in successfully. I try to
> then use an UnpackContent processor to unzip the files to their individual
> file members. Most work just fine. However there appears to be a file that
> throws this error in UnpackContent:
>
> failed to process session due to java.nio.file.InvalidPathException:
> Malformed input or input contains unmappable characters
>
> My processing stalls. Nothing else flows. What is the proper way to
> configure the UnpackContent processor step so that it shuttle such files off
> to the side when it encounters them, and permits the other files waiting in
> queue to process? I do now have a "failure" path established for my
> UnpackContent processor, but for some reason it does not send these problem
> files down that path. I suspect it may be because the zip files does unpack
> successfully but the underlying file(s) within the zip cause processing to
> choke.
>
> How can I engineer a flow to overcome this challenge? Thanks in advance for
> your help.

Re: Handling incoming file names that contain embedded spaces

[Joe Witt]

James,

I suspect there is more to the issue than the spaces.  GetFile itself
should be fine there.  Can you share logs showing what is happening
with these files?  Can you share some sample filenames that it is
struggling with?  You can also enable debug logging for that processor
which could provide some interesting details as well.

Thanks
Joe

On Wed, Dec 14, 2016 at 5:03 PM, James McMahon  wrote:
> I am using NiFi 0.6.1. I am trying to use GetFile to read in a large series
> of files I have preprocessed outside of NiFi from zip files using bash shell
> commands. GetFile is throwing errors on many of these files because the
> files contain embedded spaces. Is there a way to tell NiFi to handle each
> such filename with surrounding single quotes? Are there other processor
> options better suited to handle this challenge? Thank you.

Re: Handling incoming file names that contain embedded spaces

[Joe Witt]

James,

I think this would need to be tested and evaluated.  I'm not quite
sure offhand and also it has to do with the operating system involved.
Any details you can provide of your own findings and environment will
be helpful.

Thanks
Joe

On Thu, Dec 15, 2016 at 4:38 PM, James McMahon  wrote:
> Is the NiFi GetFile processor restricted to a range of characters, or is it
> able to handle Unicode character encoding? If it is able to map Unicode
> characters then there must be another problem causing my GetFile processor
> to throw these "unmappable characters" errors. Thank you for any thoughts.
>
> On Thu, Dec 15, 2016 at 6:07 AM, James McMahon  wrote:
>>
>> As a representative example using a random Unicode character at the front
>> and the back of a notional file name, [U+0932][U+0932][U+0932]+123
>> ABC[U+07C1]
>>
>> On Wed, Dec 14, 2016 at 7:22 PM, James McMahon 
>> wrote:
>>>
>>> Yes indeed Joe, it appears from the logs that there are non-ASCII unicode
>>> characters preceding and at end of the file name. The log shows them as odd
>>> representations of "unprintables" - for example, small inverted question
>>> marks in diamonds, etc etc. They are embedded in the file names by the
>>> application that created the files. I copied and tried to paste and save
>>> into a text file, and notepad directed me to switch to another encoding in
>>> order to save the file name string. I was able to get it to save by
>>> switching to Unicode encoding.
>>>
>>> I can't send the logs from my system. I can only relay this in this way.
>>> Would you expect that such character encoding would cause problems for
>>> GetFile? What alternatives do I have to work around this problem? Thank you
>>> once again.
>>>
>>> On Wed, Dec 14, 2016 at 6:04 PM, Joe Witt  wrote:
>>>>
>>>> James,
>>>>
>>>> I suspect there is more to the issue than the spaces.  GetFile itself
>>>> should be fine there.  Can you share logs showing what is happening
>>>> with these files?  Can you share some sample filenames that it is
>>>> struggling with?  You can also enable debug logging for that processor
>>>> which could provide some interesting details as well.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Wed, Dec 14, 2016 at 5:03 PM, James McMahon 
>>>> wrote:
>>>> > I am using NiFi 0.6.1. I am trying to use GetFile to read in a large
>>>> > series
>>>> > of files I have preprocessed outside of NiFi from zip files using bash
>>>> > shell
>>>> > commands. GetFile is throwing errors on many of these files because
>>>> > the
>>>> > files contain embedded spaces. Is there a way to tell NiFi to handle
>>>> > each
>>>> > such filename with surrounding single quotes? Are there other
>>>> > processor
>>>> > options better suited to handle this challenge? Thank you.
>>>
>>>
>>
>

Re: guaranteed delivery

[Joe Witt]

Hello

NiFi's data durability model can be well understood by reading here
[1].  This addresses how data is handled once it is under NiFi's
control.

So then there is the matter of considering the safety of data as it is
coming 'into nifi' and 'being sent out of nifi'.

When exchanging data between processes you have generally the
following things to consider and these are well discussed on-line.
1) At least once
2) At most once
3) Exactly once.

So when bring data 'into nifi' and 'out of nifi' we do generally
provide by default for 'at least once'.  It is the case that
duplicates can be received and duplicates can be sent then.  But it is
also the case that generally speaking you will not lose data
(obviously nifi cannot control what external processes do to the
data).

Achieving exactly once behavior requires the two processes to have
some agreed upon protocol which offers this behavior.  It could be
something formal like 'two phase commit' or it could be something
informal where you as an administrator have fixed all the variables to
the best you can to ensure there will be no issues.  So lets look at
your example.  You want to pull files via FTP.

There are two critical steps to this:
1) Copy bytes from remote system to nifi using FTP
2) Change the state of the remote system so we don't keep pulling the
file (either removal or rename).

If you do step 1 then step 2 you are basically implementing at least
once behavior.  You wont lose data using such a protocol but you could
have duplicates.

If you do step 2 then step 1 you are implementing at least once (not
exactly once).  It is at least once because what if there was some
error in nifi when committing the state of things.

The other thing you mentioned was sequence of processing.  NiFi will
by default process data in the order it received and as it goes
between processes it is placed in a prioritized queue.  Obviously if
you're running multiple threads and such then ordering can vary.

There is a lot here to discuss and cases to consider so happy to keep
the discussion going.  Hopefully this helps you see the difference
between 'nifi the framework and its durability model' versus 'nifi
processors and patterns of using them together'.

[1] https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html

Thanks
Joe

On Mon, Jan 2, 2017 at 3:19 PM,   wrote:
> I am reviewing NiFi as a possible replacement for an internally developed
> data transfer tool.
>
> NiFi appears to be of the "the data is guaranteed to be delivered at least
> once" variety. Where my needs are "the data is guaranteed to be delivered
> once"; to the point that I'm willing to manually review and resolve failures
> that occur "beyond the point of no return" to ensure 1x delivery and no data
> loss.
> Some of my data transfers are of the sequential transactional type, where
> they must be transferred and processed, in sequence.
>
> Take for instance GetFTP, I see holes in the commit model (from the
> perspective of what I'm trying to accomplish).
> Looking at GetFTP (via FetchFileTransfer.java), session.commit() occurs
> before deleting or renaming the source file. So, if that step fails the file
> will be retrieved and processed again and again.
> PutSQL appears to have similar issues as it relates to updating a database
> more than once should the transfer die before the db commit is recognized by
> NiFi, so the the FlowFile(s) get rolled back.
>
> Are my needs outside of Nifi's objectives?
>

Re: guaranteed delivery

[Joe Witt]

As far as the scenario you mentioned with FTP let's identify the
sequence in more detail because i wanted to be more general on the
phase about changing the remote system to avoid pulling the data
again.

Phases of NiFi interacting with an SFTP server (lets talk about GetSFTP still):

(P)ULL: Pull data of a file on remote SFTP server into nifi's
repository (not yet committed)
(C)OMMIT: Tell nifi to commit its session.
(S)AVESTATE: Either keep track of state about the remote system in
nifi OR change the remote file via delete or rename.

The default behavior we've built for is (P)(C)(S) which is at least
once and is susceptible to duplicates in the presence of ill-timed
failures.  The duplication could happen if after (C) and before (S)
happens or before (S) fully completes there is some failure either in
NiFi or in communicating with the remote system.

We could offer alternatively (P)(S)(C) which is at most once and is
susceptible to data loss in the presence of ill-timed failures.  The
loss could happen if after (S) and before or during (C) some failure
occurs in NiFi.  Failures before will simply cause a rollback and
retry and a no-harm done so to speak.

Ok, so I think we're on the same page about the failure cases/logic.
Your recent email had an important clarifying note about being single
threaded and indeed wanting to effectively stop the pipeline in the
presence of some failing data.  You might find it better to create
very specific/precise processors against your desired behavior.
Though, you may be able to use the out of the box processors and the
backpressure mechanism of NiFi (set backpressure at 1 for example).
It sounds like the source process has a very specific way of writing
data into the SFTP accessible location and with some sort of a naming
or timestamping scheme to help you ensure ordering and it sounds like
the receiving system (whatever NiFi is delivering to) has a very
specific set of preconditions to honor.  If the out of the box
processors and prescriptive settings dont' get you the results you
seek you might find that using a scripted processor to behave
precisely as you need is the best or writing a custom processor.

Thanks
Joe

On Mon, Jan 2, 2017 at 6:19 PM,   wrote:
> Yes, I had read [1]. I'm reasonably satisfied that nifi is keeping track
> of what's happening to the data when its "within it's walls". That, and
> it's inherent design to handle files vs. little XML messages, was what
> piqued my interest.
> My concerns are related 'coming into nifi' and 'being sent out of nifi'.
>
> In the FTP example, if you do step 1 and hold onto the FlowFile (now
> safely within nifi); then step 2 (let's pick DELETE) -- but the FTP
> connection fails between the DELETE and the remote FTP side returning
> SUCCESS, so there's no way for nifi to know if the delete succeeded or
> failed.  I politely left out the possibility of nifi dieing unexpectedly
> in that same timeframe.
>
> For my needs I don't want the FlowFile to be tossed away; I need nifi to
> hold onto it (and let me at it) until I can resolve the problem manually
> -- because I still need the data. And, because of the sequential
> processing requirements for the data in question I can't allow any more
> files to be FTP'd in from the source directory until the problem has
> been resolved.
> I would not be running multiple threads for these types of transfers.
>
> PutSQL has similar issues -- just going the other way. The data has
> updated the database but nifi didn't get the SUCCESS reply -- so let's
> send the data again!
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html
>
>
> On 01/02/2017 12:52 PM, Joe Witt wrote:
>> Hello
>>
>> NiFi's data durability model can be well understood by reading here
>> [1].  This addresses how data is handled once it is under NiFi's
>> control.
>>
>> So then there is the matter of considering the safety of data as it is
>> coming 'into nifi' and 'being sent out of nifi'.
>>
>> When exchanging data between processes you have generally the
>> following things to consider and these are well discussed on-line.
>> 1) At least once
>> 2) At most once
>> 3) Exactly once.
>>
>> So when bring data 'into nifi' and 'out of nifi' we do generally
>> provide by default for 'at least once'.  It is the case that
>> duplicates can be received and duplicates can be sent then.  But it is
>> also the case that generally speaking you will not lose data
>> (obviously nifi cannot control what external processes do to the
>> data).
>>
>> Achieving exactly once behavior requires the two processes to have
>&

Re: GetFile: why must source directory be writeable...

[Joe Witt]

If we don't need to rename or delete orig yes it is heavy handed and can be
made easily better.

On Jan 6, 2017 4:03 PM, "Russell Bateman"  wrote:

> I see that it's okay for the files themselves to be read-only, but it
> seems heavy-handed to enforce writeability on the parent subdirectory, no?
>
> On 01/06/2017 02:00 PM, Russell Bateman wrote:
>
> ...when Keep Source File is set to true and the sources are being
> protected anyway against deletion? Is this an oversight that should be
> JIRA'd?
>
>
>

Re: NiFi UI fails to display Processor Group: NullPointerException

[Joe Witt]

We are definitely going to make this case (misaligned node installs)
easier to identify.

On Fri, Jan 6, 2017 at 6:22 PM, Kevin Verhoeven
 wrote:
> Matt,
>
>
>
> Your instincts were spot on, one of the 12 nodes had not been properly
> updated. I stopped the nodes one-by-one until I isolated the broken install
> and then reinstalled 1.1.1 on that node. The cluster is now operating as
> expected.
>
>
>
> Thanks so much for your input, I appreciate your help.
>
>
>
> Kevin
>
>
>
> From: Kevin Verhoeven [mailto:kevin.verhoe...@ds-iq.com]
> Sent: Friday, January 6, 2017 11:57 AM
> To: users@nifi.apache.org
> Subject: RE: NiFi UI fails to display Processor Group: NullPointerException
>
>
>
> Unfortunately, I don’t see the stack trace in the app log either. I see many
> heartbeats from SocketProtocolListener and ClusterProtocolHeartbeater, but
> when I cause the UI to fail I see no exceptions or errors to indicate there
> was a problem.
>
>
>
> It makes sense that the GenerateFlowFile is causing this problem and knowing
> that it was updated in 1.1.0 confirms this. I checked all nodes and I see
> they are all running 1.1.1. So it appears that the update was successful on
> all nodes. I might try to stop the nodes one-by-one and see if I can catch
> different results. This seems like a long-shot, but I’ll try anything!
>
>
>
> Kevin
>
>
>
> From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
> Sent: Friday, January 6, 2017 10:49 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi UI fails to display Processor Group: NullPointerException
>
>
>
> Can you check the app log for the stack trace? Being a cluster, if there was
> a bug in the response merging logic it may be logged outside the user log.
>
>
>
> Also, is it possible that all the nodes did not get upgraded? I'm pretty
> sure that GenerateFlowFile received some new properties in 1.1.0. I wonder
> if there is an issue when the different nodes have different sets of
> properties for the same component.
>
>
>
> Matt
>
>
>
> On Fri, Jan 6, 2017 at 1:25 PM, Kevin Verhoeven 
> wrote:
>
> I’ll enable DEBUG level logging and see if I can capture anything.
>
>
>
> Navigating into the Processor Group still works in my 1.0.0 installation, it
> is only when I updated a cluster to 1.1.1 that I started seeing this
> behavior. I have two clusters, Dev and Prod and I updated Dev. Prod remains
> 1.0.0.
>
>
>
> The problem also happens on new Processor Groups. Here’s my test:
>
>
>
> 1. Create new Processor Group.
>
> 2. Browse into the Processor Group.
>
> 3. Add UpdateAttribute Processor, works.
>
> 4. Add GenerateFlowFile Processor, fails. (error is the same error as
> above).
>
>
>
> At this point I cannot enter my new Processor Group.
>
>
>
> Strangely, my Processor Groups that do not have a GenerateFlowFile Processor
> still work. So I might have just narrowed this down to one Processor,
> GenerateFlowFile.
>
>
>
> Kevin
>
>
>
> From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
> Sent: Friday, January 6, 2017 10:18 AM
>
>
> To: users@nifi.apache.org
> Subject: Re: NiFi UI fails to display Processor Group: NullPointerException
>
>
>
> Thanks for the extra details. So it is the endpoint for loading the group
> which contains the necessary configuration and statistics to generate the
> graph. Can you try enabling DEBUG level logging for this package in your
> conf/logback.xml?
>
>
>
> org.apache.nifi.web.api.config
>
>
>
> Does navigating into this Process Group still work in your 1.0.0
> installation? Do you know if there is anything unique about the contents of
> that group? I'm just trying to find something that might help point us in
> the correct direction so we can figure out a work around and get the issue
> addressed for upcoming releases.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> On Fri, Jan 6, 2017 at 1:01 PM, Kevin Verhoeven 
> wrote:
>
> Thank you for your response, I really appreciate your help. I reviewed the
> user log and did not see a stack trace after the NPE message. Looking
> backward in the user log I see other log messages for the same thread and I
> found the endpoint being requested:
>
>
>
> 2017-01-06 17:45:44,338 INFO [NiFi Web Server-3798]
> org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
> GET
> http://servername:1/nifi-api/flow/process-groups/e8a4ad16-1b4e-3afc-a8e0-0a2cdda95632
> (source ip: ipaddress)
>
> …
>
> 2017-01-06 17:45:44,360 ERROR [NiFi Web Server-3798]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException. Returning Internal Server Error
> response.java.lang.NullPointerException: null
>
>
>
> Using Developer Tools in Chrome I compared the endpoint and it matches,
> here’s what Chrome indicated:
>
>
>
> GET
> http://servername:1/nifi-api/flow/process-groups/e8a4ad16-1b4e-3afc-a8e0-0a2cdda95632
> 500 (Internal Server Error)
>
> An unexpected error has occurred. Please check the logs for additional
> details.
>
>
>
>
>
> send  @jquery-2

Re: Deleting millions of files from a queue...

[Joe Witt]

Millions or gajillions will indeed take a while as they have to swap in as
presently implemented.  We could certainly optimize that if is a common
need.

Blowing away the repos will certainly do the trick and be faster.  Though
is clearly a blunt instrument.

Do you think we need an express queue killer option?

On Jan 10, 2017 1:32 PM, "Russell Bateman"  wrote:

If I'm experimenting and have gajillions of flowfiles in a queue that takes
a very long time to empty from the UI, is there a quicker way? I can
certainly bounce NiFi, delete files, both, etc.

[SECURITY] CVE-2016-8748: Apache NiFi XSS vulnerability in connection details dialogue

[Joe Witt]

CVE-2016-8748: Apache NiFi XSS vulnerability in connection details dialogue

Severity: Moderate

Vendor: The Apache Software Foundation

Versions Affected:
Apache NiFi 1.0.0
Apache NiFi 1.1.0

Description: There is a cross-site scripting vulnerability in
connection details dialog when accessed by an authorized user. The
user supplied text was not be properly handled when added to the DOM.

Mitigation:
1.0.0 users should upgrade to 1.0.1 or 1.1.1.
1.1.0 users should upgrade to 1.1.1.  Additional migration guidance
can be found 
https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance.

Credit: This issue was discovered by Matt Gilman of the Apache NiFi
PMC during a code review.

References: https://nifi.apache.org/security.html

Re: ConsumeKafka Processor Dropping Messages

[Joe Witt]

Hello

Can you share a bit more about the details of the ConsumeKafka
processor and its configuration.  What are the settings you have?  Can
you describe a bit more about the input data and how you're
determining there is loss?

Thanks
Joe

On Wed, Jan 25, 2017 at 2:47 PM, Samra Kasim
 wrote:
> Hi,
>
> I am new to NiFi and I am reading off a Kafka topic that has 3 partitions.
> In my Nifi flow, I have 3 ConsumeKafka processors with the same groupId and
> Topic. However, when I push large datasets (e.g., 200,000+), 300-400 records
> don't make it to the next processor. This only happens when I have the
> Concurrent Tasks in the Scheduling Tab set to more than 1 (e.g., 2 or 3). If
> I have the Concurrent Tasks set to 1 then all the records make it through to
> the next processor just fine.
>
> I may need to define kafka.partitions to have each Nifi processor point to a
> specific Kafka partition, but am not sure where/how to do that. I tried
> adding it to the properties, but that doesn't work. Has anyone else worked
> through this issue?
>
> I am using Nifi 1.1.1 and Kafka 0.9
>
> --
>
>
> Sam
>
>

Re: ConsumeKafka Processor Dropping Messages

[Joe Witt]

Thanks.  Could you just let the data queue up in NiFi so you can see
the backlogged count?  I'm concerned about overlapping filenames/IO
issues that could impact how much gets written to disk as well.

Thanks
Joe

On Wed, Jan 25, 2017 at 3:05 PM, Samra Kasim
 wrote:
> Sure. On the Configure Processors I have set:
> Properties:
> - Kafka Brokers to the 3 broker nodes on my cluster
> - Security Protocol: PLAINTEXT
> - Topic Name: testtopic
> - Group ID: testgroup
> - Offset Reset: latest
> - Key Attribute Encoding: UTF-8 Encoded
> - Max Poll Records: 1
> - Max Uncommited Time: 1 secs
>
> In scheduling, I have the Concurrent Tasks to be 1 for each processor.
>
> The input data is a json string. I have a test data set of 200,000 records.
> My flow goes from the 3 ConsumeKafka processors to a PutFile processor,
> which saves the flowfiles to a directory. I do a count on the directory and
> so am able to see if all the records made it.
>
> On Wed, Jan 25, 2017 at 2:57 PM, Joe Witt  wrote:
>>
>> Hello
>>
>> Can you share a bit more about the details of the ConsumeKafka
>> processor and its configuration.  What are the settings you have?  Can
>> you describe a bit more about the input data and how you're
>> determining there is loss?
>>
>> Thanks
>> Joe
>>
>> On Wed, Jan 25, 2017 at 2:47 PM, Samra Kasim
>>  wrote:
>> > Hi,
>> >
>> > I am new to NiFi and I am reading off a Kafka topic that has 3
>> > partitions.
>> > In my Nifi flow, I have 3 ConsumeKafka processors with the same groupId
>> > and
>> > Topic. However, when I push large datasets (e.g., 200,000+), 300-400
>> > records
>> > don't make it to the next processor. This only happens when I have the
>> > Concurrent Tasks in the Scheduling Tab set to more than 1 (e.g., 2 or
>> > 3). If
>> > I have the Concurrent Tasks set to 1 then all the records make it
>> > through to
>> > the next processor just fine.
>> >
>> > I may need to define kafka.partitions to have each Nifi processor point
>> > to a
>> > specific Kafka partition, but am not sure where/how to do that. I tried
>> > adding it to the properties, but that doesn't work. Has anyone else
>> > worked
>> > through this issue?
>> >
>> > I am using Nifi 1.1.1 and Kafka 0.9
>> >
>> > --
>> >
>> >
>> > Sam
>> >
>> >
>
>
>
>
> --
>
>
> Samra Kasim
>
> Technologist
> HUMANgEO
>
> Virginia Office
> 4350 N Fairfax Drive
> Suite 950
> Arlington, VA 22203
>
> E-Mail:  samra.ka...@thehumangeo.com
> Web:http://www.thehumangeo.com/

Re: ConsumeKafka Processor Dropping Messages

[Joe Witt]

Samra,

Another thing that comes to mind when testing this is to ensure before
each run you clear out the state/offset we're tracking for that client
and change the 'Group ID'.

For each thread the processor has it will be assigned zero to n
partitions.  A thread could get zero partitions if there are more
processor threads than there are partitions in kafka for instance
(this would be pretty odd but just throwing that out there).
Otherwise a nifi processor thread will have 1..n partitions it is
responsible for.  The client will ascertain its proper offset from
kafka (latest meaning it will start at the end of that partition when
the client is created).  You might also which to set your 'offset
reset' to 'earliest' to ensure there is no race condition between
creating that consumer (and getting its offset) versus when the data
is placed into the partition by whatever producer you have.

Thanks
Joe

On Wed, Jan 25, 2017 at 3:09 PM, Joe Witt  wrote:
> Thanks.  Could you just let the data queue up in NiFi so you can see
> the backlogged count?  I'm concerned about overlapping filenames/IO
> issues that could impact how much gets written to disk as well.
>
> Thanks
> Joe
>
> On Wed, Jan 25, 2017 at 3:05 PM, Samra Kasim
>  wrote:
>> Sure. On the Configure Processors I have set:
>> Properties:
>> - Kafka Brokers to the 3 broker nodes on my cluster
>> - Security Protocol: PLAINTEXT
>> - Topic Name: testtopic
>> - Group ID: testgroup
>> - Offset Reset: latest
>> - Key Attribute Encoding: UTF-8 Encoded
>> - Max Poll Records: 1
>> - Max Uncommited Time: 1 secs
>>
>> In scheduling, I have the Concurrent Tasks to be 1 for each processor.
>>
>> The input data is a json string. I have a test data set of 200,000 records.
>> My flow goes from the 3 ConsumeKafka processors to a PutFile processor,
>> which saves the flowfiles to a directory. I do a count on the directory and
>> so am able to see if all the records made it.
>>
>> On Wed, Jan 25, 2017 at 2:57 PM, Joe Witt  wrote:
>>>
>>> Hello
>>>
>>> Can you share a bit more about the details of the ConsumeKafka
>>> processor and its configuration.  What are the settings you have?  Can
>>> you describe a bit more about the input data and how you're
>>> determining there is loss?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Jan 25, 2017 at 2:47 PM, Samra Kasim
>>>  wrote:
>>> > Hi,
>>> >
>>> > I am new to NiFi and I am reading off a Kafka topic that has 3
>>> > partitions.
>>> > In my Nifi flow, I have 3 ConsumeKafka processors with the same groupId
>>> > and
>>> > Topic. However, when I push large datasets (e.g., 200,000+), 300-400
>>> > records
>>> > don't make it to the next processor. This only happens when I have the
>>> > Concurrent Tasks in the Scheduling Tab set to more than 1 (e.g., 2 or
>>> > 3). If
>>> > I have the Concurrent Tasks set to 1 then all the records make it
>>> > through to
>>> > the next processor just fine.
>>> >
>>> > I may need to define kafka.partitions to have each Nifi processor point
>>> > to a
>>> > specific Kafka partition, but am not sure where/how to do that. I tried
>>> > adding it to the properties, but that doesn't work. Has anyone else
>>> > worked
>>> > through this issue?
>>> >
>>> > I am using Nifi 1.1.1 and Kafka 0.9
>>> >
>>> > --
>>> >
>>> >
>>> > Sam
>>> >
>>> >
>>
>>
>>
>>
>> --
>>
>>
>> Samra Kasim
>>
>> Technologist
>> HUMANgEO
>>
>> Virginia Office
>> 4350 N Fairfax Drive
>> Suite 950
>> Arlington, VA 22203
>>
>> E-Mail:  samra.ka...@thehumangeo.com
>> Web:http://www.thehumangeo.com/

Re: ListenTCP to receive CSV stream.

[Joe Witt]

Bryan - quickly read through the ListenTCP code and associated abstract
impl.  It did look feasible to enter a series of characters (some string)
in some Character set and that it would be able to demarcate the stream.
Is there somewhere in there that the hardcoding happens?

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenTCP.java
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventBatchingProcessor.java

On Wed, Jan 25, 2017 at 5:47 PM, Bryan Bende  wrote:

> Raymond,
>
> Currently ListenTCP uses new line characters to determine logical message
> boundaries, and coming out of the processor you can either have 1 logical
> message per flow file, or batch together a configurable number of logical
> messages into 1 flow file which would be more performant.
>
> In your case it sounds like you would want to read data until seeing the
> "end of data" marker and treat the whole CSV as one logical message. There
> is a JIRA to add this capability: https://issues.
> apache.org/jira/browse/NIFI-1985
>
> I think the best you can do currently is to us a MergeContent processor
> somewhere after ListenTCP to merge together the individual lines from the
> CSV, but since there is not other information available to tell it how many
> total lines there are, it can't guarantee that they are all merged together
> in one flow file. You might be able to make some assumptions about the
> timing and size of the data and configure MergeContent in such a way that
> it should usually get you the whole CSV as one file.
>
> Hope this helps.
>
> -Bryan
>
> On Wed, Jan 25, 2017 at 5:18 PM, Raymond Rogers <
> raymond.rog...@rmgnetworks.com> wrote:
>
>> I'm still new to NiFi and I'm trying to receive text stream containing a
>> CSV file of an unknown length (anything from ~100 bytes to almost 300 KB)
>> over a TCP socket.  The CSV does have an "end of data" marker that I can
>> look for but I am unsure of how to accumulate the text until I receive the
>> marker and create a flow-file that contains all of the data up to that
>> point.
>>
>>
>>
>> The data is being sent from an application that cannot changed to use a
>> different format.
>>
>>
>>
>> Any suggestions?
>>
>>
>>
>>
>> *Raymond Rogers*Senior Embedded Software Engineer
>>
>> 15301 N. Dallas Pkwy Suite 500
>> Dallas, TX 75001
>> D: +1 972 744 3928 <(972)%20744-3928>
>> rmgnetworks.com 
>>
>>
>>
>> Notice of Confidentiality: This transmission contains information that
>> may be confidential and that may also be privileged. Unless you are the
>> intended recipient of the message (or authorized to receive it for the
>> intended recipient) you may not copy, forward, or otherwise use it, or
>> disclose its contents to anyone else. If you have received this
>> transmission in error, please notify us immediately and delete it from your
>> system.
>>
>
>

Re: Asynchronous Request/Response Data Flows

[Joe Witt]

Ben,

One way to approach this is using the sort of capabilities this opens
up: https://issues.apache.org/jira/browse/NIFI-190

Certainly is a good case/idea to work through.  Doable and
increasingly seems to be an ask.

Thanks
Joe

On Thu, Jan 26, 2017 at 3:10 PM, Benjamin Janssen  wrote:
> Hello all,
>
> I've got a use case where I get some data, I want to fork a portion of that
> data off to an external service for asynchronous processing, and when that
> external service has finished processing the data, I want to take its
> output, marry it up with the original data, and pass the whole thing on for
> further processing.
>
> So essentially two data flows:
>
> Receive Data -> Store Some State -> Send Data To External Service
>
> Do More Processing On Original Data + Results <-  Retrieve Previously Stored
> State  <-  Receive Results From External Service
>
> Is there a way to do this while taking advantage of NiFi's State Management
> capabilities?  I wasn't finding any obvious processors for persisting and
> retrieving shared state from State Management.  The closest my googling was
> able to get me was this:  https://issues.apache.org/jira/browse/NIFI-1582
> but if I'm understanding the State Management documentation properly, that
> won't actually help me because I'd need the same processor to do all storing
> and retrieving of state?
>
> Does something exist to use State Management like this?  Or is what I'm
> proposing to do a bad idea?
>
> Or maybe I should just be using the DistributedMapCacheServer for this?
>
> Any help/advice would be appreciated.

Re: NiFi 1.1.0 stuck starting, no errors

[Joe Witt]

...ok so we definitely want to figure out why there was another nifi
process still lingering around in the background.  Do you have any
idea why that might have been? Perhaps the startup script has some
corner cases not well covered just yet.

On Fri, Jan 27, 2017 at 12:39 PM, Peter Wicks (pwicks)
 wrote:
> Restarting the box fixed it.
>
>
>
> From: Bryan Rosander [mailto:brosan...@apache.org]
> Sent: Friday, January 27, 2017 8:51 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi 1.1.0 stuck starting, no errors
>
>
>
> Hey Peter,
>
>
>
> The relevant section of the thread dump is below.  It looks like startup is
> blocking while trying to get a file lock for the flowfile repository.  Are
> you positive that there is only one instance of NiFi running with the
> repository directory you've configured?  If so, maybe something failed to
> release the lock in which case a reboot might be worth a try.
>
>
>
> Thanks,
>
> Bryan
>
>
>
> "main" #1 prio=5 os_prio=0 tid=0x7f8da000d000 nid=0xcd68 runnable
> [0x7f8da6daa000]
>
>java.lang.Thread.State: RUNNABLE
>
> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>
> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
>
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1073)
>
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
>
> at
> org.wali.MinimalLockingWriteAheadLog.(MinimalLockingWriteAheadLog.java:187)
>
> at
> org.wali.MinimalLockingWriteAheadLog.(MinimalLockingWriteAheadLog.java:112)
>
> at
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.initialize(WriteAheadFlowFileRepository.java:148)
>
> at
> org.apache.nifi.controller.FlowController.createFlowFileRepository(FlowController.java:638)
>
> - locked <0x0006bfc17cc0> (a
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository)
>
> at
> org.apache.nifi.controller.FlowController.(FlowController.java:447)
>
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375)
>
> at
> org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>
> - locked <0x00067a34fbf8> (a
> java.util.concurrent.ConcurrentHashMap)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
>
> at
> org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>
> - locked <0x00067a34fbf8> (a
> java.util.concurrent.ConcurrentHashMap)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
>
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
>
> at
> 

Re: Deleting an empty group fails

[Joe Witt]

That certainly sounds JIRA worthy though.  Templates should not be able to
block the destruction of a process group.

On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto 
wrote:

> Makoto,
>
> Templates are tied to process groups. Unfortunately, this means that to
> delete the group, you will have to delete the templates first. You can do
> this by using the “hamburger” global menu at the top right of the UI and
> selecting “Templates”, then deleting the templates which reside in this
> process group (export them to XML first if you want to save them).
>
> If you want to move those templates out of the PG, and you are comfortable
> working in XML, a less-enticing option is to stop the application,
> uncompress and open the flow.xml.gz file, and move the template entries out
> of the process group to root or another group, being sure to update any
> references in the XML. Then save and compress the file, and restart the
> application.
>
> I understand this is not an ideal solution. I’m going to file a Jira
> documenting this and suggest that the dialog allows you to select one of
> three options:
>
> 1. Delete process group and all contained templates
> 2. Delete process group and move templates to parent PG (this could cause
> issues because the components and connections will no longer be defined)
> 3. Cancel the delete operation.
>
> I’ll also suggest improving the documentation to make this situation
> clearer.
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto 
> wrote:
>
> Hi,
>
> I'm trying to delete an empty group.
>
> 
>
> However, it fails with the following error.
>
> 
>
> Is something wrong with my operation and could someone would help to
> solve this problem ?
>
> Regards,
> MH
>
>
>

Re: change permission of missing directories

[Joe Witt]

Hello

The directories nifi creates will be made with the permissions its user has
on the operating system.  In Linux you can control this by setting the
appropriate umask for that user.

Thanks
Joe

On Jan 28, 2017 8:44 AM, "mohammed shambakey"  wrote:

> Hi
>
> I'm using PutFile processor to keep received files in a newly made
> direcotry for each user like "/path/${uuid}/received_file". When I
> configure permissions in the PutFile processor, it only affects the
> received file, not the the ${uuid} directory.
>
> I wonder if there's a way to change permissions on the created missing
> directories as well without script?
>
> Regards
>
> --
> Mohammed
>

Re: Many systems - big flow graphs?

[Joe Witt]

Uwe

Happy to have this be a longer dialogue just wanted to get a quick
response back.  You mentioned having unconnected flows separate from
each other.  Yes this is absolutely how it is done for really massive
large scale sets of disparate flows.  The process group concept allows
you to logically create abstractions/groups of flows.  People tend to
name their process groups and processor in meaningful ways.  We have
the search bar in the top-right for fast-find of such components even
if buried very deep in the flow.

This approach works quite well and helps enforce that you keep things
organized and allows you to 'refactor' the flows by moving around and
restructuring groups/etc..

Thanks
Joe

On Thu, Feb 2, 2017 at 5:39 PM, Uwe Geercken  wrote:
> Hello,
>
> excuse my question, but I still have not fully understood how one would
> logically handle large flow graphs. If I have many systems involved and many
> different types of output, would I really put everything in one flow? Or is
> this a misunderstanding on my side? If you put everything in one flow is it
> not getting messy and hard to search and debug? That would have an influence
> on quality at some point in time.
>
> Or would you logically seperate things that do not really belong together
> and run them in flows on different servers - unconnected and seperated from
> the others? Would that be the idea?
>
> I would be happy to hear some of your thoughts or findings from your
> experience.
>
> Rgds,
>
> Uwe
>
>

Re: Nifi service stopped with no 'stop' request?

[Joe Witt]

Cheryl - in addition to the ask from James can you take a look for
anything interesting in the nifi-app.log about ERROR/WARN.

Thanks

On Fri, Feb 3, 2017 at 12:31 PM, James Wing  wrote:
> Cheryl,
>
> I'm not aware of any good explanation for that, or open issues on this
> topic.  Can you share more detail on the OS, JVM, NiFi versions?
>
> Thanks,
>
> James
>
> On Thu, Feb 2, 2017 at 12:14 PM, Cheryl Jennings  wrote:
>>
>> Hi Everyone,
>>
>> I had nifi running overnight, communicating between two nodes.  When I
>> checked in the morning, the nifi service on one of the nodes had stopped.
>>
>> Normally when issuing a 'service nifi stop', I see an entry in
>> nifi-bootstrap.log that says:
>> org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown
>> Command and is shutting down now
>>
>> But this time, I only see this message in nifi-app.log, and no indication
>> in nifi-bootstrap.log that
>> a Shutdown Command was issued:
>> org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
>>
>> Is there any reason nifi would stop like this without an explicit `service
>> stop|restart`?
>>
>> Thanks!
>> -Cheryl
>
>

Re: ConsumeKafka processor erroring when held up by full queue

[Joe Witt]

yeah this is probably a good case/cause for use of the pause concept
in kafka consumers.

On Thu, Feb 9, 2017 at 9:49 AM, Bryan Bende  wrote:
> I believe you are running into this issue:
>
> https://issues.apache.org/jira/browse/NIFI-3189
>
> When back-pressure happens on the queue coming out of ConsumeKafka,
> this can last for longer than session.timeout.ms, and when the
> processors resumes executing it receives this error on the first
> execution. We should be able to implement some type of keep-alive so
> that even when the processor is not executing, there is a background
> thread, or some way of keeping the connections alive.
>
> I believe any user-defined properties in the processor get passed to
> the Kafka consumer, so I believe you could add "session.timeout.ms"
> and set a much higher value as a possible work around.
>
> Thanks,
>
> Bryan
>
> On Thu, Feb 9, 2017 at 8:42 AM, Koji Kawamura  wrote:
>> Hello Nick,
>>
>> First, I assume "had a queue back up" means have a queue being
>> back-pressure. Sorry if that was different meaning.
>>
>> I was trying to reproduce by following flow:
>> ConsumeKafka_0_10
>>   -- success: Back Pressure Object Threshold = 10
>> -- UpdateAttribute (Stopped)
>>
>> Then I used ./bin/kafka-console-producer.sh to send 11 messages.
>> The result was, when NiFi received 10th messages, the success
>> relationship back-pressure was enabled.
>> When I published the 11th message, NiFi didn't do anything.
>> This is expected behavior because downstream connection is
>> back-pressured, the processor won't be scheduled.
>>
>> After I started UpdateAttribute and the queued flow files went
>> through, ConsumeKafka was executed again and received the 11th
>> message.
>>
>> Also, I checked the ConsumerLease and ConsumeKafka_0_10 source code,
>> those warning and error message is logged because NiFi received
>> KafkaException when it tried to commit offset to Kafka.
>>
>> Were there anything in Kafka server logs? I suspect something had
>> happened at Kafka server side.
>>
>> Thanks,
>> Koji
>>
>> On Thu, Feb 9, 2017 at 11:54 AM, Nick Carenza
>>  wrote:
>>> Hey team, I have a ConsumeKafka_0_10 running which normally operates without
>>> problems. I had a queue back up due to a downstream processor and I started
>>> getting these bulletins.
>>>
>>> 01:16:01 UTC WARNING a46d13dd-3231-1bff-1a99-1eaf5f37e1d2
>>> ConsumeKafka_0_10[id=a46d13dd-3231-1bff-1a99-1eaf5f37e1d2] Duplicates are
>>> likely as we were able to commit the process session but received an
>>> exception from Kafka while committing offsets.
>>>
>>> 01:16:01 UTC ERROR a46d13dd-3231-1bff-1a99-1eaf5f37e1d2
>>> ConsumeKafka_0_10[id=a46d13dd-3231-1bff-1a99-1eaf5f37e1d2] Exception while
>>> interacting with Kafka so will close the lease
>>> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@87d2ac1
>>> due to org.apache.kafka.clients.consumer.CommitFailedException: Commit
>>> cannot be completed since the group has already rebalanced and assigned the
>>> partitions to another member. This means that the time between subsequent
>>> calls to poll() was longer than the configured session.timeout.ms, which
>>> typically implies that the poll loop is spending too much time message
>>> processing. You can address this either by increasing the session timeout or
>>> by reducing the maximum size of batches returned in poll() with
>>> max.poll.records.
>>>
>>> My max.poll.records is set to 1 on my consumer and session.timeout.ms is
>>> the default 1 on the server.
>>>
>>> Since there is no such thing as coincidences, I believe this has to do with
>>> it not being able to push received messages to the downstream queue.
>>>
>>> If my flow is backed up, I expect the ConsumKafka processor not to throw
>>> errors but continue to heartbeat with the Kafka server and resume consuming
>>> once it can commit to the downstream queue?
>>>
>>> Might I have the server or consumer misconfigured to handle this scenario or
>>> should the consumer not be throwing this error?
>>>
>>> Thanks,
>>> - Nick

Re: ConsumeKafka processor erroring when held up by full queue

[Joe Witt]

That said I think we can improve our handling of the consumer (kafka
client) and session (nifi transactional logic) and solve the problem.
It is related to our backpressure/consumer handling so we can fix
that.

Thanks
Joe

On Thu, Feb 9, 2017 at 1:38 PM, Bryan Bende  wrote:
> No data loss, but you may process the same message twice in NiFi.
>
> The ordering of operations is:
>
> 1) poll Kafka
> 2) write received data to flow file
> 3) commit NiFi session so data in flow file cannot be lost
> 4) commit offsets to Kafka
>
> Doing it this way achieves at-least once processing which means you
> can't ever lose data, but you can process data twice.
>
> If we committed the offsets before committing the flow file you would
> never get duplicates, but you could lose a message if a crash happened
> between commit the offset and committing the NiFi session (at-most
> once processing).
>
> So the error is happening on #4 and NiFi has already produced a flow
> file with the message, but then Kafka says it can't update the offset,
> and then another consumer will likely pull that same message again and
> produce another flow file with the same message.
>
>
> On Thu, Feb 9, 2017 at 1:19 PM, Nick Carenza
>  wrote:
>> That makes perfect sense. To be clear, is there any potential to lose
>> messages in this scenario?
>>
>> On Thu, Feb 9, 2017 at 7:16 AM, Joe Witt  wrote:
>>>
>>> yeah this is probably a good case/cause for use of the pause concept
>>> in kafka consumers.
>>>
>>> On Thu, Feb 9, 2017 at 9:49 AM, Bryan Bende  wrote:
>>> > I believe you are running into this issue:
>>> >
>>> > https://issues.apache.org/jira/browse/NIFI-3189
>>> >
>>> > When back-pressure happens on the queue coming out of ConsumeKafka,
>>> > this can last for longer than session.timeout.ms, and when the
>>> > processors resumes executing it receives this error on the first
>>> > execution. We should be able to implement some type of keep-alive so
>>> > that even when the processor is not executing, there is a background
>>> > thread, or some way of keeping the connections alive.
>>> >
>>> > I believe any user-defined properties in the processor get passed to
>>> > the Kafka consumer, so I believe you could add "session.timeout.ms"
>>> > and set a much higher value as a possible work around.
>>> >
>>> > Thanks,
>>> >
>>> > Bryan
>>> >
>>> > On Thu, Feb 9, 2017 at 8:42 AM, Koji Kawamura 
>>> > wrote:
>>> >> Hello Nick,
>>> >>
>>> >> First, I assume "had a queue back up" means have a queue being
>>> >> back-pressure. Sorry if that was different meaning.
>>> >>
>>> >> I was trying to reproduce by following flow:
>>> >> ConsumeKafka_0_10
>>> >>   -- success: Back Pressure Object Threshold = 10
>>> >> -- UpdateAttribute (Stopped)
>>> >>
>>> >> Then I used ./bin/kafka-console-producer.sh to send 11 messages.
>>> >> The result was, when NiFi received 10th messages, the success
>>> >> relationship back-pressure was enabled.
>>> >> When I published the 11th message, NiFi didn't do anything.
>>> >> This is expected behavior because downstream connection is
>>> >> back-pressured, the processor won't be scheduled.
>>> >>
>>> >> After I started UpdateAttribute and the queued flow files went
>>> >> through, ConsumeKafka was executed again and received the 11th
>>> >> message.
>>> >>
>>> >> Also, I checked the ConsumerLease and ConsumeKafka_0_10 source code,
>>> >> those warning and error message is logged because NiFi received
>>> >> KafkaException when it tried to commit offset to Kafka.
>>> >>
>>> >> Were there anything in Kafka server logs? I suspect something had
>>> >> happened at Kafka server side.
>>> >>
>>> >> Thanks,
>>> >> Koji
>>> >>
>>> >> On Thu, Feb 9, 2017 at 11:54 AM, Nick Carenza
>>> >>  wrote:
>>> >>> Hey team, I have a ConsumeKafka_0_10 running which normally operates
>>> >>> without
>>> >>> problems. I had a queue back up due to a downstream processor and I
>>> >>> started
>>> >>> getting these bulletins.
>>> >&

Re: Is this possible to increase speed of processors in NiFi?

[Joe Witt]

For processors that support it consider using a 'run duration of 25
milliseconds'.  This allows the framework to batch many operations
into fewer transactions which can dramatically increase throughput.

100% memory usage is a problem.  Increase heap size or consider
redesign/simplifying your flow.

100% disk usage should be addressed.  Ensure each repository is on a
physically separate partition.  If this is not possible then keep in
mind they will create congestion for eachother or consider switching
to a volatile provenance repository.

Review the flow for processors which load content fully into memory.
This can harm performance greatly.

On Fri, Feb 10, 2017 at 11:42 PM, prabhu Mahendran
 wrote:
>
> Since each processor have concurrent tasks if i have set the concurrent
> tasks for processors then it boosts processing speed of processors.But it
> affects the System performance such as 100% Disk Usage,100% Memory
> Usage..etc
>
> Is there is any other way to speed up processors without use concurrent
> tasks?

NiFi Users: Powered by NiFi page

[Joe Witt]

NiFi Users

I just realized we have a 'powered by nifi' page.  It looks a
little...light :-).  So wanted to reach out and offer to anyone
interested that if you reply back on this thread with your
company/organization that you'd like referenced on there I'd be happy
to put in the change to the site for you.

We are aware of a very large user base and obviously can see quite a
bit of this on the Internet but I don't think we can/should put that
unless folks volunteer to have this on the page.  So please let us
know if you're interested in your company/organizational use of NiFi
being mentioned there.

Thanks
Joe

Re: NiFi Users: Powered by NiFi page

[Joe Witt]

Hammad

Excellent.

Can you describe the 'industry' Flexilogix is a part of and a
'description' of its relationship/use for Apache NiFi?

https://nifi.apache.org/powered-by-nifi.html

Thanks
Joe

On Tue, Feb 14, 2017 at 2:54 PM, Hammad  wrote:
> Hi Joe,
>
> That would be nice. Please mention  'FLEXILOGIX' [http://www.flexilogix.com]
>
> Thanks,
> Hammad
>
> On Tue, Feb 14, 2017 at 8:07 PM, Joe Witt  wrote:
>>
>> NiFi Users
>>
>> I just realized we have a 'powered by nifi' page.  It looks a
>> little...light :-).  So wanted to reach out and offer to anyone
>> interested that if you reply back on this thread with your
>> company/organization that you'd like referenced on there I'd be happy
>> to put in the change to the site for you.
>>
>> We are aware of a very large user base and obviously can see quite a
>> bit of this on the Internet but I don't think we can/should put that
>> unless folks volunteer to have this on the page.  So please let us
>> know if you're interested in your company/organizational use of NiFi
>> being mentioned there.
>>
>> Thanks
>> Joe
>
>

Re: NiFi Users: Powered by NiFi page

[Joe Witt]

Could you list 'company/organization name', 'industry', and a
description of how NiFi is used please.

Thanks
Joe

On Tue, Feb 14, 2017 at 3:12 PM, Corey Flowers  wrote:
> Hey Joe,
>
>   You can put us on their. Do you need a write up or anything about how 
> we are using it?
>
> Corey
>
> Sent from my iPhone
>
>> On Feb 14, 2017, at 2:07 PM, Joe Witt  wrote:
>>
>> NiFi Users
>>
>> I just realized we have a 'powered by nifi' page.  It looks a
>> little...light :-).  So wanted to reach out and offer to anyone
>> interested that if you reply back on this thread with your
>> company/organization that you'd like referenced on there I'd be happy
>> to put in the change to the site for you.
>>
>> We are aware of a very large user base and obviously can see quite a
>> bit of this on the Internet but I don't think we can/should put that
>> unless folks volunteer to have this on the page.  So please let us
>> know if you're interested in your company/organizational use of NiFi
>> being mentioned there.
>>
>> Thanks
>> Joe

Re: NiFi Users: Powered by NiFi page

[Joe Witt]

Corey, Hammad, Jeremy

I've added your information to the powered-by page.  Great info - thanks!

All, keep em coming.  This is really helpful for other users.

Joe

On Tue, Feb 14, 2017 at 7:36 PM, Corey Flowers  wrote:
> Name: Onyx Point
>
> Web: www.onyxpoint.com
>
> Industry: Commercial/Federal large scale data distribution system support
> and design
>
> Description: design large scale NIFI clusters for high volume ingest/egress
> and provide day to day operational support and maintenance.
>
> Sent from my iPhone
>
> On Feb 14, 2017, at 5:48 PM, Andy LoPresto  wrote:
>
> Thank you Jeremy and Hammad.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 14, 2017, at 2:37 PM, Hammad  wrote:
>
> Hi Joe,
> Here you go.
>
> Name: Flexilogix
> Industry: BigData/IoT
> Description: Builds all data ingestion pipelines using NiFi. Have deployed
> NiFi clusters to ingest, transform and deliver to data analytics backends
> serving all purposes of data mediation both for realtime and batch jobs.
>
> Thanks
> Hammad
>
> On Tue, Feb 14, 2017 at 8:57 PM, Joe Witt  wrote:
>>
>> Hammad
>>
>> Excellent.
>>
>> Can you describe the 'industry' Flexilogix is a part of and a
>> 'description' of its relationship/use for Apache NiFi?
>>
>> https://nifi.apache.org/powered-by-nifi.html
>>
>> Thanks
>> Joe
>>
>> On Tue, Feb 14, 2017 at 2:54 PM, Hammad  wrote:
>> > Hi Joe,
>> >
>> > That would be nice. Please mention  'FLEXILOGIX'
>> > [http://www.flexilogix.com]
>> >
>> > Thanks,
>> > Hammad
>> >
>> > On Tue, Feb 14, 2017 at 8:07 PM, Joe Witt  wrote:
>> >>
>> >> NiFi Users
>> >>
>> >> I just realized we have a 'powered by nifi' page.  It looks a
>> >> little...light :-).  So wanted to reach out and offer to anyone
>> >> interested that if you reply back on this thread with your
>> >> company/organization that you'd like referenced on there I'd be happy
>> >> to put in the change to the site for you.
>> >>
>> >> We are aware of a very large user base and obviously can see quite a
>> >> bit of this on the Internet but I don't think we can/should put that
>> >> unless folks volunteer to have this on the page.  So please let us
>> >> know if you're interested in your company/organizational use of NiFi
>> >> being mentioned there.
>> >>
>> >> Thanks
>> >> Joe
>> >
>> >
>
>
>

Re: MergeContent across a NiFi cluster

[Joe Witt]

Brian,

Great use case and you're right we don't have an easy way of handling this
now.  If you do indeed have a load balancer in front of the receiving nifi
cluster and it can support affinity of some kind then it is possible you
can set a header in HTTP Post I believe which would come from a flowfile
attribute which would be on each split and would be the hash of its full
object.  If the load balancer ensured all splits (based on that header
matching) were on the same machine then you'd be in business.  There are
some load balancers that do this (i'm thinking of a commercial one).  But,
I admit that is a lot of moving parts to keep in mind.  We need to improve
our site-to-site feature to do things like automatically split content for
you and handle the partitioning/affinity logic I suggested.  You might also
consider avoiding the splitting for now to keep things super simple though
I recognize that exposes alternative tradeoffs.

Great case for us to work on/rally around though.

Thanks
Joe

On Wed, Feb 15, 2017 at 4:29 PM, Kiran 
wrote:

> Hello,
>
> I need to send data from one organisation to another but there are data
> size limits between them (this isn't my choice and has been enforced on
> me). I've got a 4 node NiFi cluster in each organisation.
>
> The sending NiFi cluster has the following data flow:
> Ingest the data by various means
>-> Compress Data using CompressContent
>  -> If file size > X amount I use SplitContent
>-> HTTPS POST to load balancer sitting in front of the NiFi
> cluster in the other organisation
>
> On the receiving NiFi cluster I wanted to:
> -> Receive the data
>-> MergeContent
>  -> Do what ever else with the data...
>
> The problem I can't get round is that if I split the content into 3
> fragments and send them to the receiving NiFi instance because it's
> behind a load balancer I can't guarantee that the 3 fragments are
> received by the same node.
>
> Q1) I'm assuming that for MergeContent to work all the fragments of a
> single piece of data have to arrive on the same NiFi node or is there a
> option to have it working across a cluster?
>
> Q2) How long does the MergeContent processor wait for all the fragments?
> If one of the fragments gets lost does it timeout after a certain
> period?
>
> I was thinking one way to solve this of to have the HTTPListener on the
> receiving NiFi only listening on the primary node which would ensure all
> the fragments arrive on the same node. The downside would be that I end
> up with idle NiFi nodes.
>
> Is there anything obvious that I'm missed that would solve my issue?
>
> Thanks in advance,
>
> Brian
>
>
> 
>  Virus-free.
> www.avast.com
> 
>

Re: Outputting flowfiles to disk

[Joe Witt]

Hello Brian

A good way to do this pattern is to use 'MergeContent' and set the Merge
Format to Flow File Stream, v3.  This way the errors are bundled together
nicely/efficiently.  Ensure it gets a unique filename whenever it is dumped
on disk too.

When you read that bundle/file back off disk you put it through
UnpackContent with flowfile stream v3.

There are a lot of strategies for this sort of dead-letter queue behavior
so if you want to talk through it more feel free to do so.  But for what
you asked the above will do it.

Thanks
Joe

On Wed, Feb 15, 2017 at 5:18 PM, Kiran 
wrote:

> Hello,
>
> Within my NiFi flows for the error scenarios I would really like the
> option of outputting the flow file to an error directory (the outputted
> file contains the flow file contents and well as the attributes).
>
> This way once the error has been resolved I can replay the FlowFile by
> reading it back in which would read the contents as well as the flow file
> attributes.
>
> From looking through the processor list the only way I can see to do this
> is by:
> 1. Using the AttributesToJson to output the attributes and separately
> output the flowfile contents. Then read both the contents and JSON file
> back in and parse the JSON back into attributes.
> 2. Use a groovy script within the ExecuteScript processor to combine the
> attributes and contents together. Then output the results to disk. When
> reading it back in use another groovy script to parse the file and populate
> the attributes and contents.
>
> My preferred option is number 2.
>
> Can I confirm that I haven't missed anything obvious.
>
> Thanks,
>
> Brian
>
>
>
>
> 
>  Virus-free.
> www.avast.com
> 
>

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

Mike,

Can you capture a series of thread dumps as the gradual decay occurs
and signal at what point they were generated specifically calling out
the "now the system is doing nothing" point.  Can you check for space
available on the system during these times as well.  Also, please
advise on the behavior of the heap/garbage collection.  Often (not
always) a gradual decay in performance can suggest an issue with GC as
you know.  Can you run something like

jstat -gcutil -h5  1000

And capture those rules in these chunks as well.

This would give us a pretty good picture of the health of the system/
and JVM around these times.  It is probably too much for the mailing
list for the info so feel free to create a JIRA for this and put
attachments there or link to gists in github/etc.

Pretty confident we can get to the bottom of what you're seeing quickly.

Thanks
Joe

On Thu, Feb 16, 2017 at 9:43 PM, Mikhail Sosonkin  wrote:
> Hello,
>
> Recently, we've upgraded from 0.6.1 to 1.1.1 and at first everything was
> working well. However, a few hours later none of the processors were showing
> any activity. Then, I tried restarting nifi which caused some flowfiles to
> get corrupted evidenced by exceptions thrown in the nifi-app.log, however
> the processors still continue to produce no activity. Next, I stop the
> service and delete all state (content_repository database_repository
> flowfile_repository provenance_repository work). Then the processors start
> working for a few hours (maybe a day) until the deadlock occurs again.
>
> So, this cycle continues where I have to periodically reset the service and
> delete the state to get things moving. Obviously, that's not great. I'll
> note that the flow.xml file has been changed, as I added/removed processors,
> by the new version of nifi but 95% of the flow configuration is the same as
> before the upgrade. So, I'm wondering if there is a configuration setting
> that causes these deadlocks.
>
> What I've been able to observe is that the deadlock is "gradual" in that my
> flow usually takes about 4-5 threads to execute. The deadlock causes the
> worker threads to max out at the limit and I'm not even able to stop any
> processors or list queues. I also, have not seen this behavior in a fresh
> install of Nifi where the flow.xml would start out empty.
>
> Can you give me some advise on what to do about this? Would the problem be
> resolved if I manually rebuild the flow with the new version of Nifi (not
> looking forward to that)?
>
> Much appreciated.
>
> Mike.
>
> This email may contain material that is confidential for the sole use of the
> intended recipient(s).  Any review, reliance or distribution or disclosure
> by others without express permission is strictly prohibited.  If you are not
> the intended recipient, please contact the sender and delete all copies of
> this message.

Re: To create csv file without double quotes.

[Joe Witt]

Hello

PutFile will only write the bytes that exist for that flowfile.  It
won't manipulate content in any way.  I recommend using the provenance
feature of nifi to look at each stage of the flow and follow the
lineage of an object.  With this you should clearly see where the
quotation issue is being introduced.

Thanks
Joe

On Thu, Feb 16, 2017 at 7:03 AM, Nijandan D  wrote:
> Hi,
>
> I am creating custom processor, in which it uses hive database connection
> pooling service to perform file joining operation using csv file. I stored
> processed values in resultSet and then setted that resultSet value in
> AtomicLong variable named as sample. Then updated the flowfile attributes by
> using putAttribute() method as like below function:
>
> flowfile = session.putAttribute(flowfile, "key-value",
> string.valueOf(sample.get());
>
>
> Then transfered that session to flowfile by using session.transfer(flowfile,
> relationship) method. Using putfile processor, created csv file for
> processed data. But in which, it creates values with double quotes in csv
> file. Please refer the attached screenshot of sample created csv file. When
> i try to process again with created csv file, then it creates more double
> quotes in newly created csv file. I need to create csv file for processed
> resultSet without double quotes. Any help appreciated.
>
> Thanks in advance,
>
> Regards,
> Nijandan

Re: Means to access StandardHttpContextMap for monitoring purposes

[Joe Witt]

Jim,

This is a great question and good point.  For controller services it
would be really valuable to have a monitoring function exposed in some
standard manner via their REST API.  I don't believe we have any
generic support for this yet nor do I think the StandardHttpContentMap
does this.  It would be a great thing to create a JIRA for.

Thanks
Joe

On Thu, Feb 16, 2017 at 6:48 AM, James McMahon  wrote:
> Hello. I have a suite of applications that use Http POSTs to push content to
> NiFi "on [their] demand". My HandleHttpRequest processor works for the most
> part, employing an SSL Context Service and a StandardHttpContextMap.
>
> Yesterday apps were unable to post. We are in development and test, and so I
> had been sending the "success" output from HandleHttpRequest not only to my
> main workflow stream, but also to a temporary holding queue feeding into a
> MonitorActivity processor. In other words I was accumulating all the input I
> received so that I could review it using list queue if required.
>
> Bottom line: I started to get errors because HandleHttpRequest could not
> accept requests. I suspect it was due to reaching the maximum threshold for
> my StandardHttpContextMap. With all those incoming requests queued up in
> that temporary queue, I believe I was not releasing these Http contexts from
> the map.
>
> Is there a means to monitor the capacity usage for StandardHttpContextMap?
> I'd like to devise some means to throw a log alert if and when
> StandardHttpContextMap gets to within 75% of my Maximum Outstanding Requests
> associated with the context map.
>
> Thank you in advance for your thoughts. -Jim

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

Mike,

No need for more info.  Heap/GC looks beautiful.

The thread dump however, shows some problems.  The provenance
repository is locked up.  Numerous threads are sitting here

at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at 
org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)

This means these are processors committing their sessions and updating
provenance but they're waiting on a readlock to provenance.  This lock
cannot be obtained because a provenance maintenance thread is
attempting to purge old events and cannot.

I recall us having addressed this so am looking to see when that was
addressed.  If provenance is not critical for you right now you can
swap out the persistent implementation with the volatile provenance
repository.  In nifi.properties change this line

nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository

to

nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository

The behavior reminds me of this issue which was fixed in 1.x
https://issues.apache.org/jira/browse/NIFI-2395

Need to dig into this more...

Thanks
Joe

On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin  wrote:
> Hi Joe,
>
> Thank you for your quick response. The system is currently in the deadlock
> state with 10 worker threads spinning. So, I'll gather the info you
> requested.
>
> - The available space on the partition is 223G free of 500G (same as was
> available for 0.6.1)
> - java.arg.3=-Xmx4096m in bootstrap.conf
> - thread dump and jstats are here
> https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085
>
> Unfortunately, it's hard to predict when the decay starts and it takes too
> long to have to monitor the system manually. However, if you still need,
> after seeing the attached dumps, the thread dumps while it decays I can set
> up a timer script.
>
> Let me know if you need any more info.
>
> Thanks,
> Mike.
>
>
> On Thu, Feb 16, 2017 at 9:54 PM, Joe Witt  wrote:
>>
>> Mike,
>>
>> Can you capture a series of thread dumps as the gradual decay occurs
>> and signal at what point they were generated specifically calling out
>> the "now the system is doing nothing" point.  Can you check for space
>> available on the system during these times as well.  Also, please
>> advise on the behavior of the heap/garbage collection.  Often (not
>> always) a gradual decay in performance can suggest an issue with GC as
>> you know.  Can you run something like
>>
>> jstat -gcutil -h5  1000
>>
>> And capture those rules in these chunks as well.
>>
>> This would give us a pretty good picture of the health of the system/
>> and JVM around these times.  It is probably too much for the mailing
>> list for the info so feel free to create a JIRA for this and put
>> attachments there or link to gists in github/etc.
>>
>> Pretty confident we can get to the bottom of what you're seeing quickly.
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 16, 2017 at 9:43 PM, Mikhail Sosonkin 
>> wrote:
>> > Hello,
>> >
>> > Recently, we've upgraded from 0.6.1 to 1.1.1 and at first everything was
>> > working well. However, a few hours later none of the processors were
>> > showing
>> > any activity. Then, I tried restarting nifi which caused some flowfiles
>> > to
>> > get corrupted evidenced by exceptions thrown in the nifi-app.log,
>> > however
>> > the processors still continue to produce no activity. Next, I stop the
>> > service and delete all state (content_repository database_repository
>> > flowfile_repository provenance_repository work). Then the processors
>> > start
>> > working for a few hours (maybe a day) until the deadlock occurs again.
>> >
>> > So, this cycle continues where I have to periodically reset the service
>> > and
>> > delete the state to get things moving. Obviously, that's not great. I'll
>> > note that the flow.xml file has been changed, as I added/removed
>> > processors,
>> > by the new version of nifi but 95% of the flow configuration is the same
>> > as
>> > before the upgrade. So, I'm wondering if there is a configuration
>> > setting
>> > that causes these deadlocks.
>> >
>> > What I've been able to observe is that the deadlock is "gradual" in that
>> > my
>> > flow usually takes about 4-5 threads to execute. The deadlock causes the
>> > worker threads to

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

Mike

One more thing...can you please grab a couple more thread dumps for us
with 5 to 10 mins between?

I don't see a deadlock but do suspect either just crazy slow IO going
on or a possible livelock.  The thread dump will help narrow that down
a bit.

Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
system too please.

Thanks
Joe

On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt  wrote:
> Mike,
>
> No need for more info.  Heap/GC looks beautiful.
>
> The thread dump however, shows some problems.  The provenance
> repository is locked up.  Numerous threads are sitting here
>
> at 
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>
> This means these are processors committing their sessions and updating
> provenance but they're waiting on a readlock to provenance.  This lock
> cannot be obtained because a provenance maintenance thread is
> attempting to purge old events and cannot.
>
> I recall us having addressed this so am looking to see when that was
> addressed.  If provenance is not critical for you right now you can
> swap out the persistent implementation with the volatile provenance
> repository.  In nifi.properties change this line
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>
> to
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
>
> The behavior reminds me of this issue which was fixed in 1.x
> https://issues.apache.org/jira/browse/NIFI-2395
>
> Need to dig into this more...
>
> Thanks
> Joe
>
> On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin  wrote:
>> Hi Joe,
>>
>> Thank you for your quick response. The system is currently in the deadlock
>> state with 10 worker threads spinning. So, I'll gather the info you
>> requested.
>>
>> - The available space on the partition is 223G free of 500G (same as was
>> available for 0.6.1)
>> - java.arg.3=-Xmx4096m in bootstrap.conf
>> - thread dump and jstats are here
>> https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085
>>
>> Unfortunately, it's hard to predict when the decay starts and it takes too
>> long to have to monitor the system manually. However, if you still need,
>> after seeing the attached dumps, the thread dumps while it decays I can set
>> up a timer script.
>>
>> Let me know if you need any more info.
>>
>> Thanks,
>> Mike.
>>
>>
>> On Thu, Feb 16, 2017 at 9:54 PM, Joe Witt  wrote:
>>>
>>> Mike,
>>>
>>> Can you capture a series of thread dumps as the gradual decay occurs
>>> and signal at what point they were generated specifically calling out
>>> the "now the system is doing nothing" point.  Can you check for space
>>> available on the system during these times as well.  Also, please
>>> advise on the behavior of the heap/garbage collection.  Often (not
>>> always) a gradual decay in performance can suggest an issue with GC as
>>> you know.  Can you run something like
>>>
>>> jstat -gcutil -h5  1000
>>>
>>> And capture those rules in these chunks as well.
>>>
>>> This would give us a pretty good picture of the health of the system/
>>> and JVM around these times.  It is probably too much for the mailing
>>> list for the info so feel free to create a JIRA for this and put
>>> attachments there or link to gists in github/etc.
>>>
>>> Pretty confident we can get to the bottom of what you're seeing quickly.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Feb 16, 2017 at 9:43 PM, Mikhail Sosonkin 
>>> wrote:
>>> > Hello,
>>> >
>>> > Recently, we've upgraded from 0.6.1 to 1.1.1 and at first everything was
>>> > working well. However, a few hours later none of the processors were
>>> > showing
>>> > any activity. Then, I tried restarting nifi which caused some flowfiles
>>> > to
>>> > get corrupted evidenced by exceptions thrown in the nifi-app.log,
>>> > however
>>> > the processors still continue to produce no activity. Next, I stop the
>>> > service and delete all state (content_repository database_repository
>>> > flowfile_repository provenance_repository work). Then the processors
>>> > start
>>> > working for a few hours (maybe a day) until

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

when I said one more thing i definitely lied.

Can you see anything in the UI indicating provenance backpressure is
being applied and if you look in the app log is there anything
interesting that isn't too sensitive to share?

On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt  wrote:
> Mike
>
> One more thing...can you please grab a couple more thread dumps for us
> with 5 to 10 mins between?
>
> I don't see a deadlock but do suspect either just crazy slow IO going
> on or a possible livelock.  The thread dump will help narrow that down
> a bit.
>
> Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
> system too please.
>
> Thanks
> Joe
>
> On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt  wrote:
>> Mike,
>>
>> No need for more info.  Heap/GC looks beautiful.
>>
>> The thread dump however, shows some problems.  The provenance
>> repository is locked up.  Numerous threads are sitting here
>>
>> at 
>> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>> at 
>> org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>>
>> This means these are processors committing their sessions and updating
>> provenance but they're waiting on a readlock to provenance.  This lock
>> cannot be obtained because a provenance maintenance thread is
>> attempting to purge old events and cannot.
>>
>> I recall us having addressed this so am looking to see when that was
>> addressed.  If provenance is not critical for you right now you can
>> swap out the persistent implementation with the volatile provenance
>> repository.  In nifi.properties change this line
>>
>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>
>> to
>>
>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
>>
>> The behavior reminds me of this issue which was fixed in 1.x
>> https://issues.apache.org/jira/browse/NIFI-2395
>>
>> Need to dig into this more...
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin  
>> wrote:
>>> Hi Joe,
>>>
>>> Thank you for your quick response. The system is currently in the deadlock
>>> state with 10 worker threads spinning. So, I'll gather the info you
>>> requested.
>>>
>>> - The available space on the partition is 223G free of 500G (same as was
>>> available for 0.6.1)
>>> - java.arg.3=-Xmx4096m in bootstrap.conf
>>> - thread dump and jstats are here
>>> https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085
>>>
>>> Unfortunately, it's hard to predict when the decay starts and it takes too
>>> long to have to monitor the system manually. However, if you still need,
>>> after seeing the attached dumps, the thread dumps while it decays I can set
>>> up a timer script.
>>>
>>> Let me know if you need any more info.
>>>
>>> Thanks,
>>> Mike.
>>>
>>>
>>> On Thu, Feb 16, 2017 at 9:54 PM, Joe Witt  wrote:
>>>>
>>>> Mike,
>>>>
>>>> Can you capture a series of thread dumps as the gradual decay occurs
>>>> and signal at what point they were generated specifically calling out
>>>> the "now the system is doing nothing" point.  Can you check for space
>>>> available on the system during these times as well.  Also, please
>>>> advise on the behavior of the heap/garbage collection.  Often (not
>>>> always) a gradual decay in performance can suggest an issue with GC as
>>>> you know.  Can you run something like
>>>>
>>>> jstat -gcutil -h5  1000
>>>>
>>>> And capture those rules in these chunks as well.
>>>>
>>>> This would give us a pretty good picture of the health of the system/
>>>> and JVM around these times.  It is probably too much for the mailing
>>>> list for the info so feel free to create a JIRA for this and put
>>>> attachments there or link to gists in github/etc.
>>>>
>>>> Pretty confident we can get to the bottom of what you're seeing quickly.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Thu, Feb 16, 2017 at 9:43 PM, Mikhail Sosonkin 
>>>> wrote:
>>>> > Hello,
>>>> >
>>>> > Recently, we've upgrade

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

Mike

Totally get it.  If you are able to on this or another system get back
into that state we're highly interested to learn more.  In looking at
the code relevant to your stack trace I'm not quite seeing the trail
just yet.  The problem is definitely with the persistent prov.
Getting the phased thread dumps will help tell more of the story.

Also, can you tell us anything about the volume/mount that the nifi
install and specific provenance is on?  Any interesting mount options
involving timestamps, etc..?

No rush of course and glad you're back in business.  But, you've
definitely got our attention :-)

Thanks
Joe

On Fri, Feb 17, 2017 at 12:10 AM, Mikhail Sosonkin  wrote:
> Joe,
>
> Many thanks for the pointer on the Volatile provenance. It is, indeed, more
> critical for us that the data moves. Before receiving this message, I
> changed the config and restarted. The data started moving which is awesome!
>
> I'm happy to help you debug this issue. Do you need these collections with
> the volatile setting or persistent setting in locked state?
>
> Mike.
>
> On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt  wrote:
>>
>> Mike
>>
>> One more thing...can you please grab a couple more thread dumps for us
>> with 5 to 10 mins between?
>>
>> I don't see a deadlock but do suspect either just crazy slow IO going
>> on or a possible livelock.  The thread dump will help narrow that down
>> a bit.
>>
>> Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
>> system too please.
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt  wrote:
>> > Mike,
>> >
>> > No need for more info.  Heap/GC looks beautiful.
>> >
>> > The thread dump however, shows some problems.  The provenance
>> > repository is locked up.  Numerous threads are sitting here
>> >
>> > at
>> > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>> > at
>> > org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>> >
>> > This means these are processors committing their sessions and updating
>> > provenance but they're waiting on a readlock to provenance.  This lock
>> > cannot be obtained because a provenance maintenance thread is
>> > attempting to purge old events and cannot.
>> >
>> > I recall us having addressed this so am looking to see when that was
>> > addressed.  If provenance is not critical for you right now you can
>> > swap out the persistent implementation with the volatile provenance
>> > repository.  In nifi.properties change this line
>> >
>> >
>> > nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>> >
>> > to
>> >
>> >
>> > nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
>> >
>> > The behavior reminds me of this issue which was fixed in 1.x
>> > https://issues.apache.org/jira/browse/NIFI-2395
>> >
>> > Need to dig into this more...
>> >
>> > Thanks
>> > Joe
>> >
>> > On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin 
>> > wrote:
>> >> Hi Joe,
>> >>
>> >> Thank you for your quick response. The system is currently in the
>> >> deadlock
>> >> state with 10 worker threads spinning. So, I'll gather the info you
>> >> requested.
>> >>
>> >> - The available space on the partition is 223G free of 500G (same as
>> >> was
>> >> available for 0.6.1)
>> >> - java.arg.3=-Xmx4096m in bootstrap.conf
>> >> - thread dump and jstats are here
>> >> https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085
>> >>
>> >> Unfortunately, it's hard to predict when the decay starts and it takes
>> >> too
>> >> long to have to monitor the system manually. However, if you still
>> >> need,
>> >> after seeing the attached dumps, the thread dumps while it decays I can
>> >> set
>> >> up a timer script.
>> >>
>> >> Let me know if you need any more info.
>> >>
>> >> Thanks,
>> >> Mike.
>> >>
>> >>
>> >> On Thu, Feb 16, 2017 at 9:54 PM, Joe Witt  wrote:
>> >>>
>> >>> Mike,
>> >>>
>> >>>

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

k - very happy to mention there is a PR out that is under review which
will offer an alternative provenance implementation.  Sustained high
rate testing has shown out of the box 2.5x improvement with immediate
indexing/results so all kinds of fun there.

On Fri, Feb 17, 2017 at 12:13 AM, Mikhail Sosonkin  wrote:
> Let me look through the log. I've not seen anything too weird there before,
> but I'll check again. In the UI, I quite normally see flows getting slowed
> because provenance can't keep up. But it hasn't been too slow for us, so I
> didn't pay much attention.
>
> On Fri, Feb 17, 2017 at 12:03 AM, Joe Witt  wrote:
>>
>> when I said one more thing i definitely lied.
>>
>> Can you see anything in the UI indicating provenance backpressure is
>> being applied and if you look in the app log is there anything
>> interesting that isn't too sensitive to share?
>>
>> On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt  wrote:
>> > Mike
>> >
>> > One more thing...can you please grab a couple more thread dumps for us
>> > with 5 to 10 mins between?
>> >
>> > I don't see a deadlock but do suspect either just crazy slow IO going
>> > on or a possible livelock.  The thread dump will help narrow that down
>> > a bit.
>> >
>> > Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
>> > system too please.
>> >
>> > Thanks
>> > Joe
>> >
>> > On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt  wrote:
>> >> Mike,
>> >>
>> >> No need for more info.  Heap/GC looks beautiful.
>> >>
>> >> The thread dump however, shows some problems.  The provenance
>> >> repository is locked up.  Numerous threads are sitting here
>> >>
>> >> at
>> >> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>> >> at
>> >> org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>> >>
>> >> This means these are processors committing their sessions and updating
>> >> provenance but they're waiting on a readlock to provenance.  This lock
>> >> cannot be obtained because a provenance maintenance thread is
>> >> attempting to purge old events and cannot.
>> >>
>> >> I recall us having addressed this so am looking to see when that was
>> >> addressed.  If provenance is not critical for you right now you can
>> >> swap out the persistent implementation with the volatile provenance
>> >> repository.  In nifi.properties change this line
>> >>
>> >>
>> >> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>> >>
>> >> to
>> >>
>> >>
>> >> nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
>> >>
>> >> The behavior reminds me of this issue which was fixed in 1.x
>> >> https://issues.apache.org/jira/browse/NIFI-2395
>> >>
>> >> Need to dig into this more...
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin 
>> >> wrote:
>> >>> Hi Joe,
>> >>>
>> >>> Thank you for your quick response. The system is currently in the
>> >>> deadlock
>> >>> state with 10 worker threads spinning. So, I'll gather the info you
>> >>> requested.
>> >>>
>> >>> - The available space on the partition is 223G free of 500G (same as
>> >>> was
>> >>> available for 0.6.1)
>> >>> - java.arg.3=-Xmx4096m in bootstrap.conf
>> >>> - thread dump and jstats are here
>> >>> https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085
>> >>>
>> >>> Unfortunately, it's hard to predict when the decay starts and it takes
>> >>> too
>> >>> long to have to monitor the system manually. However, if you still
>> >>> need,
>> >>> after seeing the attached dumps, the thread dumps while it decays I
>> >>> can set
>> >>> up a timer script.
>> >>>
>> >>> Let me know if you need any more info.
>> >>>
>> >>> Thanks,
>> >>> Mike.
>> >>>
>>

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Joe Witt]

Cool thanks Mike.  Mount question/concern resolved.

On Fri, Feb 17, 2017 at 12:33 AM, Mikhail Sosonkin  wrote:
> I'm really happy that you guys responded so well, it's quite lonely googling
> for this stuff :)
>
> Right now the volume is high because nifi is catching up at about 2.5G/5m
> and 500FlowFile/5m, but normally we're at about 100Mb/5m with a few spikes
> here and there, nothing too intense.
>
> We are using an EC2 instance with 32G RAM and 500G SSD. All the work is done
> on the same mount. Not sure what you mean by timestamps in this case. Our
> set up is pretty close to out of the box with only heap size limit change in
> bootstrap and a few Groovy based processors.
>
> I'll try to get you some thread dumps for the decay, might have to wait
> until tomorrow or monday though. I want to see if I can get it to behave
> like this on a test system.
>
> Mike.
>
> On Fri, Feb 17, 2017 at 12:13 AM, Joe Witt  wrote:
>>
>> Mike
>>
>> Totally get it.  If you are able to on this or another system get back
>> into that state we're highly interested to learn more.  In looking at
>> the code relevant to your stack trace I'm not quite seeing the trail
>> just yet.  The problem is definitely with the persistent prov.
>> Getting the phased thread dumps will help tell more of the story.
>>
>> Also, can you tell us anything about the volume/mount that the nifi
>> install and specific provenance is on?  Any interesting mount options
>> involving timestamps, etc..?
>>
>> No rush of course and glad you're back in business.  But, you've
>> definitely got our attention :-)
>>
>> Thanks
>> Joe
>>
>> On Fri, Feb 17, 2017 at 12:10 AM, Mikhail Sosonkin 
>> wrote:
>> > Joe,
>> >
>> > Many thanks for the pointer on the Volatile provenance. It is, indeed,
>> > more
>> > critical for us that the data moves. Before receiving this message, I
>> > changed the config and restarted. The data started moving which is
>> > awesome!
>> >
>> > I'm happy to help you debug this issue. Do you need these collections
>> > with
>> > the volatile setting or persistent setting in locked state?
>> >
>> > Mike.
>> >
>> > On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt  wrote:
>> >>
>> >> Mike
>> >>
>> >> One more thing...can you please grab a couple more thread dumps for us
>> >> with 5 to 10 mins between?
>> >>
>> >> I don't see a deadlock but do suspect either just crazy slow IO going
>> >> on or a possible livelock.  The thread dump will help narrow that down
>> >> a bit.
>> >>
>> >> Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
>> >> system too please.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt  wrote:
>> >> > Mike,
>> >> >
>> >> > No need for more info.  Heap/GC looks beautiful.
>> >> >
>> >> > The thread dump however, shows some problems.  The provenance
>> >> > repository is locked up.  Numerous threads are sitting here
>> >> >
>> >> > at
>> >> >
>> >> > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>> >> > at
>> >> >
>> >> > org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>> >> >
>> >> > This means these are processors committing their sessions and
>> >> > updating
>> >> > provenance but they're waiting on a readlock to provenance.  This
>> >> > lock
>> >> > cannot be obtained because a provenance maintenance thread is
>> >> > attempting to purge old events and cannot.
>> >> >
>> >> > I recall us having addressed this so am looking to see when that was
>> >> > addressed.  If provenance is not critical for you right now you can
>> >> > swap out the persistent implementation with the volatile provenance
>> >> > repository.  In nifi.properties change this line
>> >> >
>> >> >
>> >> >
>> >> > nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>> >> >
>> >> > to
>> >> >
>>