Re: PutTCP connector not cleaning up dangling connections

Small update : No garbage / noise on 1.3.0 also so it must be in 1.4.0-SNAPSHOT. Noticed that the PutTCP processor has changed the way it processes incoming flowfiles. Might be related to that. Looked into the data provenance and noticed 3 bytes EF BF BD (in hex) when things go bad. (This

Simple check to see whether a file is compressed or not?

Is there a simple way to see whether a file is compressed or not? In my current flow, if an object is not compressed it should bypass the CompressContent processor, otherwise it needs to be decompressed.

Re: minifi-cpp s2s no protocol

Andy, Can you try adding the RPG host and port in the properties for your ports? Remote Processing Groups: - name: NiFi Flow id: 2438e3c8-015a-1000-79ca-83af40ec1998 url: http://127.0.0.1:8080/nifi timeout: 30 secs yield period: 5 sec Input Ports: -

Re: PutTCP connector not cleaning up dangling connections

Thanks for the info and for digging into this.. Just to confirm, in your PutTCP configuration, you are saying that you have "Outgoing Message Delimiter" set to \r\n ? And do you have the "Character Set" property set to the default of "UTF-8" ? On Wed, Sep 20, 2017 at 12:59 PM, ddewaele

Re: minifi-cpp s2s no protocol

> I tested a similar config on the nifi side and no changes were necessary > to nifi.properties in order to use the HTTP s2s protocol. > This is explained (thanks Aldrin) by https://issues.apache.org/jira/browse/MINIFICPP-60. We simply don't support it yet. My test will just cover the socket

Re: PutTCP connector not cleaning up dangling connections

Correct about the delimiter. Think character set is (was) automatically filled in with utf-8. (Never noticed a default option) On Wed, 20 Sep 2017 at 19:11, Bryan Bende wrote: > Thanks for the info and for digging into this.. > > Just to confirm, in your PutTCP configuration,

Re: Simple check to see whether a file is compressed or not?

On Wed, 2017-09-20 at 11:32 -0700, Laurens Vets wrote: > Is there a simple way to see whether a file is compressed or not? > > In my current flow, if an object is not compressed it should bypass the > CompressContent processor, otherwise it needs to be decompressed. > Not sure if there is a

Re: minifi-cpp s2s no protocol

On Wed, 2017-09-20 at 11:37 -0400, Marc wrote: > Andy, >I think you're looking for the Site to Site properties in your > nifi.properties. You likely have an option named > "nifi.remote.input.socket.port" as I do, below. Use the input host > ( probably nifi ) and port you have defined. > > >

Re: minifi-cpp s2s no protocol

?This is on the phrocker/MINIFI-339 branch with some additional testing code I'm working on. If it helps, in the logs, the s2s connection fails several times as my testing nifi instance boots, then once nifi is ready, it gets this config JSON back & spits out the no protocol error. I'm

Re: minifi-cpp s2s no protocol

Andy, I think you're looking for the Site to Site properties in your nifi.properties. You likely have an option named "nifi.remote.input.socket.port" as I do, below. Use the input host ( probably nifi ) and port you have defined. # Site to Site properties nifi.remote.input.host=localhost

Re: PutTCP connector not cleaning up dangling connections

Sounds good, interested to hear about it when you get a chance. On Wed, Sep 20, 2017 at 12:32 PM, Davy De Waele wrote: > Having dinner now without laptop :) Have more updates. Isolated the problem > in the puttcp class in the processor. Rest of the 1.4.0-snapshot seems fine.

Re: InvokeScriptedProcessor to set common flowfile attributes?

Jim, You should be able to do everything you want with InvokeScriptedProcessor and a handy utility class in NiFi called SynchronousFileWatcher [1]. It is used by various NiFi components such as the ScanAttribute [2] processor. You should be able to import it via: from

Re: InvokeScriptedProcessor to set common flowfile attributes?

Sounds very promising. Thank you very much for pointing me in the right direction Matt. I will start digging into this immediately, and if I'm able to get it to work will follow-up later to post my solution. --Jim On Wed, Sep 20, 2017 at 10:04 AM, Matt Burgess wrote: >

Global setting for backpressure

Good morning, Is there a way to globally change back pressure settings? To the point that new processors added to canvas would default to our set numbers? I was hoping for a NIFI properties approach. EX: I don't want to default to: Back Pressure Object: 10,000 Back Pressure Data Size: 1 GB

Re: Global setting for backpressure

Hi Juan, That improvement has been requested: https://issues.apache.org/jira/browse/NIFI-3599 -Drew > On Sep 20, 2017, at 10:46 AM, Juan Sequeiros wrote: > > Good morning, > > Is there a way to globally change back pressure settings? > To the point that new processors

minifi-cpp s2s no protocol

Hi All, I have a minifi-cpp flow using HTTP s2s to send data to a nifi instance. The minifi instance seems to be making an initial connection to nifi, but it fails to send data, and I see a "no protocol" in the logs. Here's the log output & flow yml: [2017-09-20 14:37:35.634]

Re: PutTCP connector not cleaning up dangling connections

Having dinner now without laptop :) Have more updates. Isolated the problem in the puttcp class in the processor. Rest of the 1.4.0-snapshot seems fine. Will post more info after dinner. On Wed, 20 Sep 2017 at 18:27, Bryan Bende wrote: > Is the data good at other points in

Re: PutTCP connector not cleaning up dangling connections

Don't get it either... new code looks more or less the same. been trying various things all day. Glad I got it working with a "patched" PutTCP. Better from what I got but still not ideal. Can it be a threading issue (we interact with multiple modules) in the input / output stream handling ? Also

Re: PutTCP connector not cleaning up dangling connections

The issue is on a CentOS linux server, running Nifi 1.4.0 snapshot. There it communicates with a dozen of Moxa nports over tcp on various remote sites. All moxa's are connected over a (slow) cellular connection. Will do a tcpdump with the new / old PutTCP processor. On Wed, 20 Sep 2017 at 22:10,

Re: PutTCP connector not cleaning up dangling connections

For what it is worth, I can't reproduce this on my Mac laptop either... Setup GenerateFlowFile with same sample text your provided, followed by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same NiFi instance. If I look at the content of the flow files coming out of ListenTCP,

Re: PutTCP connector not cleaning up dangling connections

The sender/output stream can only ever be in use by one thread since it acquires it from the pool at the beginning of onTrigger, which makes it unavailable to any other threads until it is placed back in the pool. The flush is there just because now it is writing to an output stream and we may be

Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files

I can confirm the sate is being managed by zookeeper. I've been doing some more testing today and I haven't been able to make the ListXXX processes fail by manually causing the primary node to change (just using tcpdump to drop zookeeper traffic, causing a node change). So I'm not sure at the

Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files

A single node still needs a state manager to survive restarts of the application, but there is no more distributed state at that point, so the local state manager is backed by a write-ahead-log store in ./state under NiFi’s home directory. - Bryan > On Sep 20, 2017, at 8:06 PM, Gino Lisignoli

Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files

Hi Joe The distributed cache service property of ListXXX doesn't seem to matter anymore. Which was quite a surprise to me. It just gets handled by the state manager:

Re: minifi-cpp s2s no protocol

Andy, What branch are you using? I believe this is was one issue resolved under the guise of https://issues.apache.org/jira/browse/MINIFICPP-4 ? On Wed, Sep 20, 2017 at 11:03 AM, Andy Christianson < achristian...@hortonworks.com> wrote: > Hi All, > > > I have a minifi-cpp flow using HTTP

InvokeScriptedProcessor to set common flowfile attributes?

Our NiFi is 0.7.x. I understand that it is possible to set customer-specific configuration parameters in a second config file similar to nifi.properties. I have this requirement: R1- allow engineering leads to set valid message types in a simple python list, like so: validTypes =