So I understand, is the concern with multiple flow files with different
destinations in the same queue would require the processor close the
connection for each flow file? Could you use the same connection pool
approach similar to fetchSFTP with keeping connections open for a period of
time.
In the variable approach it would require a processor per host:port pair? I
was trying to get around multiple processors and having to create a
spiderweb with distribute load to handle failures by using update attribute
with state and automatically assigning the endpoint based on a counter. We
do something similar with putSFTP postHTTP. It would be great if the
post/put processors could accept a list or array of hosts with some basic
load balance options.
On Fri, Jan 26, 2018 at 12:29 PM, Bryan Bende <bbe...@gmail.com> wrote:
> In the default case, "Connection Per Flow File" is false, which means
> there is one connection created and used across many flow files, which
> will perform best.
>
> Setting "Connection Per Flow File" to true, means it will close the
> connection at the end of every on trigger call.
>
> We could potentially evaluate against flows when "Connection Per Flow
> File" is true b/c then we can create a new one every time, but when it
> is false it means you would have to potentially keep open many
> connections.
>
> It also may be confusing though to have differing evaluation based on
> another property.
>
> On Fri, Jan 26, 2018 at 12:16 PM, Ryan Ward <ryan.wa...@gmail.com> wrote:
> > Thanks, that's a good idea, as I would automatically assume if a property
> > supports EL it would work against flowfiles. I see the ticket 3231 was
> > specific to variable registry and explicitly says not flow files. Any
> > particular reason?
> >
> > On Fri, Jan 26, 2018 at 12:02 PM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> >
> >> And to add a bit of info on the last remark from Joe, I'm working on
> >> NIFI-4149 to make the scope of EL clearer to users. Didn't have time to
> >> work on it lately but will definitely try to get back to it very soon.
> >>
> >> Pierre
> >>
> >> 2018-01-26 17:14 GMT+01:00 Joe Witt <joe.w...@gmail.com>:
> >>
> >> > Stated differently - it considers what it can glean from variables
> >> > made available other than the flowfile itself. With Apache NiFi 1.5.0
> >> > that means process group variables, variables set in nifi.properites,
> >> > and environment variables.
> >> >
> >> > We need to make sure that we call this out when we indicate a given
> >> > property supports expression language so the user knows the scope at
> >> > which it will help them.
> >> >
> >> > On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <marcogaid...@gmail.com
> >
> >> > wrote:
> >> > > Hi Ryan,
> >> > >
> >> > > probably the reason of the behavior is that EL on PutTCP is enabled
> but
> >> > it
> >> > > is not run on the incoming flowfile. So it doesn't care the
> attributes
> >> of
> >> > > your flowfile. It considers only environment variables.
> >> > >
> >> > > Thanks,
> >> > > Marco
> >> > >
> >> > > 2018-01-26 15:56 GMT+01:00 Ryan Ward <ryan.wa...@gmail.com>:
> >> > >
> >> > >> I'm seeing odd behavior trying to use attributes for the hostname
> and
> >> > port
> >> > >> fields.
> >> > >>
> >> > >> using ${endpoint_port} (9003) + hardcoded IP results in flowfile
> >> > yielding
> >> > >> failed to process session due to java.lang.NumberFormatException:
> For
> >> > >> input
> >> > >> string: "": {}
> >> > >> java.lang.NumberFormatException: For input string: ""
> >> > >> at
> >> > >> java.lang.NumberFormatException.forInputString(
> >> > >> NumberFormatException.java:65)
> >> > >> at java.lang.Integer.parseInt(Integer.java:592)
> >> > >> at java.lang.Integer.parseInt(Integer.java:615)
> >> > >> at
> >> > >> org.apache.nifi.attribute.expression.language.
> StandardPropertyValue.
> >> > >> asInteger(StandardPropertyValue.java:78)
> >> > >>
> >> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> > >> (PutTCP.java:111)
> >> > >> ...
> >> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> > >> (PutTCP.java:179)
> >> > >>
> >> > >> using hardcoded 9003 + ${endpoint} results in flowfile failing due
> to
> >> > >> connection refused
> >> > >> DEBUG ...No available connections, creating a new one...
> >> > >> ERROR ...No available connections, and unable to create a new one
> >> ....to
> >> > >> failure: java.net.ConnectException: Connection refused
> >> > >>
> >> > >> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and
> >> > hardcoded
> >> > >> port
> >> > >> DEBUG...Connected to local port 23250
> >> > >> DEBUG....Relinquishing sender
> >> > >> Flow files transferred to success, its unclear where the data went
> or
> >> > why I
> >> > >> needed to have the nodes listening on this port. Is the attribute
> >> value
> >> > >> being ignored and defaulting to localhost? Watching this behavior
> via
> >> > >> netstat I could see 127.0.0.1 was indeed connected to itself on
> 9003.
> >> > Odd
> >> > >> thing is no data came in on the ListenTCP either.
> >> > >>
> >> > >> Thanks,
> >> > >> Ryan
> >> > >>
> >> >
> >>
>
