The issue is on a CentOS linux server, running Nifi 1.4.0 snapshot. There it communicates with a dozen of Moxa nports over tcp on various remote sites. All moxa's are connected over a (slow) cellular connection.
Will do a tcpdump with the new / old PutTCP processor. On Wed, 20 Sep 2017 at 22:10, Bryan Bende <[email protected]> wrote: > The sender/output stream can only ever be in use by one thread since > it acquires it from the pool at the beginning of onTrigger, which > makes it unavailable to any other threads until it is placed back in > the pool. > > The flush is there just because now it is writing to an output stream > and we may be leaving the sender/output stream open depending if > "connection per flow file" is true or not. > > In the case where you do produce the issue, where is NiFi running? Is > it sending from NiFi on your Mac to the device on Linux? > > On Wed, Sep 20, 2017 at 3:45 PM, Davy De Waele <[email protected]> wrote: > > Don't get it either... new code looks more or less the same. been trying > > various things all day. Glad I got it working with a "patched" PutTCP. > > Better from what I got but still not ideal. > > > > Can it be a threading issue (we interact with multiple modules) in the > input > > / output stream handling ? Also noticed a flush that wasn't there before. > > Are there other places where charsets can be set ? > > > > Been thinking about what could cause the tcp server to react this way > but no > > idea. Can reproduce it easily. Will check if I can enable additional > logging > > on the moxa. > > > > On Wed, 20 Sep 2017 at 21:36, Bryan Bende <[email protected]> wrote: > >> > >> For what it is worth, I can't reproduce this on my Mac laptop either... > >> > >> Setup GenerateFlowFile with same sample text your provided, followed > >> by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same > >> NiFi instance. > >> > >> If I look at the content of the flow files coming out of ListenTCP, > >> and I put the content viewer in Hex mode, I see: > >> > >> 0x00000000 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND.. > >> > >> The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r which > >> would make sense since ListenTCP read-off the \r\n at the end. > >> > >> > >> On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende <[email protected]> wrote: > >> > I'll keep looking at this, but just to summarize the changes that were > >> > made in case something jumps out to someone else... > >> > > >> > The changes were to avoid reading the entire content of the flow file > >> > into memory. > >> > > >> > Previous Behavior: > >> > - Create a ByteArrayOuputStream > >> > - Use an InputStreamCallback to read entire bytes of flow file, and > >> > write them to above output stream > >> > - Get value of the outgoing delimiter property > >> > - If delimiter is not null, then get the bytes of the delimiter using > >> > the charset configured in the processor, and write the delimiter bytes > >> > to the above output stream > >> > - Get byte array from the ByteArrayOutputStream and pass it to the > >> > channel sender, which in turns calls write on a SocketChannel > >> > > >> > New behavior: > >> > - Get the direct SockeOutputStream from the sender > >> > - Get the InputStream of the flow file > >> > - Call IOUtils.copy to copy the flow file input stream to the socket > >> > output stream in chunks > >> > - Perform exact same delimiter logic as above > >> > > >> > > >> > > >> > > >> > On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele <[email protected]> > >> > wrote: > >> >> Correct about the delimiter. > >> >> Think character set is (was) automatically filled in with utf-8. > (Never > >> >> noticed a default option) > >> >> > >> >> On Wed, 20 Sep 2017 at 19:11, Bryan Bende <[email protected]> wrote: > >> >>> > >> >>> Thanks for the info and for digging into this.. > >> >>> > >> >>> Just to confirm, in your PutTCP configuration, you are saying that > you > >> >>> have "Outgoing Message Delimiter" set to \r\n ? > >> >>> > >> >>> And do you have the "Character Set" property set to the default of > >> >>> "UTF-8" > >> >>> ? > >> >>> > >> >>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele <[email protected]> > wrote: > >> >>> > I've put up a git repo that I use to test the PutTCP behavior. > >> >>> > > >> >>> > > >> >>> > > >> >>> > ( > https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1 > ) > >> >>> > > >> >>> > What I've done. > >> >>> > > >> >>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base > nifi > >> >>> > installation > >> >>> > - Noticed weird characters in the tcp responses > >> >>> > - Switched to 1.3.0. No more weird characters > >> >>> > - Isolated the PutTCP processor by creating a custom PutTCP (see > >> >>> > github > >> >>> > repo > >> >>> > below) to experiment (instead of the built-in PutTCP) > >> >>> > - Repo has 2 branches. master contains an "older" PutTCP processor > >> >>> > from > >> >>> > the > >> >>> > Nifi code base. With that one I don't have the weird characters > >> >>> > (even on > >> >>> > a > >> >>> > nifi 1.4.0-SNAPSHOT + your PR). > >> >>> > - If I use the dev_puttcp_1.4 branch on my nifi I get the weird > >> >>> > characters. > >> >>> > > >> >>> > > >> >>> > Some info on my setup : > >> >>> > > >> >>> > I'm sending commands to PutTCP using a GenerateFlowFile processor. > >> >>> > payload = ${literal('\r')}@COMMAND${literal('\r')} > >> >>> > PutTCP adds a delimier (\r\n) > >> >>> > > >> >>> > Regarding provenance. PutTCP adds the delimiter to the payload > >> >>> > before > >> >>> > sending it to the TCP socket. That delimiter is not in the > flowfile > >> >>> > that is > >> >>> > put in provenance I think. > >> >>> > > >> >>> > Cannot reproduce the issue on my mac with my mock tcp server. But > on > >> >>> > the > >> >>> > actual environment (Linux CentOS) with the Moxa devices, there is > a > >> >>> > clear > >> >>> > difference in behaviour. Will see if I can tunnel my through the > >> >>> > devices > >> >>> > > >> >>> > All the info I have right now .... > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > -- > >> >>> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/ >
