I was wrong about it not resuming after connection reset. It does. I just overlooked, and as I don't have rights to comment in Nabble, I had to wait for an email to reply on...
Anyway, thanks for clearing out the socket buffer thing! On Wed, Nov 16, 2016 at 3:31 PM, Joe Witt <[email protected]> wrote: > In 1.x back pressure happens by default so if that is what is being used > it is probably why the flow appears stopped. > > In the 1.1 release this will he visually more obvious. > > Thanks > Joe > > On Nov 16, 2016 9:29 AM, "Bryan Bende" <[email protected]> wrote: > >> Hello Raf, >> >> The message about attempting to set the socket buffer size is not really >> an error that would prevent anything from working, it is just a warning so >> that the user knows that NiFI created a new connection and tried to set the >> Socket's receive buffer to some value that was specified in the properties >> (1 MB in your case), and the OS only let it get set to some smaller value, >> which could be important for someone trying to tune various settings. >> >> This happens whenever a new connection is created, which likely happens >> your TCP servers goes down and comes back up. It can also happen during >> normal operation of the processor, PutTCP will create connections on the >> fly as needed and then close them if they have been used in the amount of >> time greater than "Idle Connection Expiration". >> >> I definitely agree it would be nice for that message to not print all the >> time though, one way to get rid of it would be to reduce the value of "Max >> Size of Socket Send Buffer" to meet what the OS is allowing, another way >> would be to configure logback.xml so that org.apache.nifi.processor.util >> .put.sender.SocketChannelSender only logged at the ERROR level, since >> this message is logged at the WARN level, but this means you could miss >> other valuable warnings. >> >> When you say "some time later it becomes worse... no flowfiles are >> generated", are you saying GenerateFlowFile is no longer generating flow >> files? when this happens do you have a lot of flow files in queues, and do >> you have back-pressure configured? >> >> Thanks, >> >> Bryan >> >> >> On Wed, Nov 16, 2016 at 5:59 AM, Raf Huys <[email protected]> wrote: >> >>> I'm having a simple flow GenerateFlowfile -> ReplaceText -> PutTCP which >>> should establish a TCP connection and send a small piece of text over that >>> connection every 60 seconds. This is established bij scheduling the first >>> processor as a cron job. >>> >>> This pipeline works, until I start restarting our TCP server upstream. >>> >>> >>> What can happen is that the TCP server is unavailable (due to >>> reasons...). Nevertheless, when the TCP server becomes available again, the >>> Nifi pipeline should continue doing it's job. >>> >>> However, what I observe is that the PutTCP processor start throwing the >>> following errors after the TCP became unavailable, and then available again: >>> >>> Attempted to set Socket Buffer Size to ... bytes but could only set to >>> ... bytes. You may want to consider changing the Operating System's maximum >>> receive buffer >>> >>> I find this message confusing because the messages we are sending are >>> about 10 characters wide. Also, the time between between stopping/starting >>> the TCP server is a couple of seconds, which means there is no backpressure >>> of unsent flowfiles. >>> >>> Properties of the PutTCP processor >>> >>> >>> - Hostname localhost >>> - Port 4001 >>> - Max Size of Socket Send Buffer 1 MB >>> - Idle Connection Expiration 5 seconds >>> - Connection Per FlowFile true >>> - Outgoing Message Delimiter \r\n >>> - Timeout 10 seconds >>> - SSL Context Service No value set >>> - Character Set UTF-8 >>> >>> I would love some help here. >>> >>> PS: Some time later it becomes worse, as every processor in the above >>> pipeline actually stops doing anything...no flowfiles are generated, no >>> errors are thrown... >>> >>> Thanks, >>> >>> Raf Huys >>> >> >> -- Mvg, Raf Huys
