If performance is the problem, then you definitely want to get rid of any SplitText / Split* processors. These processors are great when they are necessary but they should be avoided if at all possible, because splitting the data apart results in huge overhead for NiFi and will harm performance [1] (plus it just makes the flow a lot more complex).
Better options would be to use ReplaceText in order to convert the existing IIS Log message into a Syslog-formatted message or to use UpdateRecord with a CSV Reader and a Syslog Writer, adding in fields to the UpdateRecord processor like /hostname = localhost, /priority = 4, /message = CONCAT(‘IISHttp’, .), and so on. If you’re sending data over TCP, there is no need to split the data up at all. You can just send the entire text, newline delimited, over TCP using PutTCP instead of PutSyslog. If you want to send over UDP, you may end up needing to use a SplitText just before PutUDP, but at least that would offer better performance because you only have a single processor operating on tiny FlowFiles. Thanks -Mark [1] https://www.youtube.com/watch?v=RjWstt7nRVY On Jun 26, 2020, at 3:51 AM, muhyid72 <[email protected]<mailto:[email protected]>> wrote: Hi Andy, Thank you for your great support My aim is transferring all IIS logs to syslog line by line. Therefore i am using split text for parsing line. I tried Route Text yesterday but i didn't accomplish to transfer line by line to syslog. Extract Text is transferring splitted line on the attribute, in this way i can say to syslog processor "Message Body: IISHttp${msg}". Actually my problem is botleneck on the Extract Text. I have to transfer IIS Logs near-real time due to cyber security process. But it doesn't drain number of the message in the queue properly. I tried increasing Thread Number, changing Run Duration, increasing/reducing Queue size but i couldn't achive my target. The queue between split text and extract text allways full and i have log gap about 12 hours. I am trying find a way for that -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
