On Thu, Feb 10, 2022 at 4:00 PM Breno Cesar <brenocrs230...@gmail.com> wrote: > > Michael, Natan and Joe, > > Thank you very much for your replies. > Here some stuffs that i done today: > > Changing the Batching Message Delimiter had no effect, according to the > documentation it only works when a package in TCP is already identified, and > apparently how is done and their delimiter is "hardcoded" in nifi. > Searching a bit I found this project, i tested it here, and put "0" as the > delimiter of the income packages, and apparently it worked, i received some > connection flowfiles, I'll study the packages a little better and see which > delimiter (or sequence of delimiters) works best for my situation.
A fixed delimiter will not work correctly with IDPR. It's a variable length binary protocol, which means it'll need it's own RecordReader to ingest correctly into NiFi....and....honestly likely to do any real work with it anyway... So you either have to build a record reader, or put something in front that'll digest the IDPR messages into something NiFi can work with. > I'm not a developer (I just know how to google well), and making an > IDPRRecordReader from scratch knowing the basics, it might take a few years. > i'm going to open a request in jira to develop the IDPRRecordReader, as well > to check if they could add the functionality to change the delimiter of > income packages in listentcp, and pray that someone cares about =p. > > Again guys, thank you very much for your help. > > Breno > > > > Em qui., 10 de fev. de 2022 às 00:13, Joe Witt <joe.w...@gmail.com> escreveu: >> >> Looking at >> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4 >> >> I think youll want to write an IDPRRecordReader >> >> Thanks >> >> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <thena...@apache.org> wrote: >>> >>> You should be able to set the message delimiter in ListenTCP with the >>> "Message Delimiter" property. The default is \n. I don't believe there's a >>> great way to see how data is being interpreted other than using interactive >>> debug mode with NiFi set through the bootstrap.conf. >>> >>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <brenocrs230...@gmail.com> wrote: >>>> >>>> Hi guys, >>>> >>>> First of all i apologize for my english, and thank you very much for your >>>> answers. >>>> The data type i need is a data stream, which starts when the ipdr protocol >>>> connect process ends. >>>> Is there any way to change the line terminator? or somehow see how nifi >>>> is interpreting the data when It its received ? >>>> >>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <joe.w...@gmail.com> escreveu: >>>>> >>>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data and >>>>> needs some way of framing them to generate records/bundles of data as >>>>> flowfiles. >>>>> >>>>> What kind of data do you want to listen for? >>>>> >>>>> Thanks >>>>> >>>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <mlof...@wgops.com> wrote: >>>>>> >>>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented, >>>>>> default separator is Unix newline “\n”. I don’t think it will generate a >>>>>> flow if it never receives a newline before the connection closes. >>>>>> >>>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <brenocrs230...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> i'm tring to receive a tcp stream using the listentcp module, but when >>>>>>> the connection start, i see the traffic reaching my server (usgin >>>>>>> tcpdump), but the nifi does note generate a flow file. >>>>>>> Ther is no error ow warning in the module or in the nifi cluster. >>>>>>> Changing in the processor the bulleting level to "debug" , i can see >>>>>>> the following message for every package that i receive: >>>>>>> >>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] >>>>>>> Accepted incoming connection from /192.0.2.12:55209 >>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] >>>>>>> bytes read 46 >>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No >>>>>>> more data available, returning for selection >>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] >>>>>>> Reached EOF, closing connection >>>>>>> >>>>>>> bellow is the same message, but in the tcpdump format. >>>>>>> >>>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128 >>>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq: >>>>>>> 1, Ack: 1, Len: 46 >>>>>>> Source Port: 55209 >>>>>>> Destination Port: 4737 >>>>>>> <Source or Destination Port: 55209 > >>>>>>> <Source or Destination Port: 4737> >>>>>>> [Stream index: 0] >>>>>>> [TCP Segment Len: 46] >>>>>>> Sequence number: 1 (relative sequence number) >>>>>>> Sequence number (raw): 3765145522 >>>>>>> [Next sequence number: 47 (relative sequence number)] >>>>>>> Acknowledgment number: 1 (relative ack number) >>>>>>> Acknowledgment number (raw): 274707205 >>>>>>> 0101 .... = Header Length: 20 bytes (5) >>>>>>> Flags: 0x018 (PSH, ACK) >>>>>>> Window size value: 10000 >>>>>>> [Calculated window size: 10000] >>>>>>> [Window size scaling factor: 1] >>>>>>> Checksum: 0x5fd7 [unverified] >>>>>>> [Checksum Status: Unverified] >>>>>>> Urgent pointer: 0 >>>>>>> [SEQ/ACK analysis] >>>>>>> [Timestamps] >>>>>>> TCP payload (46 bytes) >>>>>>> [PDU Size: 46] >>>>>>> IPDR >>>>>>> Version: 2 >>>>>>> Message id: CONNECT (5) >>>>>>> Session id: 0 >>>>>>> Message flags: 0x00 >>>>>>> Message length: 46 >>>>>>> Initiator id: 0.0.0.0 >>>>>>> Initiator port: 0 >>>>>>> Capabilities: 0x00000006 >>>>>>> Keep-alive interval: 90 >>>>>>> Vendor id: ARRIS E6000 Exporter >>>>>>> >>>>>>> Here somethings that i tried: >>>>>>> >>>>>>> - Increase the number of simultaneos connections and batch size, but >>>>>>> it does not change at all. >>>>>>> - Use listentcprecord but the behavior is the same, it only change >>>>>>> when i select the avroreader, then i got an error informing that is a >>>>>>> "wrong data format". >>>>>>> >>>>>>> So i belive that nifi is receving the messages that come from tcp , but >>>>>>> for some reason is not generating flowfiles. >>>>>>> >>>>>>> Does anyone got same behavior ? >>>>>>> There is anything else that i can do ? >>>>>>> >>>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster. >>>>>>> >>>>>>> >>>>>>> Breno >>>>>>> >>>>>>> >>>>>> -- >>>>>> >>>>>> "Genius might be described as a supreme capacity for getting its >>>>>> possessors >>>>>> into trouble of all kinds." >>>>>> -- Samuel Butler -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
