Hi Daniel,
we tested two different diodes (two different manufacturer) for a longer time.
Both diodes count the UDP packets and log any loss. Additionally we are using
hashes to detect any form of content errors. We have tested the two diodes for
over a year and have not seen any packet loss
This is a very interesting area of integration investigation Marc, thank
you for sharing your work!
I looked into this a little after conversations with folks in security
applications, and I wonder if you investigated approaches to tracking and
reporting/handling packet loss and error rates in
Adam, that's true, although if your data size is larger than network
MTU there can be some disconnect there.
Connection per flow file is pretty slow for sustained high traffic
flows though (can't recall the establishment times off the top of my
head, but they are non-trivial).
On Tue, Aug 3,
Just spitballing a little here. If you set the configuration of the PutTCP
processor property "Connection per Flowfile" to 'true' and you leave the
"Outgoing Message Delimiter" as blank (none), then I don't think you have
the delimiter problem that you both are describing. I could be wrong though?
Hi Marc,
Thanks for the additional info. Just so you know you’re not the only
one, I’ve also had to re-implement a ListenTCP alternative to get
around the byte delimeter issue for binary and multiline text data.
Phil
On Tue, Aug 3, 2021 at 6:59 AM Marc wrote:
>
> Hi Adam,
>
> more or less it
Hi Adam,
more or less it is a ‚merge', puttcp, listentcp and unpack. I hope that I am
not wrong but the nifi ListenTCP processor uses a delimiter (\n as default?).
If you are transferring binary data the processor splits the flow into
‚pieces'. And the attributes are not transferred to the
Marc,
How would this differ from a more generic use of the existing processors,
PutTCP/ListentTCP and PutUDP/ListenUDP? I'm not sure what value is being
added above these existing processors, but I'm sure I'm missing something.
There's already an ability to serialize flowfiles via MergeContent.
Hi,
no errors can be detected on the sender side (even the nic will not detect if
the other side is down or not). If a UDP packet is lost, the receiver side will
detect and log it. There are diodes that are using an archive. If any
transmission is lost you can easily resend the data manually.
That is interesting stuff - out of interest, if it was sent over that UDP
diode, how would you know whether or not it got to the other side? I
haven’t looked into the site-to-site functionality much yet but I assume it
maintains the providence info?
On Mon, 2 Aug 2021 at 04:26, Marc wrote:
>
