On Fri, Nov 09, 2012 at 06:27:06PM +0200, Dan Shechter wrote: > I can do some assumptions regarding the TCP flow and its origins. Its > coming from the stock exchange over IPSEC gateways over leased lines. > I think I can trust the origin of the flow. At least I can trust it as > much as the off the shelf software does.
If something goes wrong with the off-the-shelf software, you can blame the vendor. Your own hand-rolled solution, not so much... > When I was saying money is not a problem, I was referring to the cost > of the server to run this. I know, you said that already. But I've worked in this industry also and I am well aware of the reality distortion that occurs when gambling with billions of dollars of other people's imaginary money. > I know that I need to implement state machine for the TCP session and > keep some buffers for out of order packets. > > Do you think the right place to place the code is in ether_input.c? I gues you mean to say either sys/net/ip_ethersubr.c, or the ether_input() function inside that file, but either way the bulk of your code should get added to a separate file if you don't want a maintenance nightmare. > Its about 1k packets per second max. This should be doable on all but the most ancient hardware. But you will need to consider how you want to handle bursts or anomalies: is it more important to never lose a packet, or is it acceptable to lose some number of packets in order to keep latency low? In the former case you need to rely on buffers; you will want to move packets off the interface recieve ring and into another buffer like the ip input queue as quickly as possible, then do your magic in software interrupt context. You'll also want to look at disabling the MCLGETI functionality in the network card driver, and possible increase the recieve ring size on the card. Here you probably want to hook your code at the very beginning of ipv4_input(). In the later case, you'll want to reduce the recieve ring size on the interface and handle the bulk of your processing in the hardware interrupt. Fragment reassembly, handling TCP resends and out-of-order packets, etc may no longer be useful (since it requires buffering packets), and you may opt to simply drop data that doesn't arrive in correct sequence. This is when ether_input() would be the right place to hook your code. > I plan to coil the path cable to make electrical filed surrounding my > device to protect it from evil. Think about how you might be able to use ACLs on a high-end switch to guarantee that the packets you recieve fit a certain profile (for example, ensure that all packets are IPv4 TCP port 12345 between hostA and hostB), to help shrink your code path. Similarly, it may be possible to configure the device that's handling the IPsec tunnel to do IP and TCP reassembly for you (if not, can you replace it with one that does?), in which case your code could be made MUCH simpler. You didn't mention the protocol you're handling, but solutions like the following may be helpful (or you might be able to implement the whole thing there, and avoid supporting a frankenkernel). http://www.brocade.com/solutions-technology/enterprise/application-delivery/fix-financial-applications/index.page It's optimized for cloud service delivery, so it's at least 9000x as awesome as OpenBSD.
