>>> I’m not sure what the purpose of this project is so maybe these >>> maintainability considerations do not apply?
Yes its a purely internal server to server system that has the objective of being as close to real time as possible - it’s not a batch processing system so the idea is to do everything to avoid the disk, pass the data as directly and quickly as possible, and to be in a position to benefit from higher powered multi CPU servers to crunch through the data as fast as possible. So there’s also no need to worry about preserving/losing data if there is a fault because the data is real time live video data and not critical - focus is on being as fast as practical (for long running processes). >>> the architecture you have chosen is non-standard and will be challenging >>> for others to grok quickly and support I’m the only person working on it so that’s ok and I take notes about what I built and why :-) nothing quite like coming backl to your own code six months later and wondering how it works. >>> You are still at risk of another process grabbing the port, you need to >>> open the socket collect the port number in the web host and hold on to it >>> until the fork. Then the child process has the open socket after the fork. Good pickup Rory I’ll do some work in this area. It also appears I might be able to instruct the OS to reserve a range of ports so I can directly control allocations via ip_local_reserved_ports ref: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt this would also ensure the OS or some other process does not grab ports that I allocated to a spawned process. Thanks for your feedback it’s nice to collaborate when in a team of one! On 26 Jul 2018, at 6:18 pm, Rory Hart <hart...@gmail.com> wrote: You are still at risk of another process grabbing the port, you need to open the socket collect the port number in the web host and hold on to it until the fork. Then the child process has the open socket after the fork. I feel the need to point out at this point the architecture you have chosen is non-standard and will be challenging for others to grok quickly and support. For example by opening random ports for incoming connections creating effective firewall rules is next to impossible, as is monitoring traffic. I’m not sure what the purpose of this project is so maybe these maintainability considerations do not apply? On 26 Jul 2018, at 4:58 pm, Andrew Stuart <andrew.stu...@supercoders.com.au> wrote: >>> Your implementation has a race condition in it. If two threads run at the >>> same time and get the same port from “find_free_port” one of your nc calls >>> will fail. > > Hmmm. I’ll have to think on that. Kinda hard to avoid because I need to > know the port number so I can feed it back to the client, and so I can tell > the netcat process which port to start on. And it is possible that the port > number might be grabbed by a subsequent request. Maybe I can do something > hacky like keep track of recently allocated port numbers for a few seconds to > give the netcat process time to start on that port. > >>> Also why are you not using the subprocess module? > > Because I want the netcat processes to be entirely independent of the Python > server that spawns them, so I do a double fork. > > _______________________________________________ > melbourne-pug mailing list > melbourne-pug@python.org > https://mail.python.org/mailman/listinfo/melbourne-pug _______________________________________________ melbourne-pug mailing list melbourne-pug@python.org https://mail.python.org/mailman/listinfo/melbourne-pug _______________________________________________ melbourne-pug mailing list melbourne-pug@python.org https://mail.python.org/mailman/listinfo/melbourne-pug
