What you are talking about is known as a stateful service <http://highscalability.com/blog/2015/10/12/making-the-case-for-building-scalable-stateful-services-in-t.html> and they're great if your problem has the right dimensions and you invest in good controls and monitoring/alerting. The main issue is they are tricky design to scale and plan disaster recovery for.
Beyond that as a general rule mixing your batch processing (PNG image processing) and your web services isn't a good idea. For example the load spikes from the batch processing can cause response timeouts and as web services don't generally have high CPU load you'll have trouble using the CPU cycles you are paying for. Do you have access to one of the cloud services? This is the sort of problem their FaaS (AWS Lambda etc) systems are well suited to. For example I would do something like this with Amazon's cloud: 1. Web service accepting PNGs via S3 (AWS's file storage service). 2. Web service triggering Lambda to do processing. 3. Client polling for processing completion. 4.a Client download image from S3. 4.b or Client tell web service new command on PNG triggering another lambda. You could also do some similar with without the cloud with Celery and a shared filesystem such as glusterfs. On Tue, 24 Jul 2018 at 07:27, Andrew Stuart < andrew.stu...@supercoders.com.au> wrote: > I have servers that send a sequence of PNG images which need to be > processed via a sequence of commands. > > My idea is this: > > A Python websockets server is listening for connections from the servers. > When a websocket connection is received, a new Python process is spawned - > presumably using https://docs.python.org/3/library/multiprocessing.html > and somehow the websocket is given to that process (how?). The spawned > process listens to the websocket, receives the PNG images and then > processes them. After completion, the spawned process dies and the > websocket is closed. So the spawned Python processes might run for up to a > few minutes each. > > Is this a workable idea in Python? > > One open question is how would I hand the websocket to the spawned process? > > Also, coud such an approach support tens, hundreds, thousands of > connections on a for example 8 core machine with 16 gig RAM ( know this is > how long is a piece of string, but just finger in the air guessing to make > sure there’s no huge gotcha in there that would impose a huge constraint & > invalidate the idea)? > > I was looking at this gist as inspiration for a start point > https://gist.github.com/gdamjan/d7333a4d9069af96fa4d > > Any ideas/feedback valued. > > thanks > > Andrew > _______________________________________________ > melbourne-pug mailing list > melbourne-pug@python.org > https://mail.python.org/mailman/listinfo/melbourne-pug >
_______________________________________________ melbourne-pug mailing list melbourne-pug@python.org https://mail.python.org/mailman/listinfo/melbourne-pug
