Hi, On Thu, Dec 30, 2021 at 3:33 PM 'Wayne Harris' via Racket Users < racket-users@googlegroups.com> wrote:
> I'm considering writing a manager for background processes --- such as > send a batch of e-mail or other process that takes a while to finish --- > for a web system. > > I see the challenge here as just writing something that will look like a > very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on. > (``Web'' because it will be used by a web system through some HTTP API.) > > This thing has to be flawless. I'm looking for design principles and > advice. > > I don't know which language I will use, but I'd like to use Racket at > least as a prototype. I am looking at section 15.4 at > > https://docs.racket-lang.org/reference/os.html > > and I'm not sure it gives me all the control I need. I have a more > lower view of the job --- fork(), execve(), waitpid(), SIGCHLD. But I > suppose Racket handles this much more elegantly that I would in C. > > Your advice will be very appreciated. > > (*) Where will it run > > It will run on GNU systems running the Linux kernel. > > (*) My own thoughts > > The interface to shell will be through HTTP requests, so this shell will > likely be a web server of some sort. But before I get involved in the > web at all, I need the shell working flawlessly. > > So I need a laboratory first. I could write a program that reads some > named pipe on disk to get commands such as ``run this or that'' while I > work. (Later I can write a web server replacing this named-pipe > interface.) > > Just like a UNIX shell, this web-api-shell must know all every process > it runs. I suppose the work is essentially fork(), execve() followed by > waitpid(). > > One concern I have is the following. Is it possible for a process to > simply ``get out of'' the shell? What do I mean by that? A process > that does fork() and puts itself into background would leave the > web-api-shell's control, wouldn't it? > > I think I must avoid that. Perhaps I can't let just any process run. > Perhaps the web-api-shell must only offer a few processes carefully > written by myself --- so that I know they won't put themselves in > background. (For instance, I can't let them change PIDs, otherwise I > won't have any idea who they are and that's a mess. I'd love to somehow > restrict system calls such as fork().) > > (*) Serialization > > I also think this web-api-shell must not be invoked in parallel. So I > guess I must use some queue of requests with no race condition and > pull each request as it comes. Any pointers on how to do this basic > thing with my zero experience? > > (*) What is my level of training? > > In the past I've studied many parts of > > Advanced Programming in the UNIX Environment > W. Richard Stevens > > I will definitely have to read it again to get work on this project. > Can you mention any UNIX concepts that are of great relevance for this > project? I don't think I ever got my mind wrapped around things like > sessions, session leaders and so on. Are these concepts relevant to > this application? > As you suspected, in Racket, the approach will be significantly different than in C—hopefully, safer and more elegant! The most basic concepts you will need to learn about are Racket's "green" threads (not OS/POSIX threads) and "synchronizable events," which are based on Concurrent ML. A good place to start would be the tutorial introduction "More: Systems Programming with Racket": https://docs.racket-lang.org/more/index.html You will surely also want to read the Racket Guide chapter on "Concurrency and Synchronization" (https://docs.racket-lang.org/guide/concurrency.html) and the associated Racket Reference sections. I have found Matthew and Robbie's paper “Kill-Safe Synchronization Abstractions” ( https://www.cs.utah.edu/plt/publications/pldi04-ff.pdf) an accessible introduction to "synchronizable events" and the Concurrent ML implementation. Andy Wingo, the Guile maintainer, has some blog posts about how Concurrent ML's approach to concurrency works under the hood: https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml For maximum detail, I can also recommend John Reppy's book *Concurrent Programming in ML* from Cambridge UP (most recently revised in 2007, IIUC). Once you understand Racket's approach to synchronization in general, I think the way OS-level facilities (especially processes: https://docs.racket-lang.org/reference/subprocess.html) interact with them will make more sense. In particular, when we say that something is "blocking" in Racket, we mean that it blocks a Racket-level green thread, while allowing other Racket threads implemented by the same OS thread to run concurrently. (It is possible to block the whole process using unsafe functionality from the FFI, but that would generally be a bug.) This means that even the simple `system*/exit-code` function, wrapped in a call to `thread`, can do very well for running a background process, especially in combination with `current-subprocess-custodian-mode` and `subprocess-group-enabled`. It's hard to give much more specific advice without knowing more about your requirements. I guess I'd say that overall, because Racket provides such powerful facilities for concurrent programming, I find that I can often just write the code I want, without needing indirection through a general-purpose background task manager. For example, a web application can very reasonably send email just by using the `net/sendmail` or `net/imap` libraries in a function wrapped in a Racket-level thread, perhaps even (depending on the behavior you want) directly in the per-request thread created for you by the Racket web server. For a different approach, while I don't know much about the implementations, you might be interested in the GNU Daemon Shepherd and Mcron, an init system and cron implementation written in Guile: see https://www.gnu.org/software/shepherd/manual/html_node/ and https://www.gnu.org/software/mcron/manual/html_node/index.html. -Philip -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/0100017e2c2ac66c-7d87be7b-b375-4019-928d-cac2e0a64aec-000000%40email.amazonses.com.
