Hi, pretty long stuff, i will answer point by point below :) > > ---------------------------------------------- > `--worker-exec` describes "controlling a pool of fast-cgi processes > [php]" ... i could get the [php] processes running, and execute python > from the master process *once*, and afterwards the children/workers > would receive requests directly ... this is all fine/makes sense ... > but how would you suggest to execute python again? the spooler runs > in a separate process (yes?) and thus it could be used as a way to > control the "fcgi pool" via uwsgi module/etc. i never tried signals > but that seems plausible as well ... in short i want to use > python/uwsgi to manage the php i _have_ to deal with; any ideas? make > a dumb php pool and proxy to it [see next paragraph]? how do i > properly (non-locking?) control this python pack leader? i'm > especially interested in combining this with zerg mode and scaling > (via an "intelligent python framework") vs. php's own process manager > ...
when you set a binary with --worker-exec, those workers will lose all of the old process image. No way to came back to python. Only the master and the spooler continue to maintain the old image. An intersting thing is that you can send signals (or tasks) via php to the spooler (simply a one byte write() ) as the signal file descriptor is inherited. A solution for your problem will be allowing worker-exec to be mapped only to a list of workers: --processes 8 --worker-exec /usr/bin/php-cgi:1,2,3,4 will run php only on the first 4 nodes, the others will continue normally. By the way, Zerg mode works without problems in --worker-exec mode > > ... once the php is running, how do i actually serve a request to it > via uwsgi alone? do i need 2 instances of uwsgi? i tried > `--http`/`--http-socket` but the children expected fastcgi protocol > and thus failed when given HTTP. i tried various combinations of > `--fastcgi-socket` and anything that seemed to make sense ... but what > is the "proper" way :-)? i want to use uwsgi alone (multiple is fine > if needed) to deliver an HTTP request to the PHP processes ... so, how > do i make uwsgi speak both ends of the fastcgi proto? > ---------------------------------------------- currently the uwsgi http router can route only to uwsgi servers. Adding fastcgi support would be easy as all of the functions are already available for the uwsgi.fcgi() function: http://projects.unbit.it/uwsgi/wiki/CustomRouting (check part 8, you could even use it to route from uwsgi to php right now) > > ---------------------------------------------- > **bonus** is/can any of the above be made compatible with op-code > caching, a la APC? > ---------------------------------------------- when run in fastcgi mode, whatever opcode caching can be used with php. > > > ---------------------------------------------- > i'm loving the embedded stuff. in the spirit of ROCK SOLID > mode/build, how about a SEALED or FROZEN mode where the only config > allowed are internal `sym://config_ini` (or `sym://*`, as long as it's > embedded [think includes])? this way the "authoritative" config is > locked away internal to the process and not easily changeable -- no > command line args would be accepted ... i suppose that a whitelist/blacklist option subsystem would be much more useful, expecially for cloud/paas service providers that want to leave a bit of control to their customer instances. a 'blacklist = all' will create a "frozen" binary. > > ... could even be a slick way to write a uwsgi-based control app for > uwsgi itself ... fire up uwsgi (`--no-server`?), and send some > commands to wherever, with it's own CLI :-) > ---------------------------------------------- this could be already accomplished with --pyshell (and --rbshell for ruby [currently broken]) > > > ---------------------------------------------- > i'd like to embed the interpreter itself, for use by uwsgi, *inside* a > `data://` section. thoughts on this? part of my single-file, > self-contained, application-deployment ... pipe dream :-). brief > high-level description: > > a. uwsgi is compiled with a "boot" device (data://<N>, default 0) > (checksum/signed else no boot? hmmm ....) > b. boot image contains python (or <insert lang>) libraries in a known > location, within a compressed image > c. boot image is placed in slot zero. > d. uwsgi reads out what it needs from archive to initialize itself, > and starts the application (could be combined with `sym://config_ini`) > e. possibly uwsgi extracts some or all of the image (rel or abs) to > local FS for scratch space? (tmp/shared/ directories, etc.) my use > case would likely need this, unless ... > > ... slight variant ... a new, dedicated location (say boot://) > contained the image, in the same sense as an initramfs image? maybe a > `boot://` AND `root://` (same purpose as the kernel analogues) ... > boot is in ram/tmpfs, and cleared once `root://` is IDENTIFIED > (netowrk/etc?) and "mounted" (extracted/etc). `root://` would have > the normal operational environment and any additional resources (just > like a normal *nix system works basically) it is all wonderful, but sadly no unix-variant out-there has support for exec/dlopen of stream of data (they must be real files). For example i am embedding psycopg2 in my uWSGI binary compiling it as a static library (.a) and linking to uWSGi itself (that will call initpsycopg2() on startup). Really hard job, but it is currently working. The problem is that automating this is pratically impossibile :( Embedding python is possible (simply link uWSGI with the static libpython variant, this is the default on some distro that do not compile python with --enable-shared), but you will need its core-modules. You can put the whole /usr/lib/pythonX.X in zip and it works out of the box, but you will always require an additional file (the .zip one) > > ... lastly, how do i even access arbitrary data in `data://` slots? > the --import and config stuff worked fine (and > `uwsgi.ZipImporter("data://0")` once i found the example :-) ... but > what if i had a PNG or an unrelated tarball? does `data://` work > anywhere in python at this point? (didn't actually try that last bit i > admit :-X) in short, data is appended, but how to use if not a python > module? > > thoughts? uwsgi.extract("url") (check bootstrap5.py) or uwsgi.embedded_data(name) (only for symbols) > ---------------------------------------------- > > > ---------------------------------------------- > i found `attach.py` in the repo, but doesn't seem to be in tarballs. > would be nice a small utility to ins/rm/reorder/repack. `data://0` > read from end first is counter-intuitive for user ... i can guess the > impl advantage, so i'd merely suggest the said tool "prepend" objects > instead, as then the numbering stays consistent. else, everytime you > add a new object, all the numbers change. another option is reversing > the numbering ... ooooor provide a means for symbolic naming of the > data objects vs. logical order -- i like this best i think :-) > > i may do this myself, as it's a incredible ... 13-liner ... but i > don't understand the `struct.pack("<Q", size)` 100% yet (any off-hand > links/tutorials for this would be great ... lower level than i usually > play ;-) so i'm not positive what the code is actually doing at this > point. > ---------------------------------------------- this simply add a 8byte (64bit) value to the end of file. This value is the size of the attached file. This is why they are in reversed order. I have to scan backward, as i do not know the size of the original binary :) > > > ---------------------------------------------- > barring whether or not it makes sense :-) [see end], could FUSE be ran > from a uwsgi app? how would they interact? would this be better > suited for `--attach-daemon` or .... ??? haven't got to this point > yet but any foresight appreciated. > ---------------------------------------------- funny enough, --attach-daemon born for this exact reason. I needed a process controller ensuring that a fuse daemon was running ONLY when uWSGI was running. Why not 'attach' it to uWSGI itself using kernel facilities ? Currently Unbit customers abuse it to attach memcached and redis without paying for additional process slot (they use directly the uWSGI assigned one). So feel free to abuse it too for your ambitious project :) > ---------------------------------------------- > how about `(sym|data)://` loading a pre-compiled python module? right > now it pukes expecting a python source file. this could possibly be > extended generically to support multiple python impls via the py3k > __pycache__ lookup system ... is there any benefit to this? besides > maybe speed, my other thought was obscuring the python module's source > from simple eyes. it is doable, maybe it is only a two-line fix. > > ... and similar for `sym://config_ini` -- possibility to compress/etc > so it's not plain text in the binary? im also thinking about total > binary size -- plain text files get excellent compression. > > i see interesting uses like encrypted/verified boot of sorts sprouting > from this and previous notes. compression/encryption functions will require new dependancies and a lot of code. This code should go to plugins (no need to increase the already big core). i will investigate on this after 1.0 > > ---------------------------------------------- > how can i escape in ini files? my app uses a "magic URL" to activate > a "developer console" ... it *was*: > > http://www.example.com/= > > ... but i was unable to convince uwsgi (via ini) that `mount = > /==myapp:app`, or any encoded variant ... was valid :-). however, i > just discovered JSON config, and it's more suited in my case > (ultimately zeromq), and i decided on `??` over `=` for other > advantages, but probably still a bug. '=' get escaped by your browser, so there will be no more match in the APP_ID/SCRIPT_NAME/mountpoint value > > i was unable to embedded a config with path components, eg. > `sym://path_to_config_ini` failed with "unknown symbols/first use", > etc ... reproduce by setting: > > embed_config = path/to/config.ini > > ... in your build conf. lmk if i need to expand this one, probably an > easy fix. the ld embedding subsystem is a bit "strange". It will rename your full path to something like _path_to_config_ini. use objdump -t <binary_name> to get the assigned symbol name > - > 0.9.9.2 announcement mentions a SCRIPT_NAME fix, and via testing it > appears to be related to `--http-socket` ... regardless, PATH_INFO is > *not* being updated correctly for `--http-socket` driven apps (works > fine for `--http`, what exaclty is the difference anyway?) > > reproduce by `--mount`'ing an app, and dumping the wsgi environ from > that app. (SCRIPT_NAME works fine, but PATH_INFO is not updated > accordingly) > ---------------------------------------------- native http protocol has currently no support for multiple mountpoints. > > > ---------------------------------------------- > is it possible to set `uwsgi.applications` from `--eval`? it doesn't > seem to be available that early. i could find no way to configure a > mounted/multi-app from `--eval`. i later realized that `--import` > works just as well for me, though i was trying to embed some init code > into the `sym://config_ini` file itself (specifically the ZipImporter, > see next ...) > ---------------------------------------------- you can specify unlimited --import options, so you can get rid of this with a bunch of embedded bootstrap files > > > ---------------------------------------------- > i see the bootstrapN.py examples, thanks for those gems :-) what do > you think about enabling such imports by default, or some persistent > way? appending a zip file -- that is a python module -- seems a > fairly elegant/common use case for `data://` (or `sym://`). Cal (the original author/mind of all this funny things) pushed me to the bootstrap approach (you embed and load a series of bootstrap files to extend uWSGI behaviour), and i think it is the most fexible choice. I would like to not include too much high level functions in uWSGI-core (plugins are for this). So maybe an auto-sym-load plugins will be a better choice. > > > my project is a small daemon of sorts that runs on a server and > manages connections to/from client websites. this code will have > various authentication modules and access methods (SFTP, webdav, not > 100% sure yet). > > however, it will also host FUSE modules that allow me to spawn a near > *instant* development environment _right over the top_ of the live > site by use of linux mount namespaces, FUSE, COW, and some > cookie/userscript magic ... this lets a site be in unlimited > stages/versions, at the _production_ address, switchable by cookie > (one-time use: client) or userscript (persistent: development team). > It is a sort of holy-grail, but i suppose only a few pieces are missing (maybe the fuse approach will be the key to success). Please continue to send updates and ideas :) -- Roberto De Ioris http://unbit.it _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
