On Sat, Sep 10, 2011 at 2:54 PM, Roberto De Ioris <[email protected]> wrote:
>
> By the way, this area is a total work in progress (most of the ideas are
> from Cal Leeming of Simplicity Media) so feel free to push your
> work/problems/suggestions here
awesome -- thanks for all the pointers -- i was able to get everything
working as described. i'm not nearly complete with the project i'm
working on to link/etc, but i have several notes of possible
bugs/features/suggestions building up soo ... here they are :-)
notes are in "discovery" order, followed by possible bugs; i can
follow thru with reports/etc if need be -- simply dumping for comments
right now. i'll conclude with a brief description of what im trying
to accomplish/build as it may improve the clarity (or not :-) ... any
recommendations/suggestions on said project are also welcomed.
warning: loosely correlated and borderline rambling ...
----------------------------------------------
`--worker-exec` describes "controlling a pool of fast-cgi processes
[php]" ... i could get the [php] processes running, and execute python
from the master process *once*, and afterwards the children/workers
would receive requests directly ... this is all fine/makes sense ...
but how would you suggest to execute python again? the spooler runs
in a separate process (yes?) and thus it could be used as a way to
control the "fcgi pool" via uwsgi module/etc. i never tried signals
but that seems plausible as well ... in short i want to use
python/uwsgi to manage the php i _have_ to deal with; any ideas? make
a dumb php pool and proxy to it [see next paragraph]? how do i
properly (non-locking?) control this python pack leader? i'm
especially interested in combining this with zerg mode and scaling
(via an "intelligent python framework") vs. php's own process manager
...
... once the php is running, how do i actually serve a request to it
via uwsgi alone? do i need 2 instances of uwsgi? i tried
`--http`/`--http-socket` but the children expected fastcgi protocol
and thus failed when given HTTP. i tried various combinations of
`--fastcgi-socket` and anything that seemed to make sense ... but what
is the "proper" way :-)? i want to use uwsgi alone (multiple is fine
if needed) to deliver an HTTP request to the PHP processes ... so, how
do i make uwsgi speak both ends of the fastcgi proto?
----------------------------------------------
----------------------------------------------
**bonus** is/can any of the above be made compatible with op-code
caching, a la APC?
----------------------------------------------
----------------------------------------------
i'm loving the embedded stuff. in the spirit of ROCK SOLID
mode/build, how about a SEALED or FROZEN mode where the only config
allowed are internal `sym://config_ini` (or `sym://*`, as long as it's
embedded [think includes])? this way the "authoritative" config is
locked away internal to the process and not easily changeable -- no
command line args would be accepted ...
... with positive effect: any args are passed to the application (eg.
`--pyargv`). this would let us write an application and wrap/seal it
with uwsgi for delivery/demo/anything. from that point on, invoking
uwsgi is _really like invoking any regular python app_, but with all
the goodness of uwsgi:
./myapp -a -b -c -- parameters for my applicat10n
... could even be a slick way to write a uwsgi-based control app for
uwsgi itself ... fire up uwsgi (`--no-server`?), and send some
commands to wherever, with it's own CLI :-)
----------------------------------------------
----------------------------------------------
i'd like to embed the interpreter itself, for use by uwsgi, *inside* a
`data://` section. thoughts on this? part of my single-file,
self-contained, application-deployment ... pipe dream :-). brief
high-level description:
a. uwsgi is compiled with a "boot" device (data://<N>, default 0)
(checksum/signed else no boot? hmmm ....)
b. boot image contains python (or <insert lang>) libraries in a known
location, within a compressed image
c. boot image is placed in slot zero.
d. uwsgi reads out what it needs from archive to initialize itself,
and starts the application (could be combined with `sym://config_ini`)
e. possibly uwsgi extracts some or all of the image (rel or abs) to
local FS for scratch space? (tmp/shared/ directories, etc.) my use
case would likely need this, unless ...
... slight variant ... a new, dedicated location (say boot://)
contained the image, in the same sense as an initramfs image? maybe a
`boot://` AND `root://` (same purpose as the kernel analogues) ...
boot is in ram/tmpfs, and cleared once `root://` is IDENTIFIED
(netowrk/etc?) and "mounted" (extracted/etc). `root://` would have
the normal operational environment and any additional resources (just
like a normal *nix system works basically)
... lastly, how do i even access arbitrary data in `data://` slots?
the --import and config stuff worked fine (and
`uwsgi.ZipImporter("data://0")` once i found the example :-) ... but
what if i had a PNG or an unrelated tarball? does `data://` work
anywhere in python at this point? (didn't actually try that last bit i
admit :-X) in short, data is appended, but how to use if not a python
module?
thoughts?
----------------------------------------------
----------------------------------------------
i found `attach.py` in the repo, but doesn't seem to be in tarballs.
would be nice a small utility to ins/rm/reorder/repack. `data://0`
read from end first is counter-intuitive for user ... i can guess the
impl advantage, so i'd merely suggest the said tool "prepend" objects
instead, as then the numbering stays consistent. else, everytime you
add a new object, all the numbers change. another option is reversing
the numbering ... ooooor provide a means for symbolic naming of the
data objects vs. logical order -- i like this best i think :-)
i may do this myself, as it's a incredible ... 13-liner ... but i
don't understand the `struct.pack("<Q", size)` 100% yet (any off-hand
links/tutorials for this would be great ... lower level than i usually
play ;-) so i'm not positive what the code is actually doing at this
point.
----------------------------------------------
----------------------------------------------
barring whether or not it makes sense :-) [see end], could FUSE be ran
from a uwsgi app? how would they interact? would this be better
suited for `--attach-daemon` or .... ??? haven't got to this point
yet but any foresight appreciated.
----------------------------------------------
----------------------------------------------
how about `(sym|data)://` loading a pre-compiled python module? right
now it pukes expecting a python source file. this could possibly be
extended generically to support multiple python impls via the py3k
__pycache__ lookup system ... is there any benefit to this? besides
maybe speed, my other thought was obscuring the python module's source
from simple eyes.
... and similar for `sym://config_ini` -- possibility to compress/etc
so it's not plain text in the binary? im also thinking about total
binary size -- plain text files get excellent compression.
i see interesting uses like encrypted/verified boot of sorts sprouting
from this and previous notes.
----------------------------------------------
# BUGS? .....
----------------------------------------------
how can i escape in ini files? my app uses a "magic URL" to activate
a "developer console" ... it *was*:
http://www.example.com/=
... but i was unable to convince uwsgi (via ini) that `mount =
/==myapp:app`, or any encoded variant ... was valid :-). however, i
just discovered JSON config, and it's more suited in my case
(ultimately zeromq), and i decided on `??` over `=` for other
advantages, but probably still a bug.
----------------------------------------------
----------------------------------------------
i was unable to embedded a config with path components, eg.
`sym://path_to_config_ini` failed with "unknown symbols/first use",
etc ... reproduce by setting:
embed_config = path/to/config.ini
... in your build conf. lmk if i need to expand this one, probably an easy fix.
----------------------------------------------
----------------------------------------------
0.9.9.2 announcement mentions a SCRIPT_NAME fix, and via testing it
appears to be related to `--http-socket` ... regardless, PATH_INFO is
*not* being updated correctly for `--http-socket` driven apps (works
fine for `--http`, what exaclty is the difference anyway?)
reproduce by `--mount`'ing an app, and dumping the wsgi environ from
that app. (SCRIPT_NAME works fine, but PATH_INFO is not updated
accordingly)
----------------------------------------------
----------------------------------------------
is it possible to set `uwsgi.applications` from `--eval`? it doesn't
seem to be available that early. i could find no way to configure a
mounted/multi-app from `--eval`. i later realized that `--import`
works just as well for me, though i was trying to embed some init code
into the `sym://config_ini` file itself (specifically the ZipImporter,
see next ...)
----------------------------------------------
----------------------------------------------
i see the bootstrapN.py examples, thanks for those gems :-) what do
you think about enabling such imports by default, or some persistent
way? appending a zip file -- that is a python module -- seems a
fairly elegant/common use case for `data://` (or `sym://`).
this *may* have worked fine from `--eval`, i don't recall off-hand,
but it feels too hacky that way.
----------------------------------------------
... mmk, that was a *bit* more than i intended to write, but hopefully
interesting and useful to others.
my project is a small daemon of sorts that runs on a server and
manages connections to/from client websites. this code will have
various authentication modules and access methods (SFTP, webdav, not
100% sure yet).
however, it will also host FUSE modules that allow me to spawn a near
*instant* development environment _right over the top_ of the live
site by use of linux mount namespaces, FUSE, COW, and some
cookie/userscript magic ... this lets a site be in unlimited
stages/versions, at the _production_ address, switchable by cookie
(one-time use: client) or userscript (persistent: development team).
the idea is to create a transparent buffer between the production
environment and mac/pc designers/developers who know squat about
revision control, linux, or CLI, and enable them pain-free and mostly
new-skill-free use (hence the "familiar" SFTP/webdav access) of the
system, while also giving me introspective superpowers as to the exact
configurations and history of changes.
tl;dr ... sound like fun, huh? :-p
thanks to all for making it this far, i greatly appreciate any
comments/insights offered.
--
C Anthony
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
