On Wed, May 25, 2011 at 12:09 PM, Alice Bevan–McGregor <al...@gothcandy.com> wrote: > Howdy! > > I've updated the Brainstorm Wiki page [1] to include some > comments-to-the-comments, copied and elaborated upon below. There seems to > be a lot of confusion and/or misinformation being spread about the Marrow > components and I'd like to correct a few points here. > >> MO: My main concerns are, marrow.script is not argparse-compatible, and >> marrow.server.http is asynchronous. I don't think we want to make such a >> large leap from multithreaded to asynchronous in the default server. > >> CM: I'm uncomfortable with the direction of these libs personally. They >> seem to be more researchy than practical in lots of cases. > > I would appreciate elaboration on these points, other than the async one > which is based on an erroneous assumption. "Researchy" isn't even in my > dictionary. ;)
Alice, thank you for correcting any misconceptions about Marrow, and for documenting it on the Brainstorm page for future reference. I apologize if I've made any mischaracterizations, either from reading something that was false or interpreting it wrongly. The Marrow suite looks like a well-thought-out, pythonic collection of services. As such, it definitely serves as an example of one way Pyramid could handle things in the future. It'll be several months before we make any definite decision though. The developers need a break from the year-long push to finish Pylons 1 and Pyramid 1, and from building our own applications under fast-changing frameworks. Pyramid 1 and Pylons 1 are stable now, and several of us would rather focus on using them rather than changing them for a while. The Python 3 porting is enough work for this summer. As you know, a few people are advocating various approaches for replacing Paste*, PasteHTTPServer and the INI file. In the next few months they'll be releasing them and writing HOWTOs for them. If you feel like it, a HOWTO for adding Marrow enhancements to a typical Pyramid or Akhet application would be welcome. After people have gotten some experience using these approaches as third-party addons, they'll be more comfortable saying whether some of them should be integrated into Pyramid 2. > My responses follow. On argparse: > >> Avoiding argparse is a feature, not a bug. If you want argparse, what >> benefit does a scripting interface API give you? Effectively none. >> marrow.script is designed to allow you to write new callables (or use >> existing ones!) and have the command line interface Just Work. Even more >> importantly, marrow.script does not provide a single central script to run; >> your scripts aren't magically part of another package. >> Optparse/argparse/getopt are terribly non-Pythonic, even though argparse had >> the opportunity to fix things! > > Honestly, if you're highly desirous of using any of the existing stdlib > parsers, you have no apparent need for a command line parsing library that > adapts any function into a command line argspec. I took a quick look at the Marrow parser both for Pyramid and for my standalone utilities. My command-line options tend to have complex specifications and interactions, and they don't always correspond directly to function arguments. Perhaps if I thought more about it, they could. But the "magic" in the decorators is not something I need, even if others find it desirable. Argparse does what it does reasonably well; I have not yet gotten sick enough of it to jettison it. For Pyramid, the developers seem to prefer conservative innovation. Make incremental improvements, but not too radical. Obviously, Pyramid is a radical departure from Pylons 1, but that's an occasional paradigm shift akin to Python 3; it's not typical for all Pyramid/Pylons development. And Pyramid did stick with the traditional Paste* wrappings. Many users are still digesting the recent innovations in Pyramid, and are converting existing Pylons programs, or writing new ones from their existing mindset (whether their background is Pylons, BFG, TurboGears, or another WSGI framework). When Python makes a language change, the criteria is, "Is it obviously the best way? Do the advantages significantly outweigh the bother of changing? Both for the users and for the developers?" It takes time for everybody to try out new libraries, and for a consensus to form that one of them is obviously the best way and is a significant advantage. > Paster really provides a centralized command (paster) which only adds > headspace confusion, and context-aware scripts which only add frustration; > e.g. I can't open a shell unless I'm sitting in my source folder… on > deployment there isn't a source folder! Yes, those are a few of the problems with Paste*. But we (Pyramid) need to stop and think awhile about what we want to replace it with. > On marrow.server.http: > >> The underlying communication stack (marrow.io/server) is fundamentally >> async. That particular fact has zero (and I mean it, zero) impact on writing >> WSGI applications that run under the HTTP/1.1 implementation. The underlying >> stack also has full support for a worker thread pool (using Futures) and >> multi-processing. Both of these details are handled for you by the web >> server; if you enable a thread pool, communication is still handled using >> async, but the WSGI application is executed in a thread. This allows you to >> tune a deployment for your requirements. Development or io-bound? Run a base >> server. CPU-bound? Run a threaded server. Need to scale? Add >> multi-processing to either of those. > > I'll mention it again, since this point is very important: the underlying > server being async has ZERO impact on your own applications. It's not like I > magically monkeypatched async support in everywhere; no, instead, only the > core reactor for I/O (which makes sense) is async. The HTTP protocol > implementation plugged into marrow.server either runs the WSGI application > inline (which blocks other requests if not fast enough) or in a thread pool, > your choice. (Choice is good! I have an RPC service running in pure async > mode that can handle ~6KRsec; threading slows it down!) If you can put this in a Pyramid/Marrow HOWTO, it would make users more comfortable about trying it. Most Pyramid/Pylons users come from a multithreaded background. They either know or have learned how to make their views thread-safe. Many have not used async servers and don't know how to make their views async-safe. Normally it involves using special database libraries, socket wrappers (if the view itself is invoking web services or doing RPC etc), and also thinking about what else might be an intolerable blockage (e.g., parsing or producing a multigigabyte file). You say that Marrow users don't have to worry about any of these. In that case, please explain in the HOWTO how to configure Pyramid/Marrow for the various modes, the actual effect of typical blocking constructs (SQLAlchemy, web services, etc) on the various modes, and how Marrow's thread pool compares to PasteHTTPServer's thread pool. Explaining it in the terminology that Pyramid and Pylons users are familiar with will go a long way in making users more willing to try Marrow, and the Pyramid developers more willing to consider Marrow. >> Proper releases of marrow packages require 100% unit test coverage, >> documentation, and multiple working examples. In that case, suitable documentation may already be written. You'd just need to paste some paragraphs into a HOWTO and reword it a bit so that users can see how THIS [what I'm familar with in Pyramid] corresponds to THAT [something new in Marrow]. The correspondence may seem obvious to you, but not necessarily to users. And users may not be willing to spend time figuring out what the correspondence is; they'll just move on to another library. > One of the easiest ways to examine these libraries are via the example code > [2][3], and through the code for the libraries themselves. Additionally, > you'll have to dig into a fork [4] to see demonstrations of WSGI1 > compatibility in marrow.server.http. The adapter [5] is simple, and likely > incomplete, but functional. I'm probably sounding redundant here, but only those who are especially motivated to use Marrow will follow all those references and figure out the correspondence between those examples and Pyramid. Others will move on unless there's a Pyramid-specific HOWTO. Of course, it may not be worth your while to write such a thing, depending on how important Pyramid is to you. But if you seriously want to spread Marrow adoption and/or Pyramid is important to you, it would be worth your while. -- Mike Orr <sluggos...@gmail.com> -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
