Hi all, this is an important discussion also for us (ipython), so I'll go in some detail into things. It would be great if out of this we got something that both ipython and matplotlib could reuse for the long haul, though I'm not sure (in a sense, ipython has some nastier requirements that mpl may not need to worry about).
On 7/17/07, John Hunter <[EMAIL PROTECTED]> wrote: > On 7/17/07, Darren Dale <[EMAIL PROTECTED]> wrote: > > > I'm really impressed with how readable and well organized the code is in > > ipython1. It looks like their approach to configuration has been carefully > > considered. Any chance we can follow their lead? It looks like it would be a > > I haven't looked at it closely, but I am willing to trust Fernando on > this for the most part. I know he has put a lot of thought into it, > and it's something we've talked about for years. Thanks for the compliments. We have put much thought into it, and Brian deserves all the implementation credit. But as you'll see in a moment, all is not quite well; hopefully after this discussion and feedback from you guys we'll find a good solution so we can (finally!) consider this problem fixed and move on to other things. > I am not too fond of the dictionary usage here: > > controllerConfig.listenForEnginesOn['ip'] = '127.0.0.1' > controllerConfig.listenForEnginesOn['port'] = 20000 > > I prefer > > controllerConfig.listenForEnginesOn.ip = '127.0.0.1' > controllerConfig.listenForEnginesOn.port = 20000 > > Since dicts and attrs are intimately linked, eg with something like > Bunch, this should be a minor tweak. Fernando, why did you prefer > dict semantics. And are you happy with the state of your config > system in ipython1 Let's not worry about this for now: I agree with you, but it's an implementation detail that is trivial to fix with a light wrapper, which already basically exists in IPython.ipstruct. The real problems lie elsewhere. Brian and I spent a lot of time discussing this today, and here's the summary of the whole thing. Again, some of these issues may not affect mpl nearly as seriously as they do ipython. Having config files be pure python has some nice benefits: - familiar syntax for users - the ability for users to put in complex logic when declaring their configuration (like make decisions based on OS, hostname, etc). - If developers want, wrapping the underlying objects in something like Traits provides automatically validation mechanisms for configuration, which is important. The security issue is for *me* moot: ipython is a full-access Python system where people are *expected* to run code. So I don't care about letting them run real code at config time. But for other projects, the story is completely different, obviously. I suspect MPL falls into ipython's category here though. So what is the problem? The main one is that we have a real difficult time with circular import dependencies and init-time logic. Basically, you need these configuration objects to exist at startup for other objects to be built, but the configuration *itself* may need to import various things if it is to work as real Python code. In IPython (the dev branches), for example, the user-visible system is assembled out of multiple objects, which themselves need to build their own internal sub-objects. Some of these simply require value parameters at init time, but in some cases even which *class* to use is tweakable at init time (for example, so users can choose amongst various classes that implement different network transport protocols but with identical API as far as IPython goes). While in any one case one can probably disentangle the various dependency chains that arise, it's a major pain to have to deal with this constantly. The fact that the problem is a recurring one is to us an indicator of a design problem that we should address differently. There are other issues as well, besides circular imports: A. Initialization time modification of live objects: while config objects are obviously meant to be built *before* the 'real' objects (who in term use the config ones to assemble themselves), there are cases where one would like to deal with the 'real thing'. For example, in today's IPython (not the dev branch, but the one y'all run) you can add things like magics at init time. This is only possible because when the rc file is parsed, the real object is already up and alive as __IPYTHON__ (also visible via ipapi.get() ). Obviously this is a hack, and a cleaner design would probably separate the declaration of code objects (say magics) that are meant to be fed into the live instance as part of the construction of the config object, and then would have the real ipython load up its tweaks 100% from this config object. But since ipython is also meant to be *runtime* modifiable, it feels natural to let users access such an API for their own tweaks, yet there is a conflict with doing this purely on a config object that is basically just a fancy 'data bag'. B. Automatic creation of default config files. With pure python, this is just hard to do, since you'd be in the business of auto-generating valid code from the structure of an object. For a pure 'class Bunch:pass' type thing this may not be *too* hard, as long as all values allowed are such that eval(repr(foo))==foo, which isn't really always the case. C. Complexity: even for Brian, who wrote the config code, it's hard to figure out where all the details go, since declaring new flags typically ends up requiring fishing inside the config classes themselves. A more plain-text, purely declarative file would summarize these things in a way that would probably be easier for both developers and end users. Given all this, we are currently looking at going back to a mixture of text-based config and real code, as a two-stage process. The ConfigObj library: http://www.voidspace.org.uk/python/configobj.html looks like a nice, self-contained step above hand-parsing rc files or Python's rather primitive ConfigParser module. It has a lot of good features, is mature and is small enough that one can ship it inside the project (license is BSD). Our current plan is something along these lines: 1. Expose for all configuration objects the part that is really just a key=value API over ConfigObj. This would dispense with all the circular import problems, and let users specify the most common things they want to tweak with a really simple syntax (that of .ini files). We would then load these and build all necessary objects. 2. Make sure that the part of any given object that we mean to expose for *runtime* modification (say adding magics to a running ipython, for example) is easily available. This would let users modify or extend those aspects of the system that can't just be easily declared as simple key=value settings. The mechanism for this would simply be to have a filename in each object's ConfigObj session like: [SomeObject] post_init_file = "my_post_init.py" This file would be run via execfile() *once SomeObject* were guaranteed to be up and running. This would then allow users to have code that manipulates SomeObject programatically, via its runtime API. All this conversation brings us to... 3. Traits. We (Brian and I) have gone back and forth a lot on Traits, and we've come very close to just making them a dependency. The only real issue holding us back is that ipython so far has exactly *zero* extension code, which is a plus in terms of ease of installation/deployment. Having said that, as far as extension code is concerned, Traits is light and clean, and nowhere near the kinds of complexities that MPL deals with. But since anything is more than zero, it is a bit of an issue for us. We may tip over though, I'm just stating what our reasoning so far has been. In terms of Traits, point (2) above makes them even more attractive. The delegation aspect of Traits is a very appealing way of combining validation with additional action for the runtime modification of an object. For example: ipython.color_scheme = "foo" If color_scheme were a Trait, the above could simply: a) validate that "foo" was acceptable as a value b) trigger the chain of other trait updates (dependent color schemes for exceptions, prompts, etc). All of this can obviously be done with python properties, but before I write Traits again, poorly (I'm no David Morrill) I'd rather ride Enthought's back. This approach via Traits (or something like them) also ensures that the work done in ensuring the consitency/robustness of an object's *runtime* configuration API becomes automatically useful to users, and it simplifies the init-time config API, since it gives us the option to defer more complicated tasks to runtime. So this is a summary of where we stand. It's surprising that the 'simple question' of configuring an application can turn out to be such a can of worms, but for us it has proven to be. Input/ideas from you all would be very welcome, as it's quite likely we've missed possible solutions. I'd love to get out of this discussion some ideas, clarity and ultimately a plan for how to proceed in the long haul. If in addition this means that ipython, mpl and others end up uniformizing further our approach to this problem, even better! Having our end users find familiar configuration mechanisms across their various libraries would only be a good thing in the long run. Cheers, Brian and f. ps - I debated on having this discussion on ipython-dev, but for now I'm going to not cross-post. The MPL team is attentive to the issue now, so I'd rather collect your wisdom here, and I'll take it upon myself to summarize our conclusions on ipython-dev later. I just want to avoid list cross-posting. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
