On Tuesday 09 May 2006 07:22, Jim Fulton wrote: > I guess we need to make this a priority for the next release. > > Python simply does not support a general robust reload, other than > restart. > > I think that there are 2 ways we can make progress in this area: > > - Speed up restart. I think there are a lot of ways that restarts > can be made faster: > > o Optimizae what we're doing now. I suspect that there are some > opportunuties here.
I have applied for the SoC with a proposal to enhance ZCML. My proposal is attached. It discusses some of the optimization options we talked of before. If accepted I would work on this and the result would be, naturally, in the next Zope 3 and 2 release. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
============== Enhancing ZCML ============== Name: Stephan Richter E-Mail: [EMAIL PROTECTED] IRC Nickname: srichter How much time do you expect to have for this project? Please list jobs, summer classes, and/or vacations that you'll need to work around: I am a Ph.D. student and do take any classes anymore. I am going to teach the first summer session Physics 11 (Calculus-based mechanics) and continue my thesis. Overall I think 3-4 weeks of time during the summer are a realistic estimate. Development experience: - 9 years of Python experience. - 6 years of Zope experience. - Zope 3 core developer and release manager. Please describe your usage experience/familiarity with the project you are applying for: I am a Zope 3 core developer. I am planning to implement new features in Zope 3 that will make it easier to develop highly-customizable content management systems, like Plone, using Zope 3. What school do you attend? How many years have you attended there? What is your specialty/major at the school? Tufts University, Somerville, MA. Ph.D. in Physics, started in 2002. I am developing models to simulate the immune system repsonse upon SIV/HIV infections. Project Details: Project Outline --------------- Zope 3 uses a component architecture as one of its most basic building blocks. Since Zope 3 is a Web Application server, it must be possible to run multiple Web sites on one application instance. The consequence is that Zope must be configurable on a site by site basis. Thus an extension of the component architecture allows us to define components globally (for all sites) and locally (site specific). Global components are commonly configured using an XML-based configuration language called ZCML. Currently, local components can only be created and configured via the Web UI. The advantage of doing so is that local components can store their state in the ZODB. But this also means that it is very difficult and cumbersome to register local components using regular filesystem-based code. Clearly, this functionality is sub-optimal. Oftentimes you want to be able to define site-specific (local) components from the filesystem. This is particularly true for presentation code, where it is often not a requirement that the state must be stored in the ZODB. I propose to allow local components to be configurable through ZCML. This goal became feasible with the recent component architecture refactorings by Jim Fulton. Any site (local or global) can now have a set of base sites that are used to provide additional components. I want to allow ZCML to specify any number of base sites and add components to it. Here is an example of how I imagine it to look like (ZCML):: <site name="my-base-site" /> <configure use-site="my-base-site"> ... </configure> The new ``site`` directive creates a new site. The ``configure`` directive will grow an ``use-site`` attribute that specifies the site to put in the components. By default, ``use-site`` will use the global site. This also ensures full backward-compatibility. There will be a new registry of all ZCML-defined sites. All existing ZCML directives have to be reviewed and it must be ensured that they are multi-site aware. The tricky part of the implementation will be to hook in those sites as bases to local sites. It must be ensured that the ZODB can load having filesystem-based sub-sites, error handling must be carefully considered and an UI must be written. I believe that this feature is essential for highly customizable applications liek Plone. It will allow skinning of sites using filesystem-based development, something several people in the Plone community have strived for for a long time. I think once the community starts to discover the implications of this development, many new possibilities will open. Another big problem in Zope 3 is the startup time. Some code profiling has determined that most of the time in the startup process is lost in parsing, converting and validating ZCML directives with their schemas. Thus, this startup problem is not purely a Zope 3 problem, but one that affects everyone using ZCML. This problem can be addressed in several ways. The most obvious one to Zope 2 developers would be not to restart the application server, but only reload the packages and modules that were affected by the code changes. This approach has been used in Zope 2 for many years, but it several serious problems and some of the smartest people I know have not been able to completely solve the problems. Based on that, I do not think that a proposal suggesting this approach would be accepted. The second approach is to reduce the ZCML processing time, which could be integrated into the reload mechanism for Zope 2. This can be accomplished by storing some binary representation of the ZCML, similarly to ``*.pyc`` files in Python. Again there are several choices to consider and they should probably all be tried. The first solution would be to store a pickle of each parsed directive, namely the action and its arguments. There would be one pickle file fore each ZCML file. When the ZCML file changed, the pickle would be updated. Pickle loading would be much faster than pure ZCML loading, since no XML-parsing, value conversion and schema validation would be necessary. On the other hand, ZCML creates actions that are eventually executed. Actions are created by executing the directive handlers. Thus the optimization in this approach would be even greater. The problem with this approach is that not all directives are easily pickable. Directive handlers often create new types/classes on the fly. This problem could be solved by ensuring that directives only create pickable actions. Clearly, this would require a lot more work, since you would have to go through all directives to ensure their pickability and also provide fallbacks for 3rd-party directives. Overall, I believe that the performance of ZCML could be improved drastically either way and that the benefit would be immediately visible for anyone using ZCML, including Zope 2 developers using Zope 3 components via Five. All contributions will be made to the Zope 3 SVN core and will be licensed under ZPL 2.1, a GPL-compatible license. If truly completed by September 1, 2006 all features will be part of the Zope 3.4 and 2.11 release. Special thanks and credit will be given to the Google SoC Program and the Plone Foundation for sponsoring this development. Time Estiamtes -------------- 3 days: Writing a proposal for each feature. Zope 3 uses a proposal-based development process and no feature can be checked into the trunk without an accepted proposal. 2 days: Proposal discussion(s). 7 days: Implementation of the local sites through ZCML proposal. This includes extensive testing and documentation as required by the Zope 3 development process. 15 days: Research and implementation of the ZCML optimizations. This includes extensive testing, documentation and profiling.
_______________________________________________ Zope3-dev mailing list Zope3firstname.lastname@example.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com