Re: [Repoze-dev] Changes to Chameleon compilation system
HI Chris Sunday morning, forgot to reply-all sorry Will forward the other emails now T On Sun, May 10, 2009 at 9:29 AM, Chris McDonough wrote: > On 5/9/09 9:10 PM, Tim Hoffman wrote: >> >> The more I think/understand the way chameleon works the less likely I >> think it is going to work. >> >> As I sent out the app engine sdk know includes a modified _ast module >> so maybe it will work >> (but haven't tried it myself the inclusion was to get genshi working >> in the main), but I think the only possibility will be compiling >> templates into an in memory cache but unless you have a busy sight the >> compile cost might be pretty high, as an app engine will spin down in >> less than 30 sec's of non use. And startup time/spin up time on a new >> request will possibly be very expensive unless the template is only >> compiled as used and then cached and reused again pretty soon. > > I think Malthe's idea is to generate .py files that will can be used during > a deployment (to GAE or anything else really). On GAE, using these .py > files should be no slower than importing and using any other .py file in > your application. In general, the biggest expense of Chameleon is emitting > Python source, the rest is quite fast. > > Is there any reason this isn't cc'ed to the list? > > - C > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
Tres Seaver wrote: > Hanno Schlichting wrote: >> Only having pyc files is also a pain with zope.testing and its eager >> cleanup mode, which would constantly delete these files. > > I think the "keep .pyc files" mode is only for deployment (to GAE or > other platforms which can't compile the templates); it would be > irrelevant for any development mode. The pt to py compilation (or right now the unpickling and unmarshaling of some cache files) is the one giving us problems on GAE. The py to pyc/pyo should be handled by any Python interpreter. If I'm not mistaken there's plenty of options for mass-compiling all py files in a package to pyc/pyo, setuptools binary eggs being one of them. I don't see why chameleon should invent something on its own here. Hanno ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > Chris McDonough wrote: >> On 5/9/09 9:12 AM, Martin Aspeli wrote: >>> It'd be nice if there was a way to name the files that meant people >>> won't accidentally commit a bunch of "compiled" .py files that may then >>> end up conflicting later. >> Maybe there could be a mode to compile template files directly to .pyc for >> folks >> that wanted it. But if it complicated Chameleon at all to do so, I'd much >> rather say to those people "don't check in .pt.py files" than have 500 lines >> of >> source to enforce it. > > The pyc files are dependent on a number of Python settings, the Python > version being one of them. I'd rather have the py files generated, so I > can also generate those up front on my machine and actually ship all of > them inside a distribution. The specific Python interpreter on the > target machine can handle py to pyc/pyo compilation just fine. I don't think Chris was arguing that "direct to .pyc" mode would be the only / default one, just an option for particular deployment scenarios. > Only having pyc files is also a pain with zope.testing and its eager > cleanup mode, which would constantly delete these files. I think the "keep .pyc files" mode is only for deployment (to GAE or other platforms which can't compile the templates); it would be irrelevant for any development mode. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKBcNk+gerLs4ltQ4RAlwtAJ9tR5+kOEjGKOi4bZHD+gU6MPU5NQCgyrEv 1joP4ghULYW56BE9ij/qbSQ= =1NmB -END PGP SIGNATURE- ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
2009/5/9 Hanno Schlichting : > Do we really need to be able to switch this mode via CHAMELEON_STATIC? > I'd be tempted to make the static mode the default and not have it > configurable. The whole compilation process is already quite complicated > and I wouldn't want to increase this by adding more and more switches. True; it could be that it did this by default, and if the ``compiler`` module was available, then it would optimize by compiling at run-time. \malthe ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
Malthe Borch wrote: > As a general improvement (in terms of clarity) and as part of the > efforts of getting Chameleon working on GAE, I've begun reworking the > compilation stage. Yeah! > 3) If the ``CHAMELEON_STATIC`` flag is enabled, compile into > one-method-fits-all-arguments render methods for main template and > each available macro (if required). > > On render: > > 4) If the ``CHAMELEON_STATIC`` flag is enabled, collapse > keyword-arguments into dynamic context (econtext). Do we really need to be able to switch this mode via CHAMELEON_STATIC? I'd be tempted to make the static mode the default and not have it configurable. The whole compilation process is already quite complicated and I wouldn't want to increase this by adding more and more switches. Hanno ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
Chris McDonough wrote: > On 5/9/09 9:12 AM, Martin Aspeli wrote: >> It'd be nice if there was a way to name the files that meant people >> won't accidentally commit a bunch of "compiled" .py files that may then >> end up conflicting later. > > Maybe there could be a mode to compile template files directly to .pyc for > folks > that wanted it. But if it complicated Chameleon at all to do so, I'd much > rather say to those people "don't check in .pt.py files" than have 500 lines > of > source to enforce it. The pyc files are dependent on a number of Python settings, the Python version being one of them. I'd rather have the py files generated, so I can also generate those up front on my machine and actually ship all of them inside a distribution. The specific Python interpreter on the target machine can handle py to pyc/pyo compilation just fine. Only having pyc files is also a pain with zope.testing and its eager cleanup mode, which would constantly delete these files. Hanno ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
On 5/9/09 9:12 AM, Martin Aspeli wrote: > Chris McDonough wrote: >> On 5/9/09 8:50 AM, Malthe Borch wrote: >>> As a general improvement (in terms of clarity) and as part of the >>> efforts of getting Chameleon working on GAE, I've begun reworking the >>> compilation stage. >>> >>> If not explicitly disabled (using ``CHAMELEON_CACHE=0``), template >>> files (ending in .pt) will be compiled with the result written to a >>> Python-module with the same filename (ending in .pt.py); Python will >>> of course attempt to write a byte-code compilation (ending in >>> .pt.pyc), which will guarantee good performance when restarting the >>> system. >>> >>> Then, >>> >>> On template instantiation: >>> >>> 1) If the ``CHAMELEON_EAGER`` flag is enabled, parse template source >>> immediately. >>> 2) Load the template source module. >>> 3) If the ``CHAMELEON_STATIC`` flag is enabled, compile into >>> one-method-fits-all-arguments render methods for main template and >>> each available macro (if required). >>> >>> On render: >>> >>> 4) If the ``CHAMELEON_STATIC`` flag is enabled, collapse >>> keyword-arguments into dynamic context (econtext). >>> >>> On compilation: >>> >>> 5) If the "render call signature" (keyword arguments and macro name) >>> matches an already compiled render-method, return it. >>> >>> Else: >>> >>> 6) Compile template (to source string). >>> 7) Append to template source module. >>> 8) Reload template source module and return render-method. >> I've lost track of what a lot of this means (particularly wrt >> CHAMELEON_STATIC) >> but the stuff I do understand sounds pretty good to me. >> >> At a higher level, let me try to repeat the use case here and tell me if I >> have >> it right: >> >> - On template instantiation (at least in eager mode and unless >> CHAMELEON_CACHE=0), a .py file will be written next to the >> template file being instantiated. >> >> - The Chameleon code that actually performs rendering will >> try to find the .py file via some importy sort of thing. >> >> - Python will write the .pyc file. >> >> - When we want to deploy to GAE or other platforms without >> instantiation/compilation support, we'll just check in >> the .py files (or maybe even the .pyc files) and >> push them up. The instantiation code won't ever need >> to do the compilation step. > > It'd be nice if there was a way to name the files that meant people > won't accidentally commit a bunch of "compiled" .py files that may then > end up conflicting later. Maybe there could be a mode to compile template files directly to .pyc for folks that wanted it. But if it complicated Chameleon at all to do so, I'd much rather say to those people "don't check in .pt.py files" than have 500 lines of source to enforce it. - C ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
2009/5/9 Martin Aspeli : > We can't go straight to .pyc? Won't GAE execute .pyc files? Who knows. By the way, can someone confirm if the GAE supports the ``execfile`` built-in? It doesn't seem to provide the ``imp`` module which we currently use to load and compile template source code. > Also see the note about the _ast module earlier today. As far as I'm aware, the ``_ast`` module is the new Python 2.6 one which merely provides a much improved AST. It won't do compilation. \malthe ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
Chris McDonough wrote: > On 5/9/09 8:50 AM, Malthe Borch wrote: >> As a general improvement (in terms of clarity) and as part of the >> efforts of getting Chameleon working on GAE, I've begun reworking the >> compilation stage. >> >> If not explicitly disabled (using ``CHAMELEON_CACHE=0``), template >> files (ending in .pt) will be compiled with the result written to a >> Python-module with the same filename (ending in .pt.py); Python will >> of course attempt to write a byte-code compilation (ending in >> .pt.pyc), which will guarantee good performance when restarting the >> system. >> >> Then, >> >> On template instantiation: >> >> 1) If the ``CHAMELEON_EAGER`` flag is enabled, parse template source >> immediately. >> 2) Load the template source module. >> 3) If the ``CHAMELEON_STATIC`` flag is enabled, compile into >> one-method-fits-all-arguments render methods for main template and >> each available macro (if required). >> >> On render: >> >> 4) If the ``CHAMELEON_STATIC`` flag is enabled, collapse >> keyword-arguments into dynamic context (econtext). >> >> On compilation: >> >> 5) If the "render call signature" (keyword arguments and macro name) >> matches an already compiled render-method, return it. >> >> Else: >> >> 6) Compile template (to source string). >> 7) Append to template source module. >> 8) Reload template source module and return render-method. > > I've lost track of what a lot of this means (particularly wrt > CHAMELEON_STATIC) > but the stuff I do understand sounds pretty good to me. > > At a higher level, let me try to repeat the use case here and tell me if I > have > it right: > > - On template instantiation (at least in eager mode and unless >CHAMELEON_CACHE=0), a .py file will be written next to the >template file being instantiated. > > - The Chameleon code that actually performs rendering will >try to find the .py file via some importy sort of thing. > > - Python will write the .pyc file. > > - When we want to deploy to GAE or other platforms without >instantiation/compilation support, we'll just check in >the .py files (or maybe even the .pyc files) and >push them up. The instantiation code won't ever need >to do the compilation step. It'd be nice if there was a way to name the files that meant people won't accidentally commit a bunch of "compiled" .py files that may then end up conflicting later. Already, people are checking in the .cache files, but at least then you can do an svn ignore filter on *.cache. You can't ignore *.py. :) I guess you can ignore *.pt.py, but I suspect people won't understand this. We can't go straight to .pyc? Won't GAE execute .pyc files? Also see the note about the _ast module earlier today. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] Changes to Chameleon compilation system
On 5/9/09 8:50 AM, Malthe Borch wrote: > As a general improvement (in terms of clarity) and as part of the > efforts of getting Chameleon working on GAE, I've begun reworking the > compilation stage. > > If not explicitly disabled (using ``CHAMELEON_CACHE=0``), template > files (ending in .pt) will be compiled with the result written to a > Python-module with the same filename (ending in .pt.py); Python will > of course attempt to write a byte-code compilation (ending in > .pt.pyc), which will guarantee good performance when restarting the > system. > > Then, > > On template instantiation: > > 1) If the ``CHAMELEON_EAGER`` flag is enabled, parse template source > immediately. > 2) Load the template source module. > 3) If the ``CHAMELEON_STATIC`` flag is enabled, compile into > one-method-fits-all-arguments render methods for main template and > each available macro (if required). > > On render: > > 4) If the ``CHAMELEON_STATIC`` flag is enabled, collapse > keyword-arguments into dynamic context (econtext). > > On compilation: > > 5) If the "render call signature" (keyword arguments and macro name) > matches an already compiled render-method, return it. > > Else: > > 6) Compile template (to source string). > 7) Append to template source module. > 8) Reload template source module and return render-method. I've lost track of what a lot of this means (particularly wrt CHAMELEON_STATIC) but the stuff I do understand sounds pretty good to me. At a higher level, let me try to repeat the use case here and tell me if I have it right: - On template instantiation (at least in eager mode and unless CHAMELEON_CACHE=0), a .py file will be written next to the template file being instantiated. - The Chameleon code that actually performs rendering will try to find the .py file via some importy sort of thing. - Python will write the .pyc file. - When we want to deploy to GAE or other platforms without instantiation/compilation support, we'll just check in the .py files (or maybe even the .pyc files) and push them up. The instantiation code won't ever need to do the compilation step. Is that close to right? - C ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] Changes to Chameleon compilation system
As a general improvement (in terms of clarity) and as part of the efforts of getting Chameleon working on GAE, I've begun reworking the compilation stage. If not explicitly disabled (using ``CHAMELEON_CACHE=0``), template files (ending in .pt) will be compiled with the result written to a Python-module with the same filename (ending in .pt.py); Python will of course attempt to write a byte-code compilation (ending in .pt.pyc), which will guarantee good performance when restarting the system. Then, On template instantiation: 1) If the ``CHAMELEON_EAGER`` flag is enabled, parse template source immediately. 2) Load the template source module. 3) If the ``CHAMELEON_STATIC`` flag is enabled, compile into one-method-fits-all-arguments render methods for main template and each available macro (if required). On render: 4) If the ``CHAMELEON_STATIC`` flag is enabled, collapse keyword-arguments into dynamic context (econtext). On compilation: 5) If the "render call signature" (keyword arguments and macro name) matches an already compiled render-method, return it. Else: 6) Compile template (to source string). 7) Append to template source module. 8) Reload template source module and return render-method. \malthe ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
