Re: [Python-Dev] Pydoc Improvements / Rewrite
On Friday 05 January 2007 02:49, Talin wrote: One issue that needs to be worked out, however, is the division of responsibility between markup processor and output formatter. Does a __markup__ plugin do both jobs, or does it just do parsing, and leave the formatting of output to the appropriate HTML / text output module? How does the HTML output module know how to handle non-standard metadata? There's already __docformat__; see: http://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format -Fred -- Fred L. Drake, Jr. fdrake at acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
Ron, Thanks for your detailed answer. I inserted comments below. 2007/1/5, Ron Adam [EMAIL PROTECTED]: Laurent Gautier wrote: [cut] Introspection is probably already available in the separate module 'inspect', and what a code pydoc would have to do is model the documentation (as a tree) and offer convenience function to navigate the data. Beside that, there would be sub-modules for the different viewers for the documentation data - the interactive console being just one of the viewers. Pydoc currently uses functions from the inspect module along with directly accessing attributes and other information where it is available. It's not a replacement for the inspect module. My first attempt used an xml tree to store the information, but to make that work requires also storing a fair amount of meta information in the tree as well. I found parsing the tree and the meta information to be more complex than using an objective approach which is (to me) more readable and easier to extend. But if you want to try it again, please do. You may come up with something far better than I did. Well, I was coining the idea from the understanding that the main split was console viewer vs other viewer. I was thinking of something a design along the lines of the Model-View-Presenter pattern and I guess that you will have to read your code if I want to debate on that ;-). Finally, I would suspect that an API-breaking modification of the module would need time to be accepted. I was considering this for python 3.0, but one of the developers suggested it would be nice to have in python 2.6 and to move the discussion here. I think any API issues could be worked out. Are there any programs you know of, (yours?), that import pydoc besides the python console? What I did barely qualifies as a hack for my own usage -it won't count-. From the top of my head, there might be ipython (the excellent interactive console) is possibly using pydoc (in any case, I would say that the authors would be interested in developments with pydoc) Otherwise a quick search lead to: - cgitb (!? - it seems that the HTML formatting functions of pydoc are only in use - wouldn't these functions belong more naturally to cgi ?) - DocXMLRPCServer (hey ! it looks like kind-of what I was needing !!!). - happydoc (reportedly having problems with python 2.4 - I am not sure that it is maintained) cgitb and DocXMLRPCServer are both distributed bundled with Python. cgitb seems to be mostly using HTML formatting helpers (and that would suggest the need for an HTML-rendering module - may be for a future improvement, a first step would be separate the rendering/viewing from extraction and modeling of documentation data). DocXMLRPCServer looks (at first sight), like a viewer that would be bundled with pydoc as a sub-module (i.e., module in a package). [cut: Ka-Ping Yee [EMAIL PROTECTED] is now in the loop] Pydoc is a fairly complex program and it would definitely help if others took a look at various parts and made contributions and or suggestions to making it better. Well, I stumbled upon your recent posts in python-ideas (that I tracked up the one in python-devel) because I looked into it I thought that it would be a *lot* of work for one person. (more on that in the next inlined comment) I may have also gotten a bit over my head, but I'm willing to stick it out and try to get it finished with any suggestions (and help) that any one is willing to give me. There are also too many important issues for me to be decided, so this isn't something that can be done in isolation. The download link again is: http://ronadam.com/dl/_pydoc.zip I would be willing to help out, as probably others will as well (I found blogs and posts of people discussing pydoc, it might be worthwhile dropping a line to the people - we can discuss that off-list if you wish), but may be at one condition. I do not think it will work as a zip file shuttled around (in my experience). A versioning system would be extremely helpful (SVN, or CVS. would come to my mind). Well, if you are ok with having the source tree hosted in a SVN/CVS/alike I am on (opening an account on sourceforge or savannah -for example- would be the next step then, as it can take few days for a project to be approved) It's not fully functional yet, but does run. Some parts like the command line file output options still need to be reimplemented. Some output formatting still needs to be cleaned up, and the MRO tree parsing section still needs to be put back in. I will have a look then. One question that I have at the moment is: * Would it be good to have the KEYWORD and TOPICS info as included data objects or files, and possibly use that to generate the python html (and other) documentation for these? (Instead of the other way around like it is now.) This would eliminate the requirement to install something extra in order for
Re: [Python-Dev] Pydoc Improvements / Rewrite
On Thu, 4 Jan 2007, Talin wrote: One issue that needs to be worked out, however, is the division of responsibility between markup processor and output formatter. Does a __markup__ plugin do both jobs, or does it just do parsing, and leave the formatting of output to the appropriate HTML / text output module? How does the HTML output module know how to handle non-standard metadata? [...] I guess the markup processor has to deliver some kind of DOM tree, which can be rendered either into text or into HTML. CSS can take over from that point on. If the markup processor is going to deliver a tree, let me just point out that it would be a pretty major project to define the format of that tree -- about as large as inventing ReST or any other markup language, except that the design of such an intermediate format has to foresee future changes to the input and be flexible enough to target multiple output formats. The design would also have to tackle the question of whether the intermediate format should contain semantic information (what about cross-references?) and what types of such information should be allowed (e.g. names of modules, arguments, exceptions, Python expressions, etc.) -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
On Friday 05 January 2007 17:40, Gregory P. Smith wrote: Whoever is subscribed to python-dev with a broken corporate autoresponder that sends everyone who posts to the list this useless response multiple times please unsubscribe yourself. Its highly annoying and entirely useless since its not even identifying the list subscriber(s) deserving the blame. Can you determine the original email domain from the Received headers, perhaps? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
2007/1/5, Ka-Ping Yee [EMAIL PROTECTED]: [cut] On the other hand, I've often seen the question of why pydoc does both text and HTML generation instead of generating some intermediate data structure from which both kinds of output are produced. The answer is: I tried it. The result turned out to be longer than I expected and needlessly more complicated than what we have now. It may be that a better job could have been done, but I think there is a rational basis for why it turned out that way. The Python objects themselves already are a data structure containing all of the information we need. I discovered that translating this data structure into another data structure and then producing text or HTML was more work than simply producing text or HTML. With CSS, the last step gets even easier and so the intermediate stage becomes even less necessary. Also, the intermediate step required me to essentially invent an API, and I decided that I trusted the stability of Python's API more than that of some API I invented just for this. Point well taken. This is very sensible. I would still try to keep common-and-presenter-independent component. Rather than a sole distinction console/HTML, I would think of a distinction between stateless and interactive presenters, and still have the console and static HTML as specific presenters. The search functions Ron suggested would be part of that presenter-independent part (and for the refinement, stateless vs. interactive would make sense for performances). The distinction may look like an unnecessary complication... but I would think that it does not have to be complicated, and that the number of practical things it would allow would make it almost necessary (ah ! delusions ;-) ). There a number of python editors/consoles/IDE around, some of which are implemented in python, and having the necessary infrastructute to let implement easily documentation presenters would be very nice. This is not to say that text and HTML generation can't be separated; it's just a caution against attempting to overgeneralize by creating an intermediate format. I'm glad you backed away from XML (or I'd have warned you that processing the XML would be a lot of extra work). Your warning regarding the creation of a n-th data structure is completely agreed upon. I also understand your point about the dangers of overgeneralizing. The inspect module was intended to pull out as much as possible of the extraction functionality that's shared by the text and HTML documentation generators. But pydoc is still big. At the time I was proposing pydoc for addition to the standard library, I didn't want to pollute the top-level module namespace with too many names, so I tried hard to minimize the number of modules. And of course it has grown since then with bits of new functionality and support for new language features in Python. But now if a package is being considered, it makes sense to split out some of the pieces (as you have done), such as the web server, the search function, and the interactive interpreter help prompt. It may even enable pydoc to provide search from the interactive help prompt, which would be a nice feature! The package could contain several modules for ease of maintenance, while still providing a single, convenient command for running pydoc from the Unix prompt. Having things already split by Ron is probably a good starting base (and generalization introduced progressively, if agreed upon). I see that there is debating on the format for documentation strings, may there is as well room for flexibility regarding how the strings are utilized. The search would be not only useful to the python console, but also to other editors, as well as to editor (as well as python programs), as well as to stateless presenters (the case I had was to work on a server (web-hosting) on which I only had FTP access and on which I did not know the python version or the package installed -hey, what about an Ajax-capable HTML viewer for the documentation ?) - Laurent -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
2007/1/5, Ka-Ping Yee [EMAIL PROTECTED]: On Thu, 4 Jan 2007, Talin wrote: One issue that needs to be worked out, however, is the division of responsibility between markup processor and output formatter. Does a __markup__ plugin do both jobs, or does it just do parsing, and leave the formatting of output to the appropriate HTML / text output module? How does the HTML output module know how to handle non-standard metadata? [...] I guess the markup processor has to deliver some kind of DOM tree, which can be rendered either into text or into HTML. CSS can take over from that point on. If the markup processor is going to deliver a tree, let me just point out that it would be a pretty major project to define the format of that tree -- about as large as inventing ReST or any other markup language, except that the design of such an intermediate format has to foresee future changes to the input and be flexible enough to target multiple output formats. The design would also have to tackle the question of whether the intermediate format should contain semantic information (what about cross-references?) and what types of such information should be allowed (e.g. names of modules, arguments, exceptions, Python expressions, etc.) Wouldn't it be conceivable to have the processing of the markup performed by a separate function, that could eventually be overridden/passed as a parameter when specific needs regarding the markup are needed ? L. -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote: Well, if you are ok with having the source tree hosted in a SVN/CVS/alike I am on (opening an account on sourceforge or savannah -for example- would be the next step then, as it can take few days for a project to be approved) The Python SVN repository has a sandbox/ directory that's intended for storing code in development; you could certainly use that. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
2007/1/5, A.M. Kuchling [EMAIL PROTECTED]: On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote: Well, if you are ok with having the source tree hosted in a SVN/CVS/alike I am on (opening an account on sourceforge or savannah -for example- would be the next step then, as it can take few days for a project to be approved) The Python SVN repository has a sandbox/ directory that's intended for storing code in development; you could certainly use that. I suspect that this is aside from the rest of the python source tree. (or I would anticipate peppered emails if the module is broken during its early days -and it will- ). I also suspect that write access is not granted easily (as I know a number of python modules being on sourceforge before being ultimately included in the standard modules). --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
No time to review this now, but I'd just like to say that the 1 thing I'd like to see is support for decent mathematical markup. I think at this point that support for latex markup is the way to achieve this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
Talin wrote: Rather than fixing on a standard markup, I would like to see support for a __markup__ module variable which specifies the specific markup language that is used in that module. Doc processors could inspect that variable and then load the appropriate markup translator. Ideally, a module should be able to specify what *documentation provider* to use. Not everyone wants to stuff everything into docstrings, and, especially if you're building larger components, automatic introspection simply doesn't work very well. fwiw, I have hacks for PythonDoc that monkey-patches inspect to provide virtual docstrings, but it would be nice to have an official API for this. It doesn't have to be much more complicated than: def __inspect__(path, format_hint=None): ... return format, data, subpaths where path is a dotted path to the target object, and format_hint is a preferred format. Why? Because its hard to get everyone to agree on which markup language is best for documentation. I personally think that reStructuredText is not a good choice, because I want to add markup that adds semantic information, whereas reStructuredText deals solely with presentation and visual appearance. And does a rather bad job at that too (the squint if you don't want to see the markup approach is fundamentally flawed), but that's another story for another forum. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
On Friday, January 05, 2007, at 02:30PM, Fredrik Lundh [EMAIL PROTECTED] wrote: Talin wrote: Rather than fixing on a standard markup, I would like to see support for a __markup__ module variable which specifies the specific markup language that is used in that module. Doc processors could inspect that variable and then load the appropriate markup translator. Ideally, a module should be able to specify what *documentation provider* to use. Not everyone wants to stuff everything into docstrings, and, especially if you're building larger components, automatic introspection simply doesn't work very well. +lots on that. This is not only true for larger components but projects like wxpython, pyqt and pyobjc could also use these hooks to add links to the C version of those libraries (I don't know about pyqt but wxpython and pyobjc both have choosen to document the differences with the C library instead of trying to duplicate their work). Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
On Fri, Jan 05, 2007 at 09:12:28PM +0800, Laurent Gautier wrote: I suspect that this is aside from the rest of the python source tree. (or I would anticipate peppered emails if the module is broken during its early days -and it will- ). Correct; it can be browsed at http://svn.python.org/view/sandbox/trunk/ I also suspect that write access is not granted easily (as I know a number of python modules being on sourceforge before being ultimately included in the standard modules). I think most of those projects weren't initially started with the goal of stdlib inclusion. The problems with a separate Sourceforge project are: * fewer people will see the commit messages. * if the project isn't completed, everyone will forget about the SF project in a few year. In sandbox/, at least the code will still be somewhat visible; maybe someone would finish it. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jan 5, 2007, at 9:29 AM, Barry Warsaw wrote: On Jan 5, 2007, at 6:06 AM, Anthony Baxter wrote: On Friday 05 January 2007 17:40, Gregory P. Smith wrote: Whoever is subscribed to python-dev with a broken corporate autoresponder that sends everyone who posts to the list this useless response multiple times please unsubscribe yourself. Its highly annoying and entirely useless since its not even identifying the list subscriber(s) deserving the blame. Can you determine the original email domain from the Received headers, perhaps? I think I found the offending address, which I won't name on this mailing list. I have disabled the address so please let me know if you continue to see the autoresponses so that we can either rejoice or I can re-enable the poor guy and we can all continue to despair. Ah, I didn't get a response spam for that message, and I /did/ get one when I sent a test email to the suspect address. So I think I nailed it. I've also sent a polite wink email to their postmaster in complaint. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) iQCVAwUBRZ5ib3EjvBPtnXfVAQKrSgP/ceak+5odGhVjgippP4Gy+ya9WbY98UJ8 71iICp5eGgvVJeD6paoMZhlHvFDezc7muWycMS43ec51TRn979xJnmSbMLNUeVZa 4HPCmOgMUuNkTz843maQaXyA/xtDcETo2EWCpQa6NuWS0e+TEiEeUeaeiosAPWnI UubWVqtBaMk= =P5FM -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5.1 plans
On Thu, Jan 04, 2007 at 01:47:13PM -0800, Neal Norwitz wrote: We have the buildbots to help with this. According to http://www.python.org/dev/buildbot/trunk/ we do not have a single working XP or Cygwin buildbot right now. Definitely! I only did a really quick review. If you want someone to think about any of these, assign them to me with a priority of 9. I'll discuss with others and see if the fixes should go in to 2.5.1 or not. Bug #1599254 is the most critical for me (possible data loss with mailbox.py); I'll try to fix it for 2.5.1. I've changed its priority but not reassigned it. Should the fix for bug #1552726 (interactive Python polls every 1/10 sec) be applied to 2.5.1? It's already been applied to trunk. The bug was noted by an OLPC person, but I don't think this bug is really very critical for them. Maybe we should ask them. I've bumped the priority as a reminder; if we decide to not apply it to 2.5.1, the bug can be closed. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
Laurent Gautier wrote: Ron, Thanks for your detailed answer. I inserted comments below. You welcome. I think any API issues could be worked out. Are there any programs you know of, (yours?), that import pydoc besides the python console? What I did barely qualifies as a hack for my own usage -it won't count-. It could be these changes will give you a way to do the same thing in a less haskish way. From the top of my head, there might be ipython (the excellent interactive console) is possibly using pydoc (in any case, I would say that the authors would be interested in developments with pydoc) According to the web site, ipython is based on twisted, and is currently still limited to python 2.3 and 2.4. Also, the output of the help() function will not change much so I doubt it would be a problem for them. Otherwise a quick search lead to: - cgitb (!? - it seems that the HTML formatting functions of pydoc are only in use - wouldn't these functions belong more naturally to cgi ?) Thanks!, These html formatting functions still exist or are small enough to move into cgitb, so it will be just a matter of making sure they can be reached. I don't think they will be a problem. - DocXMLRPCServer (hey ! it looks like kind-of what I was needing !!!). Thanks again. This might be better to move into the pydoc package. Any opinions? - happydoc (reportedly having problems with python 2.4 - I am not sure that it is maintained) Happydoc does not import pydoc as far as I could tell, so this won't effect them in any direct way I think. They've pretty much implemented everything from scratch. At worse they would just need to copy the parts from the older version into their distribution. I think you got a false positive on this because pydoc is a substring of happydoc. cgitb and DocXMLRPCServer are both distributed bundled with Python. cgitb seems to be mostly using HTML formatting helpers (and that would suggest the need for an HTML-rendering module - may be for a future improvement, a first step would be separate the rendering/viewing from extraction and modeling of documentation data). Making sure these still work would be a good sub project for someone a little later. (I'll do it if no one else has time or wants to.) I'm trying not to change thing to drastically. If the changes are too big, ie... introducing and altering a lot of other modules. Then this will need to move back to the python-3000 list. DocXMLRPCServer looks (at first sight), like a viewer that would be bundled with pydoc as a sub-module (i.e., module in a package). Yes, that was my thought too. :-) But moving it may need to wait until python-3000. (?) Pydoc is a fairly complex program and it would definitely help if others took a look at various parts and made contributions and or suggestions to making it better. Well, I stumbled upon your recent posts in python-ideas (that I tracked up the one in python-devel) because I looked into it I thought that it would be a *lot* of work for one person. (more on that in the next inlined comment) It was tedious going though the module, but now that it's split up into smaller parts it shouldn't be too difficult. I may have also gotten a bit over my head, but I'm willing to stick it out and try to get it finished with any suggestions (and help) that any one is willing to give me. There are also too many important issues for me to be decided, so this isn't something that can be done in isolation. The download link again is: http://ronadam.com/dl/_pydoc.zip I would be willing to help out, as probably others will as well (I found blogs and posts of people discussing pydoc, it might be worthwhile dropping a line to the people - we can discuss that off-list if you wish), but may be at one condition. I do not think it will work as a zip file shuttled around (in my experience). A versioning system would be extremely helpful (SVN, or CVS. would come to my mind). Well, if you are ok with having the source tree hosted in a SVN/CVS/alike I am on (opening an account on sourceforge or savannah -for example- would be the next step then, as it can take few days for a project to be approved) I don't have any experience with SVN, but it could be an opportunity to learn something new. I have a couple of other projects that could benefit by moving them to SVN or CVS like system. What I intended was to get enough feed back that I could get it to a point where it's 90% done and then upload it as a patch where it could be further polished up. To do that, I first need to verify my design goals are the correct approach. (see my reply to Ka-Ping.) If they are, then it's more a matter of cleaning up what I've already started. If not, then it's a bit (or bunch) more work and would benefit from having others work on it in a more formal way.
[Python-Dev] Rewrite of import in Python source (sans docs) is complete
Finally, after a few months worth of work, I have finally gotten far enough in my import rewrite that I am willing to stick my neck out and say it is semantically complete! You can find it in the sandbox under import_in_py. So, details of this implementation. I implemented PEP 302 importers/loaders for built-in, frozen, extension, .py, and .pyc files along with rewriting the steps __import__ goes through to do an import. I also developed an API for .py/.pyc file handling so that there is a generic filesystem importer/loader and a separate handler for .py/.pyc files. This should allow for (relatively) easy selective overriding of just how .py/.pyc files are stored (e.g., introducing a database backend) or how variants on .py/.pyc files are handled (e.g., Quixote's .ptl format). This code has extensive tests and so I am fairly confident that it does what is expected of an import rewrite. There are actually more lines in the test file than the implementation. There is also a mock implementation used for testing. Was interesting doing this in such a test-driven, XP style of only coding what I needed. I have run this code through the entire regression test suite and that is where you find out subtle differences between this implementation and the built-in import (you can see for yourself with the regrtest.sh shell script). First test_pkg will fail because currently the new import adds a __loader__ attribute on all modules (that might change for security reasons) and test_pkg is an old, stdout comparing test. Second, test_runpy fails because I have not implemented get_code on the filesystem loader which is required by runpy. Both are shallow issues that can be dealt with. Third, and the hardest difference to deal with, is that you will get some warnings that print out that you normally don't see. This is because warnings.warn and its stack_level argument don't have the effect people are used to when importing a deprecated module. Before you could set stack_level to 2 and it would look like it came from the import statement. But now, with import written in Python and thus on the call stack compared to being in C and thus not showing up, two levels back is still in the import code. I really don't know how this should be dealt with short of saying that the rule of thumb with 2 stack levels back for a warning does not work when done at the import level. It is not blazing fast at the moment. Some things, like the built-in and frozen importers/loaders could be rewritten in C without huge issue. I am also sure I have made some stupid design decisions at various points in the code. But there is benchmarking code in the sandbox called importbench and it showed up a 10x speed slowdown on a Linux box I was using in mid to late December when doing a fresh import of certain types (I don't remember exactly which kind off the top of my head). Because of this current slowness I don't know if people want to rush into trying to make this the default import implementation quite yet or if this is not too big of a thing since the common case of pulling out of sys.modules is not that much slower. I know I am currently not planning on devoting the time to bootstrap it in as I have my security work to finish first along with other Python stuff that seems more pressing to me. And since (I think) I don't need to bootstrap it in order to finish my security work I can't justify spending work time on it. But I can rearrange priorities if people really want to pursue this (especially if I can get some help with it). As with the module's name, it is currently named 'importer', but that is bad since it conflicts with the idea of importers from PEP 302. I was thinking importlib, but I wanted to wait and see what other people thought. Don't know if you guys are okay with me checking this in without having it vetted by the community first like we prefer all new modules to do. I have not done the LaTeX docs yet. I think that is all of the details that I can think of. I am still working towards implementing the security needed so that an application that embeds Python can execute arbitrary code securely. Giving a talk at PyCon on the topic for anyone interested. Special thanks needs to go to Paul Moore who I talked to through most of the design of the code. Nick Coghlan also provided some handy feedback. And Neal Norwitz for bugging about wanting something like this done. Plus thanks to everyone who has shown support. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Renaming Include/object.h
Andrea Griffini schrieb: I've a partially related question... why isn't the module structure in an include file .h and is instead in Objects/moduleobject.c ? For the cached lookup optimization I copied the definition but that's surely a bad way to do it I however wondered if there were good reasons for module objects for not being published. I guess it's included in the C file because that's the easiest way to implement it. AFAICT, it has been that way from the beginning. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
Ron Adam wrote: Laurent Gautier wrote: From the top of my head, there might be ipython (the excellent interactive console) is possibly using pydoc (in any case, I would say that the authors would be interested in developments with pydoc) Certainly :) I'd like to ask whether this discussion considers any kind of merging of pydoc with epydoc. Many projects (ipython included) are moving towards epydoc as a more capable system than pydoc, though it would be nice to see its API-doc-generation capabilities be part of the stdlib. I don't know if that's considered either too large or too orthogonal to the current direction. According to the web site, ipython is based on twisted, and is currently still limited to python 2.3 and 2.4. Also, the output of the help() function will not change much so I doubt it would be a problem for them. A few corrections: - IPython's trunk is NOT based on twisted at all, it's a self-contained Python package which depends only on the Python standard library (plus readline support under windows, which we also provide but requires ctypes). - The ipython development branch does use twisted, but only for its distributed and parallel computing capabilities. Eventually when that branch becomes trunk, there will /always/ be a non-twisted component for local, terminal-based work. - The last release (0.7.3) fully supports Python 2.5. In fact, one nasty bug in 2.5 with extremely slow traceback generation was kindly fixed by python-dev in the nick of time after my pestering (an ipython user found it first and brought it to my attention). Otherwise a quick search lead to: - cgitb (!? - it seems that the HTML formatting functions of pydoc are only in use - wouldn't these functions belong more naturally to cgi ?) Thanks!, These html formatting functions still exist or are small enough to move into cgitb, so it will be just a matter of making sure they can be reached. I don't think they will be a problem. If anyone is interested in updating cgitb, you might want to look at ipython's ultratb (which was born as a cgitb port to ANSI terminals): http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/ultraTB.py It contains functionality for generating /extremely/ detailed tracebacks with gobs of local detail. These verbose tracebacks have let me fix many ipython bugs from crash dumps triggered by remote code and libraries I don't have access to, in cases where a normal traceback would have been insufficient. Here's a link to a slightly outdated simple example (the formatting has improved a bit): http://ipython.scipy.org/screenshots/snapshot1.png Obviously the right thing to do would be to separate the ANSI coloring from the structural formatting, so that the traceback could be formatted as HTML, ANSI colors or anything else. That is very much on my todo list, since the network-enabled ipython will have browser-based interfaces in the future. All of ipython is BSD licensed, so any code in there is for the taking. Best, f ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
2007/1/6, Ron Adam [EMAIL PROTECTED]: Laurent Gautier wrote: [cut] I think any API issues could be worked out. Are there any programs you know of, (yours?), that import pydoc besides the python console? What I did barely qualifies as a hack for my own usage -it won't count-. It could be these changes will give you a way to do the same thing in a less haskish way. This precisely why I get myself into the present trouble ;-) From the top of my head, there might be ipython (the excellent interactive console) is possibly using pydoc (in any case, I would say that the authors would be interested in developments with pydoc) According to the web site, ipython is based on twisted, and is currently still limited to python 2.3 and 2.4. Also, the output of the help() function will not change much so I doubt it would be a problem for them. Sorry for answering a bit off-the-question. My meaning was that they would be interested in knowning that pydoc is changing (and would surely have ideas). Otherwise a quick search lead to: - cgitb (!? - it seems that the HTML formatting functions of pydoc are only in use - wouldn't these functions belong more naturally to cgi ?) Thanks!, These html formatting functions still exist or are small enough to move into cgitb, so it will be just a matter of making sure they can be reached. I don't think they will be a problem. I read your comment about having not too many things changed for 2.6. (or that will be bumped to 3000). A suggestion I would have would be to create an html/htmlrender module in the pydoc-package-to-be and start putting all the html formatting function (as they are completely independent of pydoc, as far as I can see, over there). You can then create wrappers to the original functions including a deprecation warning. You can refer to Michael Chermside's recipe for an example of implementation with a deprecation decorator - http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/391367 ) The suggestion above would actually apply to *anything* that is changed in pydoc, providing the benefit of allowing the necessary changes while having a temporary API to provide back-compatibility. - DocXMLRPCServer (hey ! it looks like kind-of what I was needing !!!). Thanks again. This might be better to move into the pydoc package. Any opinions? We both pretty much agree, but now we've got to find the modules depending on DocXMLRPCServer (and hope the recursion won't go on for too long). We would also have to contact the author of DocXMLRPCServer (moving it would go with potential changes, and he would certainly have suggestions about that). - happydoc (reportedly having problems with python 2.4 - I am not sure that it is maintained) Happydoc does not import pydoc as far as I could tell, so this won't effect them in any direct way I think. They've pretty much implemented everything from scratch. At worse they would just need to copy the parts from the older version into their distribution. I think you got a false positive on this because pydoc is a substring of happydoc. Yes this was a false positive. I kept it in the list thinking there might be interesting ideas there as well (but I forget to label it as such - sorry for the confusion). cgitb and DocXMLRPCServer are both distributed bundled with Python. cgitb seems to be mostly using HTML formatting helpers (and that would suggest the need for an HTML-rendering module - may be for a future improvement, a first step would be separate the rendering/viewing from extraction and modeling of documentation data). Making sure these still work would be a good sub project for someone a little later. (I'll do it if no one else has time or wants to.) I'm trying not to change thing to drastically. If the changes are too big, ie... introducing and altering a lot of other modules. Then this will need to move back to the python-3000 list. I agree that altering a number of other modules would be a little tricky to manage for version 2.6, but I would think that the makeover of pydoc would benefit from being made early (back-compatibility being ensured by the deprecation decorator mentioned above, for example). DocXMLRPCServer looks (at first sight), like a viewer that would be bundled with pydoc as a sub-module (i.e., module in a package). Yes, that was my thought too. :-) But moving it may need to wait until python-3000. (?) An other way is to move it, while keeping a dummy module that imports the module from its new location (and issue deprecation warnings). What I fear is that the let's wait for python-3000 correspond to postponing changes to some other time. [cut] I do not think it will work as a zip file shuttled around (in my experience). A versioning system would be extremely helpful (SVN, or CVS. would come to my mind). Well, if you are ok with having the source tree
Re: [Python-Dev] Pydoc Improvements / Rewrite
At 06:35 PM 1/5/2007 -0700, Fernando Perez wrote: Ron Adam wrote: Laurent Gautier wrote: From the top of my head, there might be ipython (the excellent interactive console) is possibly using pydoc (in any case, I would say that the authors would be interested in developments with pydoc) Certainly :) I'd like to ask whether this discussion considers any kind of merging of pydoc with epydoc. Unless there's been a complete rewrite of epydoc since the last time I looked at it, I'd have to give a very strong -1 against epydoc; it has all the problems of pydoc, plus new ones. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pydoc Improvements / Rewrite
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jan 5, 2007, at 9:41 PM, Phillip J. Eby wrote: Unless there's been a complete rewrite of epydoc since the last time I looked at it, I'd have to give a very strong -1 against epydoc; it has all the problems of pydoc, plus new ones. I haven't read this entire thread, so I'll just chime in to say that I've /used/ epydoc and like it quite a bit. I've even hacked on it a little to fix a few things and it didn't seem that bad, though I didn't do any major work on its internals. I've used both the 2.x version and the 3.x version but I haven't used anything in the last, I dunno, 4 or 5 months. Note that I was using it on a heavily embedded/extended application and it did a pretty good job of pulling docs out of C coded docstrings. I had to patch Python a bit here and there (I think most of those fixes are in Python 2.5) and I know that the epydoc guys fixed a few things related to C types (e.g. such as that the tp_doc has to document both the type and the constructor). Probably the biggest issue that I remember was needing to invoke it programmatically, which was an absolute requirement for us, since none of the extension modules were importable unless epydoc was run inside the embedded environment. I got it to work, but it was a bit of a pain. If you've already explained it, that's fine, but if not, could you outline what you have against epydoc? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) iQCVAwUBRZ8ww3EjvBPtnXfVAQJmnwQAnTn7W7Nri5Q+pSPmTLaIvqnmRJWDegpF HSIDY8nBcMwsST76gzpwt02GikWtyy4gujgHiAEyr4/eQyJsMcnXMptkgXixtuPz wA2pJkeo87eorPBMtOMoB9XpoyUkQTh5W/lGnR3rOinZPeiqJFEzc//DIJV+H/p7 Iqrie+FVnis= =sNBX -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
