Re: [Python-Dev] Pydoc Improvements / Rewrite

[Fred L. Drake, Jr.]

On Friday 05 January 2007 02:49, Talin wrote:
 > One issue that needs to be worked out, however, is the division of
 > responsibility between markup processor and output formatter. Does a
 > __markup__ plugin do both jobs, or does it just do parsing, and leave
 > the formatting of output to the appropriate HTML / text output module?
 > How does the HTML output module know how to handle non-standard metadata?

There's already __docformat__; see:

  http://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format


  -Fred

-- 
Fred L. Drake, Jr.   
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Laurent Gautier]

Ron,

Thanks for your detailed answer.
I inserted comments below.

2007/1/5, Ron Adam <[EMAIL PROTECTED]>:
> Laurent Gautier wrote:
[cut]
>
> > Introspection is probably already available in the separate module
> > 'inspect',
> > and what a code pydoc would have to do is model the documentation
> > (as a tree) and offer convenience function to navigate the data. Beside
> > that,
> > there would be sub-modules for the different viewers for the documentation
> > data - the interactive console being just one of the viewers.
>
> Pydoc currently uses functions from the inspect module along with directly
> accessing attributes and other information where it is available.  It's not a
> replacement for the inspect module.
>
> My first attempt used an xml tree to store the information, but to make that
> work requires also storing a fair amount of meta information in the tree as
> well.  I found parsing the tree and the meta information to be more complex 
> than
> using an objective approach which is (to me) more readable and easier to 
> extend.
>   But if you want to try it again, please do.  You may come up with something
> far better than I did.

Well, I was coining the idea from the understanding that the main split was
console viewer vs other viewer. I was thinking of something a design
along the lines of the Model-View-Presenter pattern and I guess
that you will have to
read your code if I want to debate on that ;-).

> > Finally, I would suspect that an API-breaking modification of the module
> > would
> > need time to be accepted.
>
> I was considering this for python 3.0, but one of the developers suggested it
> would be nice to have in python 2.6 and to move the discussion here.
>
> I think any API issues could be worked out.  Are there any programs you know 
> of,
> (yours?), that import pydoc besides the python console?

What I did barely qualifies as a hack for my own usage -it won't count-.

>From the top of my head, there might be  "ipython" (the excellent
interactive console) is possibly using pydoc
(in any case, I would say that the authors would be interested in
developments with pydoc)

Otherwise a quick search lead to:
- "cgitb" (!? - it seems that the HTML formatting functions of pydoc
are only in use - wouldn't these functions belong more naturally to "cgi" ?)
- "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).
- "happydoc" (reportedly having problems with python 2.4 - I am not
sure that it is maintained)

"cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.

"cgitb" seems to be mostly using HTML formatting helpers (and that would
suggest the need for an HTML-rendering module - may be for a future
improvement,  a first step would be separate the rendering/viewing
from extraction and modeling of documentation data).

"DocXMLRPCServer" looks (at first sight), like a viewer that would be bundled
with pydoc as a sub-module (i.e., module in a package).


>
[cut:  Ka-Ping Yee <[EMAIL PROTECTED]> is now in the loop]
>

> Pydoc is a fairly complex program and it would definitely help if others took 
> a
> look at various parts and made contributions and or suggestions to making it 
> better.

Well, I stumbled upon your recent posts in python-ideas (that I tracked up the
one in python-devel) because I looked into it I thought that it would be
a *lot* of work for one person.
(more on that in the next inlined comment)

> I may have also gotten a bit over my head, but I'm willing to stick it out and
> try to get it finished with any suggestions (and help) that any one is willing
> to give me.  There are also too many important issues for me to be decided, so
> this isn't something that can be done in isolation.
>
> The download link again is:
>
>  http://ronadam.com/dl/_pydoc.zip

I would be willing to help out, as probably others will as well (I
found blogs and
posts of people discussing pydoc, it might be worthwhile dropping a line to
the people - we can discuss that off-list if you wish), but may be at
one condition.

I do not think it will work as a zip file shuttled around (in my experience).
A versioning system would be extremely helpful (SVN, or CVS. would
come to my mind).
Well, if you are ok with having the source tree hosted in a
SVN/CVS/alike I am on
(opening an account on sourceforge or savannah -for example- would be
the next step then, as it can take few days for a project to be
approved)

> It's not fully functional yet, but does run. Some parts like the command line
> file output options still need to be reimplemented. Some output formatting 
> still
> needs to be cleaned up, and the MRO tree parsing section still needs to be put
> back in.

I will have a look then.

>
> One question that I have at the moment is:
>
> * Would it be good to have the "KEYWORD" and "TOPICS" info as included data
> objects or files, and possibly use that to generate the python html (and 
> other)
> documentation for these?  (Instead of the other way around like

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Ka-Ping Yee]

On Thu, 4 Jan 2007, Talin wrote:
> One issue that needs to be worked out, however, is the division of
> responsibility between markup processor and output formatter. Does a
> __markup__ plugin do both jobs, or does it just do parsing, and leave
> the formatting of output to the appropriate HTML / text output module?
> How does the HTML output module know how to handle non-standard metadata?
[...]
> I guess the markup processor has to deliver some kind of DOM tree, which
> can be rendered either into text or into HTML. CSS can take over from
> that point on.

If the markup processor is going to deliver a tree, let me just point
out that it would be a pretty major project to define the format of
that tree -- about as large as inventing ReST or any other markup
language, except that the design of such an intermediate format has
to foresee future changes to the input and be flexible enough to target
multiple output formats.  The design would also have to tackle the
question of whether the intermediate format should contain semantic
information (what about cross-references?) and what types of such
information should be allowed (e.g. names of modules, arguments,
exceptions, Python expressions, etc.)


-- ?!ng
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder

[Anthony Baxter]

On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
> Whoever is subscribed to python-dev with a broken corporate
> autoresponder that sends everyone who posts to the list this
> useless response multiple times please unsubscribe yourself.  Its
> highly annoying and entirely useless since its not even
> identifying the list subscriber(s) deserving the blame.

Can you determine the original email domain from the Received 
headers, perhaps?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Laurent Gautier]

2007/1/5, Ka-Ping Yee <[EMAIL PROTECTED]>:
[cut]
> On the other hand, I've often seen the question of why pydoc does
> both text and HTML generation instead of generating some intermediate
> data structure from which both kinds of output are produced.  The
> answer is: I tried it.  The result turned out to be longer than
> I expected and needlessly more complicated than what we have now.
> It may be that a better job could have been done, but I think there
> is a rational basis for why it turned out that way.
>
> The Python objects themselves already are a data structure containing
> all of the information we need.  I discovered that translating this
> data structure into another data structure and then producing text
> or HTML was more work than simply producing text or HTML.  With CSS,
> the last step gets even easier and so the intermediate stage becomes
> even less necessary.  Also, the intermediate step required me to
> essentially invent an API, and I decided that I trusted the stability
> of Python's API more than that of some API I invented just for this.

Point well taken. This is very sensible.

I would still try to keep common-and-presenter-independent component.
Rather than a sole distinction console/HTML, I would think of a distinction
between stateless and interactive presenters, and still have the console and
static HTML as specific presenters. The search functions Ron suggested
would be part of that presenter-independent part (and for the refinement,
stateless vs. interactive would make sense for performances).

The distinction may look like an unnecessary complication... but I would
think that it does not have to be complicated, and that the number of practical
things it would allow would make it almost necessary (ah ! delusions ;-) ).
There a number of python editors/consoles/IDE around, some of which
are implemented in python, and having the necessary infrastructute to
let implement
easily documentation presenters would be very nice.

> This is not to say that text and HTML generation can't be separated;
> it's just a caution against attempting to overgeneralize by creating
> an intermediate format.  I'm glad you backed away from XML (or I'd
> have warned you that processing the XML would be a lot of extra work).

Your warning regarding the creation of  a n-th data structure is completely
agreed upon. I also understand your point about the dangers of
overgeneralizing.

> The inspect module was intended to pull out as much as possible of
> the extraction functionality that's shared by the text and HTML
> documentation generators.  But pydoc is still big.  At the time I was
> proposing pydoc for addition to the standard library, I didn't want
> to pollute the top-level module namespace with too many names, so I
> tried hard to minimize the number of modules.  And of course it has
> grown since then with bits of new functionality and support for new
> language features in Python.
>
> But now if a package is being considered, it makes sense to split
> out some of the pieces (as you have done), such as the web server,
> the search function, and the interactive interpreter help prompt.
> It may even enable pydoc to provide search from the interactive help
> prompt, which would be a nice feature!  The package could contain
> several modules for ease of maintenance, while still providing a
> single, convenient command for running pydoc from the Unix prompt.

Having things already split by Ron is probably a good starting base
(and generalization introduced progressively, if agreed upon).

I see that there is debating on the format for documentation strings,
may there is as well room for flexibility regarding how the strings are
utilized.

The search would be not only useful to the python console, but also to
other editors, as well as to editor (as well as python programs),
as well as to stateless presenters (the case I had was to work on a
server (web-hosting) on which I only had FTP access and on which I did
not know the python version or the package installed -hey, what about
an Ajax-capable HTML viewer for the documentation ?) -



Laurent


>
> -- ?!ng
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Laurent Gautier]

2007/1/5, Ka-Ping Yee <[EMAIL PROTECTED]>:
> On Thu, 4 Jan 2007, Talin wrote:
> > One issue that needs to be worked out, however, is the division of
> > responsibility between markup processor and output formatter. Does a
> > __markup__ plugin do both jobs, or does it just do parsing, and leave
> > the formatting of output to the appropriate HTML / text output module?
> > How does the HTML output module know how to handle non-standard metadata?
> [...]
> > I guess the markup processor has to deliver some kind of DOM tree, which
> > can be rendered either into text or into HTML. CSS can take over from
> > that point on.
>
> If the markup processor is going to deliver a tree, let me just point
> out that it would be a pretty major project to define the format of
> that tree -- about as large as inventing ReST or any other markup
> language, except that the design of such an intermediate format has
> to foresee future changes to the input and be flexible enough to target
> multiple output formats.  The design would also have to tackle the
> question of whether the intermediate format should contain semantic
> information (what about cross-references?) and what types of such
> information should be allowed (e.g. names of modules, arguments,
> exceptions, Python expressions, etc.)
>

Wouldn't it be conceivable to have the processing of the markup
performed by a separate function, that could eventually be overridden/passed
as a parameter  when specific needs regarding the markup are needed  ?


L.

> -- ?!ng
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[A.M. Kuchling]

On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote:
> Well, if you are ok with having the source tree hosted in a
> SVN/CVS/alike I am on
> (opening an account on sourceforge or savannah -for example- would be
> the next step then, as it can take few days for a project to be
> approved)

The Python SVN repository has a sandbox/ directory that's intended for
storing code in development; you could certainly use that.

--amk

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Laurent Gautier]

2007/1/5, A.M. Kuchling <[EMAIL PROTECTED]>:
> On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote:
> > Well, if you are ok with having the source tree hosted in a
> > SVN/CVS/alike I am on
> > (opening an account on sourceforge or savannah -for example- would be
> > the next step then, as it can take few days for a project to be
> > approved)
>
> The Python SVN repository has a sandbox/ directory that's intended for
> storing code in development; you could certainly use that.

I suspect that this is aside from the rest of the python source tree.
(or I would anticipate peppered emails if the module is broken during
its early days
-and it will- ).

I also suspect that write access is not granted easily (as I know a
number of python modules being on sourceforge before being ultimately
included in
the standard modules).

> --amk
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Neal Becker]

No time to review this now, but I'd just like to say that the 1 thing I'd
like to see is support for decent mathematical markup.  I think at this
point that support for latex markup is the way to achieve this.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Fredrik Lundh]

Talin wrote:

> Rather than fixing on a standard markup, I would like to see support for
> a __markup__ module variable which specifies the specific markup
> language that is used in that module. Doc processors could inspect that
> variable and then load the appropriate markup translator.

Ideally, a module should be able to specify what *documentation provider*
to use.  Not everyone wants to stuff everything into docstrings, and, especially
if you're building larger components, automatic introspection simply doesn't
work very well.

fwiw, I have hacks for PythonDoc that monkey-patches "inspect" to provide
"virtual docstrings", but it would be nice to have an official API for this.  It
doesn't have to be much more complicated than:

def __inspect__(path, format_hint=None):
...
return format, data, subpaths

where path is a dotted path to the target object, and format_hint is a preferred
format.

> Why? Because its hard to get everyone to agree on which markup language
> is best for documentation. I personally think that reStructuredText is
> not a good choice, because I want to add markup that adds semantic
> information, whereas reStructuredText deals solely with presentation and
> visual appearance.

And does a rather bad job at that too (the "squint if you don't want to see the
markup" approach is fundamentally flawed), but that's another story for another
forum.

 



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Ronald Oussoren]

 
On Friday, January 05, 2007, at 02:30PM, "Fredrik Lundh" <[EMAIL PROTECTED]> 
wrote:
>Talin wrote:
>
>> Rather than fixing on a standard markup, I would like to see support for
>> a __markup__ module variable which specifies the specific markup
>> language that is used in that module. Doc processors could inspect that
>> variable and then load the appropriate markup translator.
>
>Ideally, a module should be able to specify what *documentation provider*
>to use.  Not everyone wants to stuff everything into docstrings, and, 
>especially
>if you're building larger components, automatic introspection simply doesn't
>work very well.

+lots on that. This is not only true for larger components but projects like 
wxpython, pyqt and pyobjc could also use these hooks to add links to the C 
version of those libraries (I don't know about pyqt but wxpython and pyobjc 
both have choosen to document the differences with the C library instead of 
trying to duplicate their work).

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder

[Barry Warsaw]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Jan 5, 2007, at 6:06 AM, Anthony Baxter wrote:

> On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
>> Whoever is subscribed to python-dev with a broken corporate
>> autoresponder that sends everyone who posts to the list this
>> useless response multiple times please unsubscribe yourself.  Its
>> highly annoying and entirely useless since its not even
>> identifying the list subscriber(s) deserving the blame.
>
> Can you determine the original email domain from the Received
> headers, perhaps?

I think I found the offending address, which I won't name on this  
mailing list.  I have disabled the address so please let me know if  
you continue to see the autoresponses so that we can either rejoice  
or I can re-enable the poor guy and we can all continue to despair.

- -Barry

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ5gznEjvBPtnXfVAQImwwP8DDnOUXWDbwN0ZnHsAHj18qxAUDEL7sH/
JfXUH74RYIp2h/YGKUqY1WCEtcqRaTeKpqh2TZLChZkAsXcSEU9r8cwJY9ReUOkE
13JZRy4xh52wlnF5G6gChUBSAueks74/OWsk8XYO46bsc1OGSsMQB3ee/9LI57DF
cpLJCPW4jKw=
=aPTq
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[A.M. Kuchling]

On Fri, Jan 05, 2007 at 09:12:28PM +0800, Laurent Gautier wrote:
> I suspect that this is aside from the rest of the python source tree.
> (or I would anticipate peppered emails if the module is broken during
> its early days
> -and it will- ).

Correct; it can be browsed at 

> I also suspect that write access is not granted easily (as I know a
> number of python modules being on sourceforge before being ultimately
> included in
> the standard modules).

I think most of those projects weren't initially started with the goal
of stdlib inclusion.  The problems with a separate Sourceforge project are:

* fewer people will see the commit messages.
* if the project isn't completed, everyone will forget about the SF 
  project in a few year.  In sandbox/, at least the code will still be
  somewhat visible; maybe someone would finish it.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder

[Barry Warsaw]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Jan 5, 2007, at 9:29 AM, Barry Warsaw wrote:

> On Jan 5, 2007, at 6:06 AM, Anthony Baxter wrote:
>
>> On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
>>> Whoever is subscribed to python-dev with a broken corporate
>>> autoresponder that sends everyone who posts to the list this
>>> useless response multiple times please unsubscribe yourself.  Its
>>> highly annoying and entirely useless since its not even
>>> identifying the list subscriber(s) deserving the blame.
>>
>> Can you determine the original email domain from the Received
>> headers, perhaps?
>
> I think I found the offending address, which I won't name on this
> mailing list.  I have disabled the address so please let me know if
> you continue to see the autoresponses so that we can either rejoice
> or I can re-enable the poor guy and we can all continue to despair.

Ah, I didn't get a response spam for that message, and I /did/ get  
one when I sent a test email to the suspect address.  So I think I  
nailed it.  I've also sent a polite  email to their postmaster  
in complaint.

- -Barry

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ5ib3EjvBPtnXfVAQKrSgP/ceak+5odGhVjgippP4Gy+ya9WbY98UJ8
71iICp5eGgvVJeD6paoMZhlHvFDezc7muWycMS43ec51TRn979xJnmSbMLNUeVZa
4HPCmOgMUuNkTz843maQaXyA/xtDcETo2EWCpQa6NuWS0e+TEiEeUeaeiosAPWnI
UubWVqtBaMk=
=P5FM
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5.1 plans

[A.M. Kuchling]

On Thu, Jan 04, 2007 at 01:47:13PM -0800, Neal Norwitz wrote:
> We have the buildbots to help with this.

According to http://www.python.org/dev/buildbot/trunk/ we do not have
a single working XP or Cygwin buildbot right now.

> Definitely!  I only did a really quick review.  If you want someone to
> think about any of these, assign them to me with a priority of 9.
> I'll discuss with others and see if the fixes should go in to 2.5.1 or
> not.

Bug #1599254 is the most critical for me (possible data loss with
mailbox.py); I'll try to fix it for 2.5.1.  I've changed its priority
but not reassigned it.

Should the fix for bug #1552726 (interactive Python polls every 1/10
sec) be applied to 2.5.1?  It's already been applied to trunk.  The
bug was noted by an OLPC person, but I don't think this bug is really
very critical for them.  Maybe we should ask them.  I've bumped the
priority as a reminder; if we decide to not apply it to 2.5.1, the bug
can be closed.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Ron Adam]

Laurent Gautier wrote:
 > Ron,
 >
 > Thanks for your detailed answer.
 > I inserted comments below.

You welcome.

 >> I think any API issues could be worked out.  Are there any programs
 >> you know of,
 >> (yours?), that import pydoc besides the python console?
 >
 > What I did barely qualifies as a hack for my own usage -it won't count-.

It could be these changes will give you a way to do the same thing in a less 
haskish way.


 > From the top of my head, there might be  "ipython" (the excellent
 > interactive console) is possibly using pydoc
 > (in any case, I would say that the authors would be interested in
 > developments with pydoc)

According to the web site, ipython is based on twisted, and is currently still 
limited to python 2.3 and 2.4.  Also, the output of the help() function will 
not 
change much so I doubt it would be a problem for them.

 > Otherwise a quick search lead to:
 > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
 > are only in use - wouldn't these functions belong more naturally to
 > "cgi" ?)

Thanks!, These html formatting functions still exist or are small enough to 
move 
into cgitb, so it will be just a matter of making sure they can be reached.  I 
don't think they will be a problem.


 > - "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).

Thanks again.  This might be better to move into the pydoc package.  Any 
opinions?


 > - "happydoc" (reportedly having problems with python 2.4 - I am not
 > sure that it is maintained)

Happydoc does not import pydoc as far as I could tell, so this won't effect 
them 
in any direct way I think.  They've pretty much implemented everything from 
scratch.  At worse they would just need to copy the parts from the older 
version 
into their distribution.

I think you got a false positive on this because pydoc is a substring of 
happydoc.


 > "cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.
 >
 > "cgitb" seems to be mostly using HTML formatting helpers (and that would
 > suggest the need for an HTML-rendering module - may be for a future
 > improvement,  a first step would be separate the rendering/viewing
 > from extraction and modeling of documentation data).

Making sure these still work would be a good sub project for someone a little 
later.  (I'll do it if no one else has time or wants to.)  I'm trying not to 
change thing to drastically.  If the changes are too big, ie... introducing and 
altering a lot of other modules.  Then this will need to move back to the 
python-3000 list.


 > "DocXMLRPCServer" looks (at first sight), like a viewer that would be
 > bundled
 > with pydoc as a sub-module (i.e., module in a package).

Yes, that was my thought too. :-)

But moving it may need to wait until python-3000.  (?)


 >> Pydoc is a fairly complex program and it would definitely help if
 >> others took a
 >> look at various parts and made contributions and or suggestions to
 >> making it better.
 >
 > Well, I stumbled upon your recent posts in python-ideas (that I tracked
 > up the
 > one in python-devel) because I looked into it I thought that it would be
 > a *lot* of work for one person.
 > (more on that in the next inlined comment)

It was tedious going though the module, but now that it's split up into smaller 
parts it shouldn't be too difficult.


 >> I may have also gotten a bit over my head, but I'm willing to stick it
 >> out and
 >> try to get it finished with any suggestions (and help) that any one is
 >> willing
 >> to give me.  There are also too many important issues for me to be
 >> decided, so
 >> this isn't something that can be done in isolation.
 >>
 >> The download link again is:
 >>
 >>  http://ronadam.com/dl/_pydoc.zip
 >
 > I would be willing to help out, as probably others will as well (I
 > found blogs and
 > posts of people discussing pydoc, it might be worthwhile dropping a line to
 > the people - we can discuss that off-list if you wish), but may be at
 > one condition.
 >
 > I do not think it will work as a zip file shuttled around (in my
 > experience).
 > A versioning system would be extremely helpful (SVN, or CVS. would
 > come to my mind).
 > Well, if you are ok with having the source tree hosted in a
 > SVN/CVS/alike I am on
 > (opening an account on sourceforge or savannah -for example- would be
 > the next step then, as it can take few days for a project to be
 > approved)

I don't have any experience with SVN, but it could be an opportunity to learn 
something new.  I have a couple of other projects that could benefit by moving 
them to SVN or CVS like system.

What I intended was to get enough feed back that I could get it to a point 
where 
it's 90% done and then upload it as a patch where it could be further polished 
up.  To do that, I first need to verify my design goals are the correct 
approach. (see my reply to Ka-Ping.)  If they are, then it's more a matter of 
cleaning up what I've already started.  If not, then it'

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Ron Adam]

Talin wrote:
> Ron Adam wrote:
>> Larry Hastings wrote:
>>> For those of us without eidetic memories, PEP 287 is "use 
>>> reStructuredText for docstrings":
>>> http://www.python.org/dev/peps/pep-0287/
>> Thanks for the link. PEP 287 looks to be fairly general in that it expresses 
>> a 
>> general desire rather than a specification.
> 
> Apologies for the digression, but I have a comment on this.
> 
> Rather than fixing on a standard markup, I would like to see support for 
> a __markup__ module variable which specifies the specific markup 
> language that is used in that module. Doc processors could inspect that 
> variable and then load the appropriate markup translator.
> 
> Why? Because its hard to get everyone to agree on which markup language 
> is best for documentation. I personally think that reStructuredText is 
> not a good choice, because I want to add markup that adds semantic 
> information, whereas reStructuredText deals solely with presentation and 
> visual appearance. (In other words, I'd like to be able to define 
> machine-readable metadata that identifies parameters, return values, and 
> exceptions -- not just hyperlinks and text styles.) Having used a lot of 
> different documentation markup languages, and written a few of them, I 
> prefer "non-invasive" semantic markup as seen in markup processors such 
> as Doc-o-matic and NaturalDocs. (By non-invasive, I mean that the markup 
> doesn't detract in any way from the readability of the marked-up text. 
> Doc-o-matic's markup language is very powerful, and yet unless you know 
> what you are looking for you'd think its just regular prose.) I have a 
> prototype (called "DocLobster") which does similar types of processing 
> on Python docstrings, but I haven't publicized it because I didn't feel 
> like competing against ReST.
> 
> However, I realize that I'm in the minority with this opinion; I don't 
> want to force anyone to conform to my idea of markup, but at the same 
> time I'd prefer not to have other people dictate my choice either.
> 
> Instead, what I'd like to see is a way for multiple markup languages to 
> coexist and compete with each other on a level playing field, instead of 
> one being chosen as the winner.

How about if plain text be the default, with the ability to over ride it to 
generate another type of output?  This is pretty much the design I'm following. 
  It doesn't choose any markup style or have a preference other than plain text.

Basically you import the gettext module, then add methods to the class's for 
each section to produce the marked up output you want. Then add a few functions 
to assemble it into a page.  I'm sure there's room for improvements, but this 
seems like the most direct way to do it.

Pydoc doesn't just process doc strings.  If that was all there was to it, then 
it would be much easier to automate in a generalized way.  Each section, may 
get 
it's data from different sources other than doc strings, and those may need to 
be handled in special ways, especially in the cases where it is nested within 
other sections.  This is what makes it difficult to generalize into a 
sequential 
producer to consumer pattern.  Not weather or not the doc string has additional 
markup in it.

I think the main goal for 2.6 should be a cleaned up package with some modest 
user interface and API improvements.  But also to keep things open ended for 
later enhancements.


[Maybe for python 3.0]

While reading some of the other discussions on multi-methods, generic 
functions, 
  and ABC's with mixins, it occurred to me that pydoc may be a good place to 
test some of those ideas in.  It's complex enough of a problem that it may 
benefit from those more advanced python features.

But I think that we shouldn't wait for those to do a basic face lift now.  
Think 
of it a preliminary clean up if you will in this case.

Ron

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Rewrite of import in Python source (sans docs) is complete

[Brett Cannon]

Finally, after a few months worth of work, I have finally gotten far enough
in my import rewrite that I am willing to stick my neck out and say it is
semantically complete!  You can find it in the sandbox under import_in_py.

So, details of this implementation.  I implemented PEP 302 importers/loaders
for built-in, frozen, extension, .py, and .pyc files along with rewriting
the steps __import__ goes through to do an import.  I also developed an API
for .py/.pyc file handling so that there is a generic filesystem
importer/loader and a separate handler for .py/.pyc files.  This should
allow for (relatively) easy selective overriding of just how .py/.pyc files
are stored (e.g., introducing a database backend) or how variants on
.py/.pyc files are handled (e.g., Quixote's .ptl format).

This code has extensive tests and so I am fairly confident that it does what
is expected of an import rewrite.  There are actually more lines in the test
file than the implementation.  There is also a mock implementation used for
testing.  Was interesting doing this in such a test-driven, XP style of only
coding what I needed.

I have run this code through the entire regression test suite and that is
where you find out subtle differences between this implementation and the
built-in import (you can see for yourself with the regrtest.sh shell
script).  First test_pkg will fail because currently the new import adds a
__loader__ attribute on all modules (that might change for security reasons)
and test_pkg is an old, stdout comparing test.  Second, test_runpy fails
because I have not implemented get_code on the filesystem loader which is
required by runpy.  Both are shallow issues that can be dealt with.

Third, and the hardest difference to deal with, is that you will get some
warnings that print out that you normally don't see.  This is because
warnings.warn and its stack_level argument don't have the effect people are
used to when importing a deprecated module.  Before you could set
stack_level to 2 and it would look like it came from the import statement.
But now, with import written in Python and thus on the call stack compared
to being in C and thus not showing up, two levels back is still in the
import code.  I really don't know how this should be dealt with short of
saying that the rule of thumb with 2 stack levels back for a warning does
not work when done at the import level.

It is not blazing fast at the moment.  Some things, like the built-in and
frozen importers/loaders could be rewritten in C without huge issue.  I am
also sure I have made some stupid design decisions at various points in the
code.  But there is benchmarking code in the sandbox called importbench and
it showed up a  10x speed slowdown on a Linux box I was using in mid to late
December when doing a fresh import of certain types (I don't remember
exactly which kind off the top of my head).

Because of this current slowness I don't know if people want to rush into
trying to make this the default import implementation quite yet or if this
is not too big of a thing since the common case of pulling out of
sys.modules is not that much slower.  I know I am currently not planning on
devoting the time to bootstrap it in as I have my security work to finish
first along with other Python stuff that seems more pressing to me.  And
since (I think) I don't need to bootstrap it in order to finish my security
work I can't justify spending work time on it.  But I can rearrange
priorities if people really want to pursue this (especially if I can get
some help with it).

As with the module's name, it is currently named 'importer', but that is bad
since it conflicts with the idea of importers from PEP 302.  I was thinking
importlib, but I wanted to wait and see what other people thought.

Don't know if you guys are okay with me checking this in without having it
vetted by the community first like we prefer all new modules to do.  I have
not done the LaTeX docs yet.

I think that is all of the details that I can think of.  I am still working
towards implementing the security needed so that an application that embeds
Python can execute arbitrary code securely.  Giving a talk at PyCon on the
topic for anyone interested.

Special thanks needs to go to Paul Moore who I talked to through most of the
design of the code.  Nick Coghlan also provided some handy feedback.  And
Neal Norwitz for bugging about wanting something like this done.  Plus
thanks to everyone who has shown support.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Renaming Include/object.h

[Martin v. Löwis]

Andrea Griffini schrieb:
> I've a partially related question... why isn't the module structure in 
> an include file .h
> and is instead in Objects/moduleobject.c ?
> For the cached lookup optimization I copied the definition but that's surely
> a bad way to do it I however wondered if there were good reasons for
> module objects for not being published.

I guess it's included in the C file because that's the easiest way to
implement it. AFAICT, it has been that way from the beginning.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Fernando Perez]

Ron Adam wrote:

> Laurent Gautier wrote:

>  > From the top of my head, there might be  "ipython" (the excellent
>  > interactive console) is possibly using pydoc
>  > (in any case, I would say that the authors would be interested in
>  > developments with pydoc)

Certainly :)  I'd like to ask whether this discussion considers any kind of
merging of pydoc with epydoc.  Many projects (ipython included) are moving
towards epydoc as a more capable system than pydoc, though it would be nice
to see its API-doc-generation capabilities be part of the stdlib.  I don't
know if that's considered either too large or too orthogonal to the current
direction.

> According to the web site, ipython is based on twisted, and is currently
> still
> limited to python 2.3 and 2.4.  Also, the output of the help() function
> will not change much so I doubt it would be a problem for them.

A few corrections:

- IPython's trunk is NOT based on twisted at all, it's a self-contained
Python package which depends only on the Python standard library (plus
readline support under windows, which we also provide but requires ctypes).

- The ipython development branch does use twisted, but only for its
distributed and parallel computing capabilities.  Eventually when that
branch becomes trunk, there will /always/ be a non-twisted component for
local, terminal-based work.

- The last release (0.7.3) fully supports Python 2.5.  In fact, one nasty
bug in 2.5 with extremely slow traceback generation was kindly fixed by
python-dev in the nick of time after my pestering (an ipython user found it
first and brought it to my attention).

>  > Otherwise a quick search lead to:
>  > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
>  > are only in use - wouldn't these functions belong more naturally to
>  > "cgi" ?)
> 
> Thanks!, These html formatting functions still exist or are small enough
> to move
> into cgitb, so it will be just a matter of making sure they can be
> reached.  I don't think they will be a problem.

If anyone is interested in updating cgitb, you might want to look at
ipython's ultratb (which was born as a cgitb port to ANSI terminals):

http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/ultraTB.py

It contains functionality for generating /extremely/ detailed tracebacks
with gobs of local detail.  These verbose tracebacks have let me fix many
ipython bugs from crash dumps triggered by remote code and libraries I
don't have access to, in cases where a normal traceback would have been
insufficient.  Here's a link to a slightly outdated simple example (the
formatting has improved a bit):

http://ipython.scipy.org/screenshots/snapshot1.png

Obviously the right thing to do would be to separate the ANSI coloring from
the structural formatting, so that the traceback could be formatted as
HTML, ANSI colors or anything else.  That is very much on my todo list,
since the network-enabled ipython will have browser-based interfaces in the
future.

All of ipython is BSD licensed, so any code in there is for the taking.

Best,

f

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Laurent Gautier]

2007/1/6, Ron Adam <[EMAIL PROTECTED]>:
> Laurent Gautier wrote:
[cut]
>  >> I think any API issues could be worked out.  Are there any programs
>  >> you know of,
>  >> (yours?), that import pydoc besides the python console?
>  >
>  > What I did barely qualifies as a hack for my own usage -it won't count-.
>
> It could be these changes will give you a way to do the same thing in a less
> haskish way.
>

This precisely why I get myself into the present trouble ;-)

>  > From the top of my head, there might be  "ipython" (the excellent
>  > interactive console) is possibly using pydoc
>  > (in any case, I would say that the authors would be interested in
>  > developments with pydoc)
>
> According to the web site, ipython is based on twisted, and is currently still
> limited to python 2.3 and 2.4.  Also, the output of the help() function will 
> not
> change much so I doubt it would be a problem for them.

Sorry for answering a bit off-the-question. My meaning was that they would be
interested in knowning that "pydoc" is changing (and would surely have ideas).

>  > Otherwise a quick search lead to:
>  > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
>  > are only in use - wouldn't these functions belong more naturally to
>  > "cgi" ?)
>
> Thanks!, These html formatting functions still exist or are small enough to 
> move
> into cgitb, so it will be just a matter of making sure they can be reached.  I
> don't think they will be a problem.
>

I read your comment about having not too many things changed for 2.6.
(or that will be bumped to 3000).

A suggestion I would have would be to create an html/htmlrender module
in the pydoc-package-to-be and start putting all the html formatting function
(as they are completely independent of pydoc, as far as I can see, over there).
You can then create wrappers to the original functions including a deprecation
warning. You can refer to Michael Chermside's recipe for an example of
implementation with a deprecation decorator -
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/391367
)
The suggestion above would actually apply to *anything* that is changed
in pydoc, providing the benefit of allowing the necessary changes while having
a temporary API to provide back-compatibility.


>
>  > - "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).
>
> Thanks again.  This might be better to move into the pydoc package.  Any 
> opinions?
>

We both pretty much agree, but now we've got to find the modules depending
on DocXMLRPCServer (and hope the recursion won't go on for too long). We
would also have to contact the author of DocXMLRPCServer (moving it would
go with potential changes, and he would certainly have suggestions about that).

>  > - "happydoc" (reportedly having problems with python 2.4 - I am not
>  > sure that it is maintained)
>
> Happydoc does not import pydoc as far as I could tell, so this won't effect 
> them
> in any direct way I think.  They've pretty much implemented everything from
> scratch.  At worse they would just need to copy the parts from the older 
> version
> into their distribution.
>
> I think you got a false positive on this because pydoc is a substring of 
> happydoc.
>

Yes this was a false positive.
I kept it in the list thinking there might be interesting ideas there as well
(but I forget to label it as such - sorry for the confusion).

>  > "cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.
>  >
>  > "cgitb" seems to be mostly using HTML formatting helpers (and that would
>  > suggest the need for an HTML-rendering module - may be for a future
>  > improvement,  a first step would be separate the rendering/viewing
>  > from extraction and modeling of documentation data).
>
> Making sure these still work would be a good sub project for someone a little
> later.  (I'll do it if no one else has time or wants to.)  I'm trying not to
> change thing to drastically.  If the changes are too big, ie... introducing 
> and
> altering a lot of other modules.  Then this will need to move back to the
> python-3000 list.
>

I agree that altering a number of other modules would be a little
tricky to manage
for version 2.6, but I would think that the makeover of pydoc would
benefit from being made early (back-compatibility being ensured by the
deprecation decorator
mentioned above, for example).

>  > "DocXMLRPCServer" looks (at first sight), like a viewer that would be
>  > bundled
>  > with pydoc as a sub-module (i.e., module in a package).
>
> Yes, that was my thought too. :-)
>
> But moving it may need to wait until python-3000.  (?)

An other way is to move it, while keeping a dummy module that imports the
module from its new location (and issue deprecation warnings).
What I fear is that the "let's wait for python-3000"  correspond to postponing
changes to "some other time".

[cut]
>  > I do not think it will work as a zip file shuttled around (in my
>  > experience).
>  > A versioning 

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Phillip J. Eby]

At 06:35 PM 1/5/2007 -0700, Fernando Perez wrote:
>Ron Adam wrote:
>
> > Laurent Gautier wrote:
>
> >  > From the top of my head, there might be  "ipython" (the excellent
> >  > interactive console) is possibly using pydoc
> >  > (in any case, I would say that the authors would be interested in
> >  > developments with pydoc)
>
>Certainly :)  I'd like to ask whether this discussion considers any kind of
>merging of pydoc with epydoc.

Unless there's been a complete rewrite of epydoc since the last time I 
looked at it, I'd have to give a very strong -1 against epydoc; it has all 
the problems of pydoc, plus new ones.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Barry Warsaw]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Jan 5, 2007, at 9:41 PM, Phillip J. Eby wrote:

> Unless there's been a complete rewrite of epydoc since the last time I
> looked at it, I'd have to give a very strong -1 against epydoc; it  
> has all
> the problems of pydoc, plus new ones.

I haven't read this entire thread, so I'll just chime in to say that  
I've /used/ epydoc and like it quite a bit.  I've even hacked on it a  
little to fix a few things and it didn't seem that bad, though I  
didn't do any major work on its internals.  I've used both the 2.x  
version and the 3.x version but I haven't used anything in the last,  
I dunno, 4 or 5 months.

Note that I was using it on a heavily embedded/extended application  
and it did a pretty good job of pulling docs out of C coded  
docstrings.  I had to patch Python a bit here and there (I think most  
of those fixes are in Python 2.5) and I know that the epydoc guys  
fixed a few things related to C types (e.g. such as that the tp_doc  
has to document both the type and the constructor).  Probably the  
biggest issue that I remember was needing to invoke it  
programmatically, which was an absolute requirement for us, since  
none of the extension modules were importable unless epydoc was run  
inside the embedded environment.  I got it to work, but it was a bit  
of a pain.

If you've already explained it, that's fine, but if not, could you  
outline what you have against epydoc?

- -Barry

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ8ww3EjvBPtnXfVAQJmnwQAnTn7W7Nri5Q+pSPmTLaIvqnmRJWDegpF
HSIDY8nBcMwsST76gzpwt02GikWtyy4gujgHiAEyr4/eQyJsMcnXMptkgXixtuPz
wA2pJkeo87eorPBMtOMoB9XpoyUkQTh5W/lGnR3rOinZPeiqJFEzc//DIJV+H/p7
Iqrie+FVnis=
=sNBX
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com