Re: [Zope-dev] Site-crawler (find unused objects)
[EMAIL PROTECTED] wrote: Hi, has anyone written a script which crawls a site and lists all objects which aren't referenced anymore? Crawling a site means going from a root object, and following all of its references, recursively. Any objects you find through this process are, by virtue of the fact they have been found by that process, referenced. Your requirement is self-contradictory. -- Steve Alexander ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...
* Starting at root, check all objects if they are referenced, * and produce a list of those which are not, for cleanup purposes. * Packing the database cleans up in this manner. Oh well. Third try: I know about the 'pack database' button. Garbage collection of this kind is not my problem. I have lots of scripts, dtml methods etc. everywhere which are perfectly well-known to the ZODB, nothing wrong with that, but which are simply not used by me anymore. No usage from other scripts nor methods nor documents. And these buggers I'd like to find. Finally unmistakeable? Tobias Herp ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...
extremely expensive. you would have to... - assemble a list of all objects IDs in the ZODB and then... - parse all contents of all objects and check against that ID list. you would probably need a little counter for every single ID that gets incremented upon finding its ID referenced, and all those that are left over with the counter at 0 in the end would probably be candidates for removal. this schema will fall down the moment your object IDs are not unique across the whole ZODB. besides, every object is potentially different in how you access and read its contents. if your whole intent is to have a cleaner ZODB and there is no pressing reason to do this cleanup, i would just forget about it. jens On Friday, Aug 30, 2002, at 07:48 US/Eastern, [EMAIL PROTECTED] wrote: * Starting at root, check all objects if they are referenced, * and produce a list of those which are not, for cleanup purposes. * Packing the database cleans up in this manner. Oh well. Third try: I know about the 'pack database' button. Garbage collection of this kind is not my problem. I have lots of scripts, dtml methods etc. everywhere which are perfectly well-known to the ZODB, nothing wrong with that, but which are simply not used by me anymore. No usage from other scripts nor methods nor documents. And these buggers I'd like to find. Finally unmistakeable? Tobias Herp ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...:o)
[EMAIL PROTECTED] wrote: * I have lots of scripts, dtml methods etc. everywhere which are * perfectly well-known to the ZODB, nothing wrong with that, but which * are simply not used by me anymore. No usage from other scripts nor * methods nor documents. And these buggers I'd like to find. * There is no easy way to find such things. * Especially given acquisition, and the dynamic nature of Zope. Finally we got it :o) Ok, the reason for me posting this question was the difficult nature. So I take your answer as a 'no' to my original question if anyone has already written such a script. Maybe this would be a good thing to accompany future Zope versions. There could be a tab which shows all objects which are called explicitely by the actual one in a sortable list. By the way: is this the correct mailing list for such suggestions? If not, which one is it? You forgot to post my suggestions: You can use Zope Find to get a list of them, and then manually audit them. You could instrument such methods to output logging, and check the logs every so often to see what is typically being used. You could use Zope Find to see if the name of particular methods is used within other methods. -- Steve Alexander ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...
[EMAIL PROTECTED] wrote:
I have lots of scripts, dtml methods etc. everywhere which are
perfectly well-known to the ZODB, nothing wrong with that, but which
are simply not used by me anymore. No usage from other scripts nor
methods nor documents. And these buggers I'd like to find.
They're impossible to find :)) Take for example the 'index_html' of
a folder: nothing else in my Zope app refers to it. But whenever
someone browses to http://my.zopeserver.com/folder they see that
'index_html'. No automatic method can guess what objects have become
irrelevant.
If you want to find everything that isn't refered to by another Zope
object, you could use 'sitecopy' to make a filesystem copy of everything
in your ZODB, 'ls -R' to get a list of filenames, and
'grep -r filename *' for each filename in the list. If the grep returns
nothing, nothing refers to that object.
You could do the same kind of thing ('ls -R' and 'grep') from the Zope
Management Interface using Jerome Alet's zshell.
--
Jean Jordaan
Upfront Systems http://www.upfrontsystems.co.za
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding... :o)
* I have lots of scripts, dtml methods etc. everywhere which are * perfectly well-known to the ZODB, nothing wrong with that, but which * are simply not used by me anymore. No usage from other scripts nor * methods nor documents. And these buggers I'd like to find. * There is no easy way to find such things. * Especially given acquisition, and the dynamic nature of Zope. Finally we got it :o) Ok, the reason for me posting this question was the difficult nature. So I take your answer as a 'no' to my original question if anyone has already written such a script. Maybe this would be a good thing to accompany future Zope versions. There could be a tab which shows all objects which are called explicitely by the actual one in a sortable list. By the way: is this the correct mailing list for such suggestions? If not, which one is it? Thanks, Tobias ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Potential Improvements for xml-rpc debugging
I am working on improved xml-rpc fault output because I find the current output from Zope less than useful. Basically what my version does is strip the html tags from the error value returned from Zope/standard_error_message, formats it in a simple way (basically justs trims line breaks down) and then generates a formatted traceback (if you are in debug mode) as the xml-rpc fault string. If you aren't in debug mode then it just outputs the error type and the formatted value as the fault string. Now my question is whether this is a good thing in general. I like it especially when accessing Zope using Python's xmlrpclib, but do any of have reasons for keeping the html tags in the fault string? Thought I would ask before I check this in. -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Potential Improvements for xml-rpc debugging
On 30 Aug 2002 at 10:10, Casey Duncan wrote: Now my question is whether this is a good thing in general. I like it especially when accessing Zope using Python's xmlrpclib, but do any of have reasons for keeping the html tags in the fault string? No, drop the HTML! I do a lot of xml-rpc with IE 5.5 clients, and getting a fault is a pita. I keep modifying xmlrpc.py (each time I update Zope) to print the traceback on the server, but that's a pita. Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding... :o)
Original problem:
I have lots of scripts, dtml methods etc. everywhere which are
perfectly well-known to the ZODB, nothing wrong with that, but which
are simply not used by me anymore. No usage from other scripts nor
methods nor documents. And these buggers I'd like to find.
@Steve:
You forgot to post my suggestions:
Sorry, no offense meant. Here they are:
You can use Zope Find to get a list of them, and then manually audit
them. You could instrument such methods to output logging, and check the
logs every so often to see what is typically being used.
You could use Zope Find to see if the name of particular methods is used
within other methods.
Well, I simply liked to know if anyone had already done it, so I simply
forgot to add them.
@ Jean:
They're impossible to find :)) Take for example the 'index_html' of
a folder: nothing else in my Zope app refers to it. But whenever
someone browses to http://my.zopeserver.com/folder they see that
'index_html'. No automatic method can guess what objects have become
irrelevant.
Well, the search seed could be an optional argument w/default value
'index_html'.
If you want to find everything that isn't refered to by another Zope
object, you could use 'sitecopy' to make a filesystem copy of
everything in your ZODB, 'ls -R' to get a list of filenames, and
'grep -r filename *' for each filename in the list. If the grep
returns nothing, nothing refers to that object.
You could do the same kind of thing ('ls -R' and 'grep') from the
Zope Management Interface using Jerome Alet's zshell.
Interesting suggestions! Of course, these unfortunately won't handle
the acquisition thing correctly.
To dream a dream:
Consider a tab for folderish objects which allows to search them for
unused scripts/methods/... (after pressing a button because it would
possibly take a while), producing a sortable list of links, with
checkboxes for deletion...
Consider a tab for methods... which allows to parse them and produces
a sortable list of links to the other referenced methods...
Maybe I'll write something like that, but it would be my very first
Zope product.
By the way, new question:
Is this the correct mailing list for such suggestions?
If not, which one is it?
Tobias Herp
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Potential Improvements for xml-rpc debugging
Now my question is whether this is a good thing in general +1 and not just for xmlrpc, the less HTML that gets sent in errors the better. -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...:o)
On Fri, 30 Aug 2002 [EMAIL PROTECTED] wrote: Consider a tab for methods... which allows to parse them and produces a sortable list of links to the other referenced methods... Just to make it clear what I'm talking about when I say effectively impossible, consider the following bit of DTML: dtml-var somefolder[myvar](_.None,_) How are you going to figure out which objects that references? And this is a *very* simple example, just in DTML. You should see some of the indirection gyrations certain of my python code goes through grin. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...:o)
On Fri, 30 Aug 2002 [EMAIL PROTECTED] wrote: Consider a tab for methods... which allows to parse them and produces a sortable list of links to the other referenced methods... Good luck grin. You might manage a Quick and Dirty implementation, but to guarantee you've not missed anything you pretty much have to run the site as a program, which leads you to the classic computer science halting problem, I think. And that still doesn't address the question of external references. By the way, new question: Is this the correct mailing list for such suggestions? If not, which one is it? For the original has anyone implemented question, probably 'zope'. For a discussion of how to implement this (or rather the effective impossiblity of implementing this reliably) this is probably the appropriate place. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] No Data.fs.in in Win32 binary releases (was Re: Unit testing on win32...)
Shane Hathaway wrote: raise ValueError, can\'t create a read-only file ValueError: can't create a read-only file Apparently the file Data.fs.in doesn't exist. Why? (Try adding print dfi to custom_zodb.py.) Problem found! dfi points to the right place, but for some reason, the 2.5.1 Win32 binary release doesn't actually have a Data.fs.in in it. The reason the testrunners don't barf is that the Zope-2_5-branch _does_ have a Data.fs.in in it. What on earth is going on here? cheers, Chris PS: I do remember rumblings of getting rid of Data.fs.in, perhaps this is fallout of that? How does that impact the testing package? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...:o)
R. David Murray wrote: On Fri, 30 Aug 2002 [EMAIL PROTECTED] wrote: Consider a tab for methods... which allows to parse them and produces a sortable list of links to the other referenced methods... Good luck grin. You might manage a Quick and Dirty implementation, but to guarantee you've not missed anything you pretty much have to run the site as a program, which leads you to the classic computer science halting problem, I think. And that still doesn't address the question of external references. By the way, new question: Is this the correct mailing list for such suggestions? If not, which one is it? For the original has anyone implemented question, probably 'zope'. For a discussion of how to implement this (or rather the effective impossiblity of implementing this reliably) this is probably the appropriate place. Ok, quick and probably dumb idea how one could write a ZLinkBot. If there is a central part in zope (ZPublisher?) which always runs when methods/scripts/etc are called, one could patch it to also log the path of the object which is called (the physical path!). Then just use a web spider which crawls the whole site, and after that compare the list which has been produced by the method above to the list of objects in your ZODB. Everything which hasn't been touched is an orphan. Could ZPublisher.Publish.call_object() be that central piece of code? cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last
As people already noted, this is essentially impossible for a bunch of reasons. ...but... You could add to Zope some sort of access counters on all objects. You could reset this on a live site, twiddle your thumbs for a week, then dump those counters. Objects with the smallest counters are clear candidates for removal but you would have to make the decision with your brain or other equivalent thought-organ. No automated process can remove all dead objects safely in a system like Zope. There would be a fairly large performance impact if those counters live in the objects since then the assumption the object DB is primarly a read-only resource would be false. However you could keep it in-core since the counters would be just a guide and it wouldn't be much larger than the DB index. In fact I bet you could attach such a sytem to the ZODB index logic pretty easily... Hmm... I think I'm talking beyond my knowledge. I think I'll shut up now. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] find unused objects: hopefully the last misunderstanding...:o)
On Fri, 30 Aug 2002, Oliver Bleutgen wrote: If there is a central part in zope (ZPublisher?) which always runs when methods/scripts/etc are called, one could patch it to also log the path of the object which is called (the physical path!). Then just use a web spider which crawls the whole site, and after that compare the list which has been produced by the method above to the list of objects in your ZODB. Everything which hasn't been touched is an orphan. Combine this with a regression test suite that exercised all of the site's functionality and pages, and you'd have something. As with Unit Tests, if it isn't tested, it should be deleted grin. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope-Coders] Re: [Zope-Checkins] CVS: Zope/lib/python/ZPublisher - xmlrpc.py:1.14
I agree that error values containing html are pretty evil, but I think they go back to the days before we had any notion of using Zope beyond direct web publishing. Many of these originate from the response object itself, so I plan to override them in the xml-rpc reponse class. In any case, in the process of refactoring my first try (below) at untangling error values in xmlrpc, I found a small diamond in the rough that is worth pointing out: Zope response objects have a class attribute _error_format that is set to text/plain in BaseResponse and overridden as text/html in HTTPResponse. I overrode it back to text/plain in xmlrpc.Response in my latest check-in. Zope's top level exception handler (in Zope/__init__.py) sniffs this value and only fires off the rendering of standard_error_message if response._error_format is 'text/html'. So this change alleviates much of the noise in the xmlrpc fault string and eliminates the need for much of the formatting hacks I put in. The refactored xmlrpc exception handler still strips tags from the error_value, I could be convinced to escape them instead, if that's what user's would prefer. In any case I plan to fix the most egregious offenders (like not found errors) entirely by making them a plain text message for xml-rpc. -Casey On Friday 30 August 2002 12:16 pm, Florent Guillaume wrote: Casey Duncan [EMAIL PROTECTED] wrote: Update of /cvs-repository/Zope/lib/python/ZPublisher In directory cvs.zope.org:/tmp/cvs-serv24443 Modified Files: xmlrpc.py Log Message: Improved error output for xmlrpc faults - Value is stripped of HTML tags and minimally formatted +# Strip HTML tags and format the error value +v = str(v) +v = re.sub(rbr\s*/?, \n, v) +remove = [r[^]*, r[A-Za-z]+;] +for pat in remove: +v = re.sub(pat, , v) +v = re.sub(r\n(?:\s*\n)+, \n\n, v) + +from Globals import DevelopmentMode +if DevelopmentMode: +from traceback import format_exception +value = ''.join(format_exception(t, v, tb)) +else: +value = '%s\n\n%s' % (t, v) + IMHO this goes to show that the error value should never have contained HTML in the first place. I think error values should be defined as pure text, and the HTML publishing would html_quote it as needed, and the XMLRPC publishing would do what it deems. Florent ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Proxy Object / __getattr__ / Acquisition
[EMAIL PROTECTED] writes: I am using __getattr__ within my product, and the code pasted below works ... However, because of the way that this messes with Acquisition, certain things like accessing the ZMI pages or acquired methods can be quite slow (but work). ... def __getattr__(self, name): return getattr(self._CurrentVersion, name) I am (almost) sure, you run into the same problem than I did and reported to mailto:[EMAIL PROTECTED]. You get an infinite __getattr__ loop, broken via a RuntimeError, maximal recursion exceeded which is silently ignored. Use def __getattr__(self,name): # avoid acquisition current= self.__dict__['_CurrentVersion'] if hasattr(current,name): return getattr(self._CurrentVersion,name) More info in the ZPT mailing list archives. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
