[Zope] ZEO ClientDisconnected error
Hi there,
One of my clients is using Plone on Solaris 9. We have set up one ZEO
client on a Solaris machine and a ZEO server on another Solaris machine
(for testing purposes only at this stage).
Performance is great but every morning, the first hit to the Plone site
returns a ClientDisconnected error (see full error log below). It looks
like the socket on the server gets corrupted or something.
When the error occurs, the server is non-responsive for a while
(anywhere between 10 sec and 2 minutes) and then eventually the server
responds.
Has anybody seen that before? Is it Zope related or should I look in
other directions?
Researching Zope on Solaris, I read somewhere (can't remember where now)
that Solaris doesn't like long-running processes too much (well, the
Zope process, for instance). Could it be the culprit here?
Any advise or pointer welcome.
Cyrille
(in the logs below, appserv05 is the ZEO client, pehi is the ZEO server)
From event.log:
=
ERROR ZEO.zrpc.Connection(C) (pehi.myclient.govt.nz:8100) Error caught
in asyncore raise socket.error, why error: (9, 'Bad file descriptor')
2006-04-28T09:09:52
=
From the error log:
=
Time
2006-04-28 09:09
User Name
admin (admin)
Request URL
http://plone.appserv05.myclient.govt.nz/front-page/document_view
Exception Type
ClientDisconnected
Exception Value
Traceback (innermost last):
•Module ZPublisher.Publish, line 114, in publish
•Module ZPublisher.mapply, line 88, in mapply
•Module ZPublisher.Publish, line 40, in call_object
•Module Shared.DC.Scripts.Bindings, line 311, in __call__
•Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
•Module Products.CMFCore.FSPageTemplate, line 195, in _exec
•Module Products.CMFCore.FSPageTemplate, line 134, in pt_render
•Module Products.PageTemplates.PageTemplate, line 104, in pt_render
FSPageTemplate at /myclient/Plone/document_view used for
/myclient/Plone/front-page
•Module TAL.TALInterpreter, line 206, in __call__
•Module TAL.TALInterpreter, line 250, in interpret
•Module TAL.TALInterpreter, line 711, in do_useMacro
•Module TAL.TALInterpreter, line 250, in interpret
•Module TAL.TALInterpreter, line 426, in do_optTag_tal
•Module TAL.TALInterpreter, line 411, in do_optTag
•Module TAL.TALInterpreter, line 406, in no_tag
•Module TAL.TALInterpreter, line 250, in interpret
•Module TAL.TALInterpreter, line 711, in do_useMacro
•Module TAL.TALInterpreter, line 250, in interpret
•Module TAL.TALInterpreter, line 481, in do_setGlobal_tal
•Module Products.PageTemplates.TALES, line 221, in evaluate
URL: file:CMFPlone/skins/plone_templates/global_defines.pt
Line 3, Column 0
Expression: PythonExpr language or here.Language() or default_language
Names:
•{'container': PloneSite at /myclient/Plone,
• 'context': ATDocument at /myclient/Plone/front-page,
• 'default': Products.PageTemplates.TALES.Default instance at
0xacaf58,
• 'here': ATDocument at /myclient/Plone/front-page,
• 'loop': Products.PageTemplates.TALES.SafeMapping object at
0x22027d8,
• 'modules':
Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at
0xabd5d0,
• 'nothing': None,
• 'options': {'args': ()},
• 'repeat': Products.PageTemplates.TALES.SafeMapping object at
0x22027d8,
• 'request': HTTPRequest,
URL=http://plone.appserv05.myclient.govt.nz/front-page/document_view,
• 'root': Application at ,
• 'template': FSPageTemplate at /myclient/Plone/document_view used
for /myclient/Plone/front-page,
• 'traverse_subpath': [],
'user': admin}
•Module Products.PageTemplates.ZRPythonExpr, line 47, in __call__
__traceback_info__: language or here.Language() or default_language
•Module Python expression language or here.Language() or
default_language, line 1, in expression
•Module Products.Archetypes.ClassGen, line 58, in generatedAccessor
•Module Products.Archetypes.Field, line 768, in get
•Module Products.Archetypes.Field, line 637, in get
__traceback_info__: ('language', ATDocument at
/myclient/Plone/front-page, {'field': Field language(string:rw),
'schema': Products.Archetypes.Schema.Schema object at 0x171a330})
•Module Products.Archetypes.Storage, line 175, in get
•Module UserDict, line 19, in __getitem__
•Module ZODB.Connection, line 704, in setstate
•Module ZODB.Connection, line 760, in _setstate
•Module ZODB.serialize, line 495, in setGhostState
•Module ZODB.serialize, line 488, in getState
•Module ZODB.serialize, line 436, in _persistent_load
•Module ZODB.Connection, line 207, in get
•Module ZEO.ClientStorage, line 746, in load
•Module ZEO.ClientStorage, line 769, in loadEx
•Module ZEO.ClientStorage, line
[Zope] Re: ZEO ClientDisconnected error
Hi Tres, thanks for your response: it is very helpful. The posting you refer to is particularly helpful. Thanks for digging it up for me! Cheers, Cyrille Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Cyrille Bonnet wrote: Hi there, One of my clients is using Plone on Solaris 9. We have set up one ZEO client on a Solaris machine and a ZEO server on another Solaris machine (for testing purposes only at this stage). Performance is great but every morning, the first hit to the Plone site returns a ClientDisconnected error (see full error log below). It looks like the socket on the server gets corrupted or something. When the error occurs, the server is non-responsive for a while (anywhere between 10 sec and 2 minutes) and then eventually the server responds. Has anybody seen that before? Is it Zope related or should I look in other directions? Researching Zope on Solaris, I read somewhere (can't remember where now) that Solaris doesn't like long-running processes too much (well, the Zope process, for instance). Could it be the culprit here? Any advise or pointer welcome. I would check for a rude firewall between your appserver and the storage server: one which is closing sockets it sees as idle for too long. Dieter Maurer has a product which works around such scenarios: http://mail.zope.org/pipermail/zodb-dev/2005-June/008967.html Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEVsxj+gerLs4ltQ4RAh+IAKChzq20oj0sFmD9jNxpzQWtis98xQCfamtY Fyeu1abM7pUBG4P1ANbQ8mM= =eXdz -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope product update question
Hi there, I have a question regarding product update. I have updated a product (ATContentTypes) on the file system. Upon restarting the server, the new version of the product appears in red in the QuickInstaller (as expected). Even though I haven't re-installed the product, some changes are picked up (for instance, it uses the new schemata for ATDocument). Is this the expected behaviour? What is exactly the difference between a non-installed product on the file system and an installed product? Is the only difference that the Install.py is run when you install the product?? I am using Zope 2.7.4.0, Python 2.3.4. The versions of ATContentTypes are 0.2.0-rc3 (current) and 0.2.1-final (future, uninstalled). Any help appreciated. Cheers, Cyrille ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope product update question
Hi guys, thanks for your replies. It is very helpful. Essentially, I was trusting the Quick Installer tool too much: files on the file system will be used, regardless if you've installed the product or not. Thanks for your help. Cyrille Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20 Apr 2006, at 22:39, Cyrille Bonnet wrote: Hi there, I have a question regarding product update. I have updated a product (ATContentTypes) on the file system. Upon restarting the server, the new version of the product appears in red in the QuickInstaller (as expected). Even though I haven't re-installed the product, some changes are picked up (for instance, it uses the new schemata for ATDocument). Is this the expected behaviour? Yes. Once the new code is on the file system it will be read in and used. I believe the QuickInstaller does nothing more than execute external methods that have special well-known names which then trigger whatever Plone magic needs to be triggered to complete the upgrade. The QuickInstaller is a special Plone-only item, normal Zope product installation does not use it. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFESAKVRAx5nvEhZLIRArwXAJ0biLjUrcfVI3h+BVI2uBHJ1n9Z/gCgqnCw pZ6TPGApiF5Uw5iv86fVnu8= =nPcN -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Question about Zope and security
Hi Dieter, thanks for your response. It helps a lot. It looks like DigestAuth is a step in the right direction, but needs more work to be completely secure. I'll get back to my client and see where they want to go from here. Thx for your help. Cheers, Cyrille Dieter Maurer wrote: Cyrille Bonnet wrote at 2006-3-30 14:43 +1200: ... I did find Dieter Mauer's DigestAuth product: http://www.dieter.handshake.de/pyprojects/zope/#DigestAuth It looks good. I have used other produts from Dieter before and was very pleased with the quality of his code. Now, have other people used it? Does it work with WebDAV? It should work with WebDAV, provided the WebDAV client supports HTTP Digest Authentication. How secure is it (I am no security/encryption expert)? The corresponding RFC (RFC 2617) explains in detail how secure the basic mechanism is. My DigestAuth DigestAuthCrumber adds a bit of insecurity: * the passwords must be stored (inside Zope (!) not in the request) in plain text. This could be improved a bit, either by - using two way encryption -- but Zope must be able to get the plain text password back. - fixing the domain and using storing the MD5 hash of username, password and domain instead of the plain text password. Other authentication schemes would then need to be changed -- to use the same MD5 hash. Also, if it is good, why is not part of default Zope?? There are two sides of an answer: the Zope developpers/maintainers side and my side. Adding even a good package to the core means a (rather) long term commitment to support and maintain this package. When you follow comp.lang.python (or the corresponding mailing list), you see how reluctant the Python developpers are to include additional packages into the Python core -- to avoid these responsibilities. The Zope maintainers are even stricter: they look what they can get rid of rather than what they can include On my side: developping for the Zope core imposes much more overhead than developping independently: I would have to make a proposal, follow (partially stupid) style guides, add more tests (than necessary to convince me that the quality is sufficient)... Thus, I am reluctant to develop for the Zope core. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Question about Zope and security
Thanks to all for your feedback: I understand better what is going on now. SSL is definitely the way to go, that would solve all my problems. Now, just to push the problem a bit further: ideally, I'd like to put SSL just on the login form. Zope would authenticate the user in that request and return a session ID that would then be passed back and forth in each request (without SSL). That would be a balanced approach to security: I don't have to put SSL across the entire site. The site will be vulnerable to man-in-the-middle attacks, but only for the duration of a session. Is it possible to do that with Zope? Or does Zope require to identify the user on each request? Thanks for the help. Cyrille bruno desthuilliers wrote: Cyrille Bonnet wrote: Hi there, I have been telling all my clients about how great Zope is for security: fine-grained permissions, security framework, roles, etc. Now, one of my clients has a security expert who took a close look at how Zope authenticates users. The results were not good. The main problem is that Zope stores the username and password in a cookie in clear text (base64 encoded). *Zope* don't do that. It's the (infamous) CookieCrumbler products that is responsible for this horror. Even though it only happens in their internal network, my client wasn't too happy, because it makes them vulnerable to a man-in-the-middle attack. I know, the odds of that happening are low, but storing the username and password in clear text is clearly not best practice. That's an understatement. So, my question is: is there a way to secure Zope authentication? yes : use https. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Question about Zope and security
Hi there, I have been telling all my clients about how great Zope is for security: fine-grained permissions, security framework, roles, etc. Now, one of my clients has a security expert who took a close look at how Zope authenticates users. The results were not good. The main problem is that Zope stores the username and password in a cookie in clear text (base64 encoded). Even though it only happens in their internal network, my client wasn't too happy, because it makes them vulnerable to a man-in-the-middle attack. I know, the odds of that happening are low, but storing the username and password in clear text is clearly not best practice. So, my question is: is there a way to secure Zope authentication? I did find Dieter Mauer's DigestAuth product: http://www.dieter.handshake.de/pyprojects/zope/#DigestAuth It looks good. I have used other produts from Dieter before and was very pleased with the quality of his code. Now, have other people used it? Does it work with WebDAV? How secure is it (I am no security/encryption expert)? Also, if it is good, why is not part of default Zope?? Finally, a little side story: you know how in Windows XP, you can connect a drive to a WebDAV server? Well, if you install Service Pack 2, you can't use that feature to connect to Zope anymore. Interestingly enough, it seems that it is precisely because of that authentication vlunerability: Win XP SP2 refuses to connect to a WebDAV that doesn't at least encode the username/password in Digest authentication... Any comment or pointers are very welcome. Cyrille ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Question about Zope and security
Hi Terry, thanks for your comment. Stock Zope doesn't use cookie authentication, so you're actually talking about an alternate user folder product (which you don't specify and I don't know that many of them, so I can't really comment much -- except that SimpleUserFolder with CookieCrumbler will indeed put you in this situation (or did the last time I checked)). I am using Plone 2.1.2, which uses CookieCrumbler. I wanted to put the problem in a Zope perspective, though: this is why I didn't mention that. The fact that Zope stores passwords as plain text is not the issue if you're worried about man-in-the-middle attacks, though. The problem there is that you are passing passwords plain text in the request, and there is almost no way around that unless you run an SSL (HTTPS) server. Which you should if you want real security. Sorry, I wasn't even aware that Zope stores the passwords in plain text. My primary concern (for the moment) is passwords in plain text in the request. I had thought of SSL, but it doesn't solve the problem for WebDAV access. I should also mention that the site is for the general public, with a few users logging in. Of course, I can't put the public site on SSL, so I would have to have a separate URL for logged-in users with SSL. And I still have to worry about the ZMI and WebDAV access. It seems so much simpler to solve the problem at the root: change Zope authentication. Encrypting your password database without moving your server login to HTTPS is only going to create inconvenience without improved security (you can no longer send password reminders, for example) -- it's a false sense of security. Ouch, so on top of my concerns, passwords are stored in plain text?? Thanks for pointing that out. I'd rather encrypt passwords with a hash and reset the password if the users have lost it. Is it possible to do that in Zope? Obviously, I don't understand the ins and outs of Zope as well as most people on this list. So, my questions really are: * why is Zope authentication implemented that way? * Is it really complex to secure the authentication process? * Is there any documentation summing up Zope security (authentication process, password storage, etc.)? Cheers, Cyrille ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: HTML post processing in Zope
Chris Withers wrote: Cyrille Bonnet wrote: It works for me so far. But if you have specific examples that I can use to improve the filter, they would be very welcome. I think a filter is a totally abhorent way of attempting to tackle this... OK, but again, if you have a better idea, it is welcome. Modifying 12 templates does not look much better to me. * replace html ... with html (remove the namespace information) * remove the login portlet: Plone uses form parameters __ac_name and __ac_password, which the W3C validator rejects as invalid. Tee hee, so much for Plone's amazing standards compliance ;-) Looking closer, it was actually the ids that were causing the problem. Ids can't start with _ in HTML 4.01 but it is perfectly legitimate in XHTML. So, Plone is compliant with XHTML. I have been customising the templates in the past and it takes a lot of work, on many templates, all over the place. Well, your filter only changes things that are in main_template... ??? The filter runs on the HTTPResponse object, thus changing all the HTML output, not just the ouput from main_template. In addition, I'd like to keep the content stored in the ZODB as XHTML. Why? Well, looking forward, if the NZ government guidelines finally support XHTMl, we'll just need to remove the filter. In addition, we want to be able to transform the content with XSL transformations. Finally, Kupu and Epoz are good at producing XHTML, but don't support HTML 4.01. And, last but not least, I can upgrade Plone without having to rework all my templates now. Bwahahaha... the other great myth ;-) Before, i had to modify 10-12 templates at least. Between Plone 2.0.4 and 2.0.5, these templates got changed and I had to spend 20 hours or so reworking the HTML ouput and testing. Now, when we move to Plone 2.1, I hope to do no work at all. I don't think it is a myth :-) Cyrille ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: HTML post processing in Zope
Hi Chris,
Well, you'd better believe it :-)
It works for me so far. But if you have specific examples that I can use
to improve the filter, they would be very welcome.
Two additional things that I had to do to be HTML 4.01 compliant:
* replace html ... with html (remove the namespace information)
* remove the login portlet: Plone uses form parameters __ac_name and
__ac_password, which the W3C validator rejects as invalid.
I have been customising the templates in the past and it takes a lot of
work, on many templates, all over the place. In addition, I'd like to
keep the content stored in the ZODB as XHTML. And, last but not least, I
can upgrade Plone without having to rework all my templates now.
But if you have a better idea, your suggestions are most weclome.
Back to the insanity ;-)
Cheers.
Cyrille
Chris Withers wrote:
Cyrille Bonnet wrote:
Hi all,
I got the filter to work. I just added 3 lines of code in
ZPublisher.HTTPResponse.HTTPResponse (thanks for your suggestion,
Dieter):
doctype_str_search = re.compile(r'!DOCTYPE.*')
body = doctype_str_search.sub('!DOCTYPE HTML PUBLIC
-//W3C//DTD HTML 4.01 Transitional//EN
http://www.w3.org/TR/html4/loose.dtd;', body)
body = body.replace('/', '')
You can't be serious, right?
The above do NOT suddenly make it HTML 4.01, I'm 90% sure ;-)
Really, you should be customising the templates to serve HTML in the
format you need rather than persuing this insanity...
Chris
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: HTML post processing in Zope
Hi all,
I got the filter to work. I just added 3 lines of code in
ZPublisher.HTTPResponse.HTTPResponse (thanks for your suggestion, Dieter):
doctype_str_search = re.compile(r'!DOCTYPE.*')
body = doctype_str_search.sub('!DOCTYPE HTML PUBLIC
-//W3C//DTD HTML 4.01 Transitional//EN
http://www.w3.org/TR/html4/loose.dtd;', body)
body = body.replace('/', '')
It works great except that... it breaks the WebDAV ( I can still connect
but can't see the files).
I'd like to add a condition there: if WebDAV, don't do anything. But all
I have is the body. No port, for instance...
I thought of testing: if content type = 'text/html', byt WebDAV will
probably be of that type.
Any suggestion will be wecome.
Cyrille
Dieter Maurer wrote:
Cyrille Bonnet wrote at 2005-5-5 14:58 +1200:
I am trying to perform a post-processing on all HTTP responses, before
they get sent to the browsers. I am using Zope 2.7.3 nad Plone 2.0.5.
I had a look at the ZServer class: it seems to be the right place, but I
don't understand all the code there and I am afraid to break something :-(
I think ZPublisher.HTTPResponse.HTTPResponse is the better
place for tidying up.
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: HTML post processing in Zope
Answering my own questions: Testing for text/html does the trick as WebDAV follows some other content type (probably text/xml). It works great! Cyrille Cyrille Bonnet wrote: Thanks for all your answers, I usually use Apache to change HTTP headers. But here, I need to post-process the HTML. The reason is that the NZ Government Webguidelines require HTML 4.01 :-( and I'd like to keep Plone content and templates XHTML compliant. One way to do that is obviously to post-process the HTML with a language that is good at regular expressions (Perl?). But I thought it could be neat if the post-processing could be done in Zope itself. Anyway, I am looking at ZPublisher.HTTPResponse.HTTPResponse and it looks like the right place. Thanks for your help! Cyrille Dieter Maurer wrote: Cyrille Bonnet wrote at 2005-5-5 14:58 +1200: I am trying to perform a post-processing on all HTTP responses, before they get sent to the browsers. I am using Zope 2.7.3 nad Plone 2.0.5. I had a look at the ZServer class: it seems to be the right place, but I don't understand all the code there and I am afraid to break something :-( I think ZPublisher.HTTPResponse.HTTPResponse is the better place for tidying up. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Epoz and Tidy
I agree with you, Duncan, the tidy up can not be much more aggressive by default. And Kupu probably does the best possible job there. Now, the button Clean this up is a good idea, I think. Did you get started on this? I am happy to help if you do develop that feature. Also, another option for users that need to convert a lot of Word documents is, of course, WebDAV + PortalTransform. Cheers Cyrille Duncan Booth wrote: Cyrille Bonnet wrote: Daniel Dekany wrote: BTW, anybody has found a solution for fixing HTML copy-pasted from Microsoft Word (mostly 2000/XP)? Lot of users has MS Word, and the HTML pasted from it is a CSS killer mess. I tried mxTidy but it didn't improved substantially the HTML. So how do you guys do it? I have looked after solutions for Epoz, but didn't found any. But I don't stick to Epoz... if there is a solution already for Kupu (is Kupu already recommended over Epoz anyway?). Certainly the solution would be an Epoz post-tidy Python script, but I didn't found any for Word tidying. (However, the ideal would be if the HTML is tidied right on the client when it pastes it in -- thus user would really get what it sees, i.e. the HTML wouldn't be changed when he saves it. That effect is really evil.) As Shane pointed out, there is a tidy up in Kupu. However, in my experience, it is not a very good tidy up (if I remember correctly, a lot of tags are still there after the tidy up). Unfortunately there is a fine line between tidying up the cruft pasted from Word, and not stripping out things which might actually have been entered legitimately. I think Kupu does this pretty well (but then I'm a bit biased), but without any way to detect that the user is pasting from Word I don't see how much more could be stripped. So far as I know the only thing which doesn't really get stripped from the pasted Word text are the mso classnames. These can be manually blacklisted, but I never got round to producing a definitive blacklist. One of my thoughts is to provide a separate 'clean this up' button which would apply a more aggressive tidy-up than the one when saving. Also, I agree that only applying the tidy on save is bad, but there isn't a cross- browser way to detect a paste, and applying the cleanup on a large document every time you cut/paste one word wouldn't be nice either. Suggestions for improvements are most welcome. P.S. It isn't just pasting bad HTML which is a problem: some Microsoft applications supply RTF on the clipboard but not HTML and it turns out that if you paste RTF into IE it generates seriously invalid HTML with a totally weird and corrupted DOM. That is another area where I think the cleanup code finally does a passable job but not yet a perfect one. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: HTML post processing in Zope
Thanks for all your answers, I usually use Apache to change HTTP headers. But here, I need to post-process the HTML. The reason is that the NZ Government Webguidelines require HTML 4.01 :-( and I'd like to keep Plone content and templates XHTML compliant. One way to do that is obviously to post-process the HTML with a language that is good at regular expressions (Perl?). But I thought it could be neat if the post-processing could be done in Zope itself. Anyway, I am looking at ZPublisher.HTTPResponse.HTTPResponse and it looks like the right place. Thanks for your help! Cyrille Dieter Maurer wrote: Cyrille Bonnet wrote at 2005-5-5 14:58 +1200: I am trying to perform a post-processing on all HTTP responses, before they get sent to the browsers. I am using Zope 2.7.3 nad Plone 2.0.5. I had a look at the ZServer class: it seems to be the right place, but I don't understand all the code there and I am afraid to break something :-( I think ZPublisher.HTTPResponse.HTTPResponse is the better place for tidying up. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Epoz and Tidy
As Shane pointed out, there is a tidy up in Kupu. However, in my experience, it is not a very good tidy up (if I remember correctly, a lot of tags are still there after the tidy up). AFAIK, Kupu is integrated in Plone 2.1. Daniel Dekany wrote: Friday, April 29, 2005, 7:12:30 PM, Maik Jablonski wrote: Robert (Jamie) Munro wrote: How do I control tidy options when using Epoz and uTidyLib? I'd like it to output xhtml, but it is currently outputting uppercase tag names etc. If uTidyLib or mxTidy (recommended) is installed correctly, Epoz should output XHTML. Please check if you've installed uTidyLib with the correct python (same as running your Zope-Server). BTW, anybody has found a solution for fixing HTML copy-pasted from Microsoft Word (mostly 2000/XP)? Lot of users has MS Word, and the HTML pasted from it is a CSS killer mess. I tried mxTidy but it didn't improved substantially the HTML. So how do you guys do it? I have looked after solutions for Epoz, but didn't found any. But I don't stick to Epoz... if there is a solution already for Kupu (is Kupu already recommended over Epoz anyway?). Certainly the solution would be an Epoz post-tidy Python script, but I didn't found any for Word tidying. (However, the ideal would be if the HTML is tidied right on the client when it pastes it in -- thus user would really get what it sees, i.e. the HTML wouldn't be changed when he saves it. That effect is really evil.) -mj ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
