Re: [Zope3-dev] zope.tal.xmlparser.XMLParser() dislikes unicode
On 13.01.2007, at 18:49, Andreas Jung wrote:
Hi,
the XMLParser.parseString() method raises an exception
File /opt/python-2.4.4/lib/python2.4/unittest.py, line 260, in run
testMethod()
File /Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/
tests/test_xmlparser.py, line 127, in test_xx
self._run_check(xml, ())
File /Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/
tests/test_xmlparser.py, line 106, in _run_check
parser.parseString(source)
File /Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/
xmlparser.py, line 77, in parseString
self.parser.Parse(s, 1)
UnicodeEncodeError: 'ascii' codec can't encode characters in
position 43-48: ordinal not in range(128)
if the string to be parsed is a unicode strings and contains some
non-ascii
chars. The following snippet from a private unittest
(test_xmlparsers.py)
shows the error.
def test_xx(self):
xml = unicode('?xml version=1.0 encoding=utf-8?
fooüöä/foo', 'iso-8859-15')
self._run_check(xml, ())
I am not sure if this behavior is intentional?! Is the XMLParser
supposed
to deal with unicode strings or will it only accept a standard
Python string? A workaround inside parseString() would to check for
unicode
and convert the string on-the-fly to a Python string with utf-8
encoding.
This is possibly a limitation of the underlying Expat parser...any
recommendation how to deal with this issue?
IMHO it should only accept strings, because in the value should be a
xml string and therefore always has to be encoded in 'utf-8' or in
the encoding specified in the processing instruction.
Bernd
Andras
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/zope-
mailinglist%40mopa.at
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] zope.tal.xmlparser.XMLParser() dislikes unicode
--On 14. Januar 2007 10:48:06 +0100 Bernd Dorn [EMAIL PROTECTED] wrote: I am not sure if this behavior is intentional?! Is the XMLParser supposed to deal with unicode strings or will it only accept a standard Python string? A workaround inside parseString() would to check for unicode and convert the string on-the-fly to a Python string with utf-8 encoding. This is possibly a limitation of the underlying Expat parser...any recommendation how to deal with this issue? IMHO it should only accept strings, because in the value should be a xml string and therefore always has to be encoded in 'utf-8' or in the encoding specified in the processing instruction. I disagree with that. Since Zope 3 is supposed to use unicode internally (at least that's the legend) it should support unicode also at the parser level. Other languages like Java store XML also as unicode strings and support parsing it. Andreas pgp8ib4BIWYFC.pgp Description: PGP signature ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: test.py -u -f
Marius Gedminas wrote: Due to an accident of implementation, if you pass both -u (run unit tests) and -f (run non-unit tests) to the test runner, it will not run any tests at all. That is not useful. I have a branch where test.py -fu is equivalent to specifying neither -f nor -u and runs all the tests: http://svn.zope.org/zope.testing/branches/test-fu/?rev=71988view=rev If there are no objections, I would like to merge that branch. +1 -- http://worldcookery.com -- Professional Zope documentation and training 2nd edition of Web Component Development with Zope 3 is now shipping! ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: test.py -s src/zope/something
Marius Gedminas wrote: A long time ago test.py -s accepted directory names. Later it was changed to take package names instead. There is some backwards-compatibility code that replaces slashes with dots, but it is incomplete: test.py -s src/zope would try to run the tests for a nonexistent package src.zope. I have expanded the backwards-compatibility/convenience code in a branch: http://svn.zope.org/zope.testing/branches/filter-by-dirs/?rev=71986view=rev The test runner has a mapping of directory names to package names, and I use it in reverse to get the package name from the directory name. As a result you can use your shell's directory name completion for the test runner command line, and, for example, test.py -s src/zope now works. I would appreciate it if someone could check out this branch and run its tests on Windows. C:\filter-by-dirspython test.py Running unit tests: Ran 129 tests with 0 failures and 0 errors in 17.259 seconds. -- http://worldcookery.com -- Professional Zope documentation and training 2nd edition of Web Component Development with Zope 3 is now shipping! ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: test.py --list-modules
Gary Poster wrote: On Jan 12, 2007, at 6:33 PM, Marius Gedminas wrote: On Fri, Jan 12, 2007 at 05:17:18PM -0500, Benji York wrote: Marius Gedminas wrote: I have implemented a --list-modules option in a branch. It causes the test runner to apply package and module name filters, and then print the list of Python module names that would be imported. It is very fast because it doesn't actually import the modules. It is not very accurate because it does not pay attention to test name patterns or layer filtering. Would this option be useful to anybody? If its output were identical to the tests that were actually going to be selected, then I think it'd be worth including. That would be the proposed --list-tests option, wouldn't it? Or do you want a list of modules that have at least one test remaining after all the filtering (including --layer, --level and --test) is done? I might use --list-modules for test-running debugging (i.e., it's not finding my test--is it because it is not finding my module, or what?). That said, I can imagine more helpful tools for that use case (i.e., a way to verbosely list all filters in play...if there's not already something like that). That's a good use case for --list-modules, the only one I can see frankly. I'd really like --list-tests and wouldn't mind having to wait the extra five seconds to get an accurate list. So what's your use cases for all of these? * to see whether your test is actually found and run * to see whether certain test selection mechanisms work the way you expect them to -- http://worldcookery.com -- Professional Zope documentation and training 2nd edition of Web Component Development with Zope 3 is now shipping! ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: test.py --list-modules
I am also thinking about two other options: * --list-files -- essentially the same as --list-modules, but outputs file names instead of module names. * --list-tests -- apply package and module filters, import the modules, apply test name and layer filters, then list just the names of tests. Would these be useful to anybody? Yes a lot, I am using them in 2 cases, when I want to be sure that only 1-2 tests will be run before even trying to run them (with some functional tests running a lot of time I don't want to run more tests than I need). And with Zope3 testrunner's idea about the amount of doctests being run --list-tests would be the only way to find out how many real tests are being executes as at the moment I can't tell if my doctest has 11 tests in it or am I running some other test that matches the pattern. SchoolTool's custom test runner used to have --list-tests and --list-files. Now that SchoolTool has finally switched to the Zope 3 test runner, I wonder if anyone working on SchoolTool would miss those (hi, Ignas ;-). Yes especially on long winter evenings while running functional tests and wondering how many functional tests are there in this functional test layer I so long for the good old --list-files ;) Ignas ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: Zope 3.3.1
Christian Theune wrote: after the initial Zope 3.3 release, we fixed quite a few bugs. I think it's time for a 3.3.1 release now, which would also help the upcoming Zope 2.10.2 release. If anybody has time to do the 3.3.1 release, that would be very welcome! I'll make a 3.3.1 today. -- http://worldcookery.com -- Professional Zope documentation and training 2nd edition of Web Component Development with Zope 3 is now shipping! ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: zope.tal.xmlparser.XMLParser() dislikes unicode
Andreas Jung wrote:
Hi,
the XMLParser.parseString() method raises an exception
File /opt/python-2.4.4/lib/python2.4/unittest.py, line 260, in run
testMethod()
File
/Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/tests/test_xmlparser.py,
line 127, in test_xx
self._run_check(xml, ())
File
/Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/tests/test_xmlparser.py,
line 106, in _run_check
parser.parseString(source)
File
/Users/ajung_data/sandboxes/Zope/Zope/lib/python/zope/tal/xmlparser.py,
line 77, in parseString
self.parser.Parse(s, 1)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
43-48: ordinal not in range(128)
if the string to be parsed is a unicode strings and contains some non-ascii
chars. The following snippet from a private unittest (test_xmlparsers.py)
shows the error.
def test_xx(self):
xml = unicode('?xml version=1.0
encoding=utf-8?fooüöä/foo', 'iso-8859-15')
self._run_check(xml, ())
I am not sure if this behavior is intentional?! Is the XMLParser supposed
to deal with unicode strings or will it only accept a standard Python
string?
Traditionally, you parse an 8bit string, figure out its encoding (e.g.
from ?xml encoding=utf-8? and return some representation of that XML
with unicode data. That's why it's actually quite ok for XML parsers to
only accept string data.
With ZPTs it's a bit different: When editing ZPTs TTW for example, we
like to store its source in unicode. So it makes sense for us to be able
to parse unicode input as XML.
A workaround inside parseString() would to check for unicode
and convert the string on-the-fly to a Python string with utf-8 encoding.
This is possibly a limitation of the underlying Expat parser...any
recommendation how to deal with this issue?
Fixed it in 3.3 and trunk. If you had given me a bit more time, this
could even have been in 2.10.2b :). Oh well, I guess that's what 2.10.2
will be for ;)
--
http://worldcookery.com -- Professional Zope documentation and training
2nd edition of Web Component Development with Zope 3 is now shipping!
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: Zope 3.3.1
Philipp von Weitershausen wrote: Christian Theune wrote: after the initial Zope 3.3 release, we fixed quite a few bugs. I think it's time for a 3.3.1 release now, which would also help the upcoming Zope 2.10.2 release. If anybody has time to do the 3.3.1 release, that would be very welcome! I'll make a 3.3.1 today. Done. Can somebody please make a Windows installer? Adam? -- http://worldcookery.com -- Professional Zope documentation and training 2nd edition of Web Component Development with Zope 3 is now shipping! ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: zope.tal.xmlparser.XMLParser() dislikes unicode
Philipp von Weitershausen wrote at 2007-1-14 14:59 +0100: ... Traditionally, you parse an 8bit string, figure out its encoding (e.g. from ?xml encoding=utf-8? and return some representation of that XML with unicode data. That's why it's actually quite ok for XML parsers to only accept string data. Parsing usually means rebuilding the structure from a text string and *NOT* encoding guessing or Unicode decoding. Therefore, it is actually quite stupid for a parser to try to encode an already decoded string (i.e. a Unicode string) only that it can guess the encoding ;-) A halfway intelligent parser would accept Unicode when it gets it and concentrate on the remaining part of its task: either reporting structural events or building a parse tree. -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: zope.tal.xmlparser.XMLParser() dislikes unicode
Dieter Maurer wrote:
A halfway intelligent parser would accept Unicode when it gets it
and concentrate on the remaining part of its task: either reporting
structural events or building a parse tree.
The trivial fix I use in Twiddler is as follows:
if isinstance(source,unicode):
source = source.encode('utf-8')
Of course, this assumes a heading of either ?xml version=1.0
encoding=utf-8? or a missing encoding attribute, in which case the
xml spec states that the string must be utf-8 encoded.
The problem comes when someone sends you something like:
u'?xml version=1.0 encoding=something-else?node /'
What should be done then?
Chris
--
Simplistix - Content Management, Zope Python Consulting
- http://www.simplistix.co.uk
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Zope Sprints at PyCon
There will be a couple of Zope sprints at PyCon this year: http://us.pycon.org/TX2007/Sprints Benji York and I will be there on Monday and Tuesday, February 26 and 27. I'd like to work on Zope3 eggification with a goal of being able to assemble the Zope 3 application included in Zope 3 releases from a collection of eggs. This will be part of the Zope 3.4 sprint, http://us.pycon.org/TX2007/ Zope34Sprint. I hope t see some more folks there. Maybe there should be a Zope 2 sprint. BTW, PyCon Early Bird Registration ends soon! Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] server error in apidoc (encoding pb)
Hello,
When I put some accentuated character in the docstring of an interface
and I want to see the interface doc in apidoc-interfaces-search-IMyInterface,
I then get a server error:
File /usr/lib/python2.4/site-packages/zope/tales/tales.py, line 696, in
evaluate
return expression(self)
- /usr/lib/zope3/lib/python/zope/app/apidoc/ifacemodule/index.pt
- Line 23, Column 4
- Expression: PathExpr standard:u'view/getDoc'
- Names:
{'args': (),
'context': InterfaceClass zblog.category.interfaces.ICategoryContainer,
'default': object object at 0xb7db0548,
'loop': {},
'nothing': None,
'options': {},
'repeat': {},
'request': zope.publisher.browser.BrowserRequest instance
URL=http://localhost:9673/++apidoc++/Interface/zblog.category.interfaces.ICategoryContainer/index.html,
'template':
zope.app.pagetemplate.viewpagetemplatefile.ViewPageTemplateFile object at
0xb6a319ac,
'usage': zope.pagetemplate.pagetemplate.TemplateUsage object at
0xb5c3ea4c,
'view': zope.app.pagetemplate.simpleviewclass.SimpleViewClass from
/usr/lib/zope3/lib/python/zope/app/apidoc/ifacemodule/index.pt object at
0xb5c3ef4c,
'views': zope.app.pagetemplate.viewpagetemplatefile.ViewMapper object
at 0xb5bf9d6c}
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: zope.tal.xmlparser.XMLParser() dislikes unicode
On 14 Jan 2007, at 18:37 , Dieter Maurer wrote: Philipp von Weitershausen wrote at 2007-1-14 14:59 +0100: ... Traditionally, you parse an 8bit string, figure out its encoding (e.g. from ?xml encoding=utf-8? and return some representation of that XML with unicode data. That's why it's actually quite ok for XML parsers to only accept string data. Parsing usually means rebuilding the structure from a text string and *NOT* encoding guessing or Unicode decoding. Therefore, it is actually quite stupid for a parser to try to encode an already decoded string (i.e. a Unicode string) only that it can guess the encoding ;-) A halfway intelligent parser would accept Unicode when it gets it and concentrate on the remaining part of its task: either reporting structural events or building a parse tree. Yes, I agree. Unfortunately, expat isn't smart enough, which caused this whole discussion. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: zope.tal.xmlparser.XMLParser() dislikes unicode
On 14 Jan 2007, at 19:14 , Chris Withers wrote:
Dieter Maurer wrote:
A halfway intelligent parser would accept Unicode when it gets it
and concentrate on the remaining part of its task: either reporting
structural events or building a parse tree.
The trivial fix I use in Twiddler is as follows:
if isinstance(source,unicode):
source = source.encode('utf-8')
It's the same fix I used.
Of course, this assumes a heading of either ?xml version=1.0
encoding=utf-8? or a missing encoding attribute, in which case
the xml spec states that the string must be utf-8 encoded.
The problem comes when someone sends you something like:
u'?xml version=1.0 encoding=something-else?node /'
What should be done then?
Not sure. We could ignore it or raise an error. I'm inclined to
ignore it.
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: test.py -u -f
On Sat, Jan 13, 2007 at 01:37:17AM +0200, Marius Gedminas wrote: Due to an accident of implementation, if you pass both -u (run unit tests) and -f (run non-unit tests) to the test runner, it will not run any tests at all. That is not useful. I have a branch where test.py -fu is equivalent to specifying neither -f nor -u and runs all the tests: http://svn.zope.org/zope.testing/branches/test-fu/?rev=71988view=rev If there are no objections, I would like to merge that branch. Merged. Marius Gedminas -- It is a mess, pure and simple. Sucks to be away from Unix, huh? -- man perlfaq3 signature.asc Description: Digital signature ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: test.py --list-modules
Marius Gedminas wrote: On Fri, Jan 12, 2007 at 05:17:18PM -0500, Benji York wrote: OTOH, being not very accurate would make the feature as implemented a bit of an attractive nuisance. I phrased it badly. s/It is not very accurate/It may not match user expectations/. Ahh, I see now. As for the question then, I would prefer it list only the tests that would be run. Perhaps a separate switch could be provided that would list all tests found. Then a person trying to debug their test not being run would know if the test runner found it or not, and if so, if a filter precluded it from running. -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: test.py -s src/zope/something
On Sat, Jan 13, 2007 at 01:44:17AM +0200, Marius Gedminas wrote: A long time ago test.py -s accepted directory names. Later it was changed to take package names instead. There is some backwards-compatibility code that replaces slashes with dots, but it is incomplete: test.py -s src/zope would try to run the tests for a nonexistent package src.zope. I have expanded the backwards-compatibility/convenience code in a branch: http://svn.zope.org/zope.testing/branches/filter-by-dirs/?rev=71986view=rev Merged. Marius Gedminas -- phrakture i learned vim by reading the source code. am I hardcore yet? Axioplase phrakture: not enought yet... read it rot13'd, and you'll be hardcore ^^ phrakture I read it as analog output on an oscilliscope mgedmin rot13, bah, try reading the gzipped source tarball Zathrus encrypted. lack standing on your head. signature.asc Description: Digital signature ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: test.py --list-modules
On Fri, Jan 12, 2007 at 11:30:15PM +0200, Marius Gedminas wrote: The Zope 3 test runner can filter tests by package, module, test name and layer. Sometimes your expectations do not match reality and either the test you want is not being run, or more tests than you wanted are being run. When you want to debug filter patterns it helps if you can see what tests would be run without wasting time actually running them. I have implemented a --list-modules option in a branch. It causes the test runner to apply package and module name filters, and then print the list of Python module names that would be imported. It is very fast because it doesn't actually import the modules. I have now implemented --list-tests in a different branch. That option pays attention to all of -t, -m, -s, -u, -f, --layer, --level options. http://svn.zope.org/zope.testing/branches/list-tests/?rev=72034view=rev I intend to merge this one, unless somebody suggests a cleaner way of implementing it. I do not think I care enough about --list-modules or --list-files to push those to completion. Marius Gedminas -- If all else fails, read the documentation. signature.asc Description: Digital signature ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re[2]: Zope 3.3.1
Hello Philipp, Let's try. Hope that .../repos/main/Zope3/tags/3.3.1 is the right one. Sunday, January 14, 2007, 3:24:45 PM, you wrote: PvW Philipp von Weitershausen wrote: Christian Theune wrote: after the initial Zope 3.3 release, we fixed quite a few bugs. I think it's time for a 3.3.1 release now, which would also help the upcoming Zope 2.10.2 release. If anybody has time to do the 3.3.1 release, that would be very welcome! I'll make a 3.3.1 today. PvW Done. Can somebody please make a Windows installer? Adam? -- Best regards, Groszer Adam -- Quote of the day: I wish you humans would leave me alone. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: Zope 3.3.1
On 15 Jan 2007, at 08:46 , Adam Groszer wrote: Hello Philipp, Let's try. Hope that .../repos/main/Zope3/tags/3.3.1 is the right one. Yup. Thanks! Lemme know when the installer is uploaded, I can then publish the files and send the announcement. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
