On Tue, May 15, 2012 at 9:25 PM, Nathan Robertson <nath...@nathanr.net> wrote: > Hi Peter, > > On 15 May 2012 22:10, Peter Bigot <big...@acm.org> wrote: >> Thanks for the detailed investigation. Before getting into the >> contextualized responses below, did you use PyXB 1.1.3 for that, or >> were you using the current development version (the next branch from >> git)? I'm seeing some differences in behavior locally, and don't know >> whether it's because of the large number of changes to PyXB's >> date/time classes since the last release, or environment differences. > > I was using PyXB 1.1.3. I repeated the tests against the git-next > branch this morning using Python 2.7.3 (compiled from source) on SuSE > Linux Enterprise 11 SP2 x64. Here's what I get (first block is > 1.1.4-DEV; second block is 1.1.3): > > (pyenv-pyxb-next)nathanr@gionta:~/work/junkcode/pyxb-datetime-test> > python test2.py > datetime.datetime.now() prints 2012-05-16 09:38:58.946262 > Using PyXB 1.1.4-DEV, Python 2.7.3 (default, Apr 30 2012, 14:51:37) > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> > --> 2012-05-14 06:36:02.157000+00:00 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > --> 2012-05-14 06:36:02.157000+00:00 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> > --> 2012-05-14 16:36:02.157000 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> > (pyenv-pyxb-next)nathanr@gionta:~/work/junkcode/pyxb-datetime-test> > > > (pyenv-zktraining)nathanr@gionta:~/work/junkcode/pyxb-datetime-test> > python test2.py > datetime.datetime.now() prints 2012-05-16 09:45:35.325685 > Using PyXB 1.1.3, Python 2.7.3 (default, Apr 30 2012, 14:51:37) > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> > --> 2012-05-14 06:36:02.157000 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > --> 2012-05-14 06:36:02.157000 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> > --> 2012-05-14 16:36:02.157000 --> <?xml version="1.0" > ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> > (pyenv-zktraining)nathanr@gionta:~/work/junkcode/pyxb-datetime-test> > > > The only difference between the behaviour in 1.1.3 and 1.1.4-DEV is > that "+00:00" is printed, and confirms that it is printing GMT for > some reason.
Yes. The error in 1.1.3 was that the tzinfo field was not being set to preserve the fact that the time had been converted to UTC. Python will print the tzinfo field if it is non-None. > Is this environmental or something to do with my > configuration? The Java version of these tests picks up I'm +10:00, > and I also re-ran the tests on 1.1.4-DEV with both the SuSE provided > Python 2.6.0 and on Ubuntu 11.10 with their packaged version of Python > 2.7.2 and the output is still the same (+00:00). What do you get in > your environment? As best I can tell with gmail's wrapping (which is a hassle for me too), I get the same thing as you do. >[...] >> PyXB should always use the canonical representation when converting a >> value to text in a context where it will be used in XML, i.e. via >> v.xsdLiteral(). It will defer to the underlying Python implementation >> when converting a value to text by implicit or explicit use of str(). > > Section 3.2.7.3 (http://www.w3.org/TR/xmlschema-2/#dateTime) suggests > that canonical and non-canonical formats equivalent and > interchangeable. It would be a feature to be able to configure which > of the formats XML generated by PyXB will produce. It needn't be the > same time zone as the input - it makes sense to me to output > non-canonical times with respect to the local time zone, not the > inputted time zone (if any). The question here is expectation. Personally, I would expect UTC or the original timezone. >>> 2. Python printing out pyxb.binding.datatypes.dateTime's as GMT when >>> it prints out datetime.datetime's as local time is inconsistent. It >>> should represent itself to code (and print out its value) as local >>> time, not GMT. It should be compatible with datetime.datetime, which >>> it extends, and hence can be (and does get) used in place of in client >>> code. For mine, this is a PyXB bug, and is causing us problems in our >>> code at the moment. >> >> I believe this is fixed. > > Perhaps the issue I'm seeing is environmental. As I noted above, could > you re-run the test I'm running (I'll include the code at the bottom > of this post) and confirm whether you're seeing the same behaviour as > me? > > >> If you use my test program and add: >> >> showdt(xsd.dateTime.today().aslocal()) >> >> you can see behavior more like what you may want: >> >> class <type 'datetime.datetime'> >> timezone <pyxb.utils.utility.LocalTimeZone object at 0x260db50> >> str 2012-05-15 02:02:27.455888-05:00 >> >> Note that you have to use today() because now() produces a >> non-timezoned time, and the addition of the timezone screws things up. >> I'm not sure why I made PyXB's today() overload explicitly include a >> timezone. At the moment, that seems like a mistake, and I should have >> used a different name for that, since the documentation for >> datetime.today() explicitly says the tzinfo in the result is None. >> http://docs.python.org/library/datetime.html#datetime-objects > > Now that explains a lot, actually. When I add aslocal() in, I get: > > <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> > --> 2012-05-14 06:36:02.157000+00:00 2012-05-14 16:36:02.157000+10:00 > --> <?xml version="1.0" ?><ns1:timestamp > xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> > > Which would work, except aslocal() doesn't exist in the base datetime type: > >>>> print datetime.datetime.now() > 2012-05-16 10:35:10.274615 >>>> print datetime.datetime.now().aslocal() > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > AttributeError: 'datetime.datetime' object has no attribute 'aslocal' >>>> Yes, aslocal() was a mistake, added in part to help me understand the same sort of issues you're seeing. (E.g., when you see an absolute time as HH:MM:SS + TZ, does the TZ indicate that the offset has already been applied, or needs to be applied, in order to present the time as UTC? Huge difference, and at the moment misuse of aslocal() can easily get it wrong.) > This morning we decided to make a change to our Java code to always > output dates with no timezone information, and PyXB puts out > datetime.datetime objects without tzinfo set as without timezone > information as well. So totally turning off time zones information in > both directions is a workaround that we can work with in our > particular situation. Not knowing your situation, it's good you have a workaround. Personally, my gut clenches at the thought of eliminating timezones from the data. It really complicates correlation between events unless you're entirely within a single timezone that does not observe daylight savings time. Please review the enhancement I've made, and see if you can change your mind. >> One possibility I would entertain is an option to prevent PyXB from >> performing the normalize-to-UTC step when processing timezoned times. >> In that case, the tzinfo information would be present and not have a >> zero offset, and the underlying Python's datetime class should provide >> the preserved timezone when being converted via str(). >> >> I'm not absolutely convinced that this is a great idea, though, since >> it's so incredibly easy to screw things up when dealing with >> timezones. > > I would have thought this would have been the right way. If I'm > understanding you correctly, this should make pyxb's dateTime object > symmetrical and interchangable with the base datetime object, which > should be a design goal. Consistency with similar Python data types is a secondary goal. The primary goal is conformance to XML. This is why xsd:date is now based on datetime.datetime instead of datetime.date: XML supports the concept of a timezoned date, and Python's date type doesn't allow that. I don't think I've explained this well: XML times do not have a time zone as part of the value space. All values are converted to UTC as soon as they're translated into Python. This is why PyXB can't reconstruct your +10:00 offset: it's not there anymore. The tzinfo attribute is used solely to prove that the value is in fact in UTC, and is not a non-timezoned time (what Python calls a "naive time") for which zone information is not available. A non-timezoned time could technically be anywhere within a 28 [sic] hour period, given XML's constraint that timezones have a magnitude of no more than 14 hours. To make this more clear and because the feature makes sense, I've gone ahead and created a ticket for and implemented the customization point I proposed. Using pyxb.PreserveInputTimeZone(True) will inhibit the automatic canonicalization, keeping the original tzinfo in the value space. There is a cost of potential interoperability issues with other XML processors, and code errors if whatever uses the value doesn't do timezone-aware manipulations. Use "git pull" to update your 1.1.4-DEV workspace. The result is different from your proposal earlier that PyXB convert times to the non-canonical form representing the local timezone for output. I'd be somewhat more reluctant to do that, as it would involve actively manipulating the data in a manner not supported by the specification. If you want that, please add a ticket, but I'd probably implement it by adding a hook mechanism to allow a developer to override how PyXB creates the xsdLiteral form. Expected results from a modified version of your test program are: datetime.datetime.now() prints 2012-05-16 09:51:04.584878 Using PyXB 1.1.4-DEV, Python 2.7.3 (default, Apr 13 2012, 20:16:59) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> --> 2012-05-14 06:36:02.157000+00:00 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> --> 2012-05-14 06:36:02.157000+00:00 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> --> 2012-05-14 16:36:02.157000 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> Preserving input time zone: <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> --> 2012-05-14 16:36:02.157000+10:00 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> --> 2012-05-14 06:36:02.157000+00:00 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> --> 2012-05-14 16:36:02.157000 --> <?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp> The program itself is below. import sys import types import datetime import pyxb.binding.generate import os.path xsd='''<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"; targetNamespace="http://test.com/test"; xmlns:zenkai="http://test.com/test";> <xsd:element name="timestamp" type="xsd:dateTime"/> </xsd:schema>''' #file('schema.xsd', 'w').write(xsd) code = pyxb.binding.generate.GeneratePython(schema_text=xsd) #file('code.py', 'w').write(code) rv = compile(code, 'test', 'exec') eval(rv) xml_list = ('<?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp>', '<?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp>', '<?xml version="1.0" ?><ns1:timestamp xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp>',) print "datetime.datetime.now() prints {0}".format(datetime.datetime.now()) print "Using PyXB {0}, Python {1}".format(pyxb.__version__, sys.version) for xml in xml_list: obj = CreateFromDocument(xml) print "{0} --> {1} --> {2}".format(xml, obj, obj.toxml()) print 'Preserving input time zone:' pyxb.PreserveInputTimeZone(True) for xml in xml_list: obj = CreateFromDocument(xml) print "{0} --> {1} --> {2}".format(xml, obj, obj.toxml()) ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ pyxb-users mailing list pyxb-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pyxb-users
