Thanks for the detailed investigation. Before getting into the
contextualized responses below, did you use PyXB 1.1.3 for that, or
were you using the current development version (the next branch from
git)? I'm seeing some differences in behavior locally, and don't know
whether it's because of the large number of changes to PyXB's
date/time classes since the last release, or environment differences.
In particular, it appears to me that datetime.datetime.now() and
xsd.dateTime.now() are essentially the same:
llc[21]$ cat /tmp/p1.py
import datetime
import pyxb.binding.datatypes as xsd
def showdt (dt):
print 'class %s\n\ttimezone %s\n\tstr %s' % (type(dt), dt.tzinfo, str(dt))
showdt(datetime.datetime.now())
showdt(xsd.dateTime.now())
llc[22]$ python /tmp/p1.py
class <type 'datetime.datetime'>
timezone None
str 2012-05-15 06:30:12.653698
class <class 'pyxb.binding.datatypes.dateTime'>
timezone None
str 2012-05-15 06:30:12.653771
llc[23]$
On Tue, May 15, 2012 at 1:14 AM, Nathan Robertson <nath...@nathanr.net> wrote:
> Hi Peter,
>
> On 14 May 2012 20:49, Peter Bigot <big...@acm.org> wrote:
>> I'm not sure. As you saw, http://www.w3.org/TR/xmlschema-2/#dateTime
>> is pretty clear that anything that had a timezone is to be treated as
>> UTC. If the original lexical space didn't have a timezone, I think I
>> set it up to be given the local timezone; it might be argued that that
>> is a mistake, but it might also have been necessary for consistency.
>>
>> If you would file this as a trac request I'll see what I can do.
>> Please make it clear whether you just want xsd.dateTime.now() to be
>> symmetrical or whether you need values that lack a timezone to do
>> something other than what they do now.
>
> Ok, so I wrote a Java / JAXB version of my python test posted late
> yesterday to the mailing list. I'm using Python 2.7.3, PyXB 1.1.3,
> Java 7u4 (which includes JAXB 2.2.4). The code reads in an XML string,
> prints out the value it got, then converts it back to XML and prints
> out that string. Here's the result - first in Python/PyXB, then
> Java/JAXB:
>
> (pyenv-zktraining)nathanr@gionta:~/work/junkcode/pyxb-datetime-test>
> python test2.py
> datetime.datetime.now() prints 2012-05-15 15:16:12.167164
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp>
> --> 2012-05-14 06:36:02.157000 --> <?xml version="1.0"
> ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp>
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp>
> --> 2012-05-14 06:36:02.157000 --> <?xml version="1.0"
> ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp>
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp>
> --> 2012-05-14 16:36:02.157000 --> <?xml version="1.0"
> ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp>
>
> (pyenv-zktraining)nathanr@gionta:~/work/junkcode/pyxb-datetime-test> java
> Test2
> Calendar.getInstance() prints: 2012-05-15 15:16:14.612
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157+10:00</ns1:timestamp>
> --> 2012-05-14 16:36:02.157 --> <?xml version="1.0" encoding="UTF-8"
> standalone="yes"?><timestamp
> xmlns="http://test.com/test";>2012-05-14T16:36:02.157+10:00</timestamp>
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T06:36:02.157Z</ns1:timestamp>
> --> 2012-05-14 06:36:02.157 --> <?xml version="1.0" encoding="UTF-8"
> standalone="yes"?><timestamp
> xmlns="http://test.com/test";>2012-05-14T06:36:02.157Z</timestamp>
> <?xml version="1.0" ?><ns1:timestamp
> xmlns:ns1="http://test.com/test";>2012-05-14T16:36:02.157</ns1:timestamp>
> --> 2012-05-14 16:36:02.157 --> <?xml version="1.0" encoding="UTF-8"
> standalone="yes"?><timestamp
> xmlns="http://test.com/test";>2012-05-14T16:36:02.157+10:00</timestamp>
> (pyenv-zktraining)nathanr@gionta:~/work/junkcode/pyxb-datetime-test>
>
>
> So, for the purposes of below:
>
> - I will call yyyy-mm-ddThh:mm:ss.mmm+TZ:TZ "format 1" (timestamp with
> +timezone on the end)
This is a non-canonical representation in XML.
> - I will call yyyy-mm-ddThh:mm:ss.mmmZ "format 2" (Z on the end,
> signifying GMT time)
This is the canonical representation for timezoned objects in XML.
> - I will call yyyy-mm-ddThh:mm:ss.mmm "format 3" (no time zone)
This is the canonical representation for non-timezoned objects.
PyXB should always use the canonical representation when converting a
value to text in a context where it will be used in XML, i.e. via
v.xsdLiteral(). It will defer to the underlying Python implementation
when converting a value to text by implicit or explicit use of str().
>
> - Given an xsd:dateTime with format 1, PyXB will print the object out
> in format 3, in GMT time, and marshal the date back out to XML in
> format 2.
> - Given an xsd:dateTime with format 1, JAXB will print the object out
> in format 3, in local time, and marshal the date back out to XML in
> format 1.
>
> - Given an xsd:dateTime with format 2, PyXB will print the object out
> in format 3, in GMT time, and marshal the date back out to XML in
> format 2.
> - Given an xsd:dateTime with format 2, JAXB will print the object out
> in format 3, in GMT time, and marshal the date back out to XML in
> format 2.
>
> - Given an xsd:dateTime with format 3, PyXB will print the object out
> in format 3, in local time, and marshal the date back out to XML in
> format 3.
> - Given an xsd:dateTime with format 3, JAXB will print the object out
> in format 3, in local time, and marshal the date back out to XML in
> format 1 (assuming local time by default)
>
>
> Notes on the above:
>
> 1. JAXB handling of format 2 ("GMT time with trailing Z") appears
> inconsistent. Why they would represent the object to client code in
> GMT in that case, but as local time in the case of format 1 input,
> I'll never know, and makes JAXB Calendar's inconsistent with the rest
> of the Java platform when using format 2 input. I think this is
> probably a JAXB bug. So, as a JAXB consumer, it is better to have
> format 1 input, as JAXB Calendar's with format 1 input are symmetrical
> and compatible with Calendar.getInstance() generated (standard
> platform) calendars.
>
> 2. Python printing out pyxb.binding.datatypes.dateTime's as GMT when
> it prints out datetime.datetime's as local time is inconsistent. It
> should represent itself to code (and print out its value) as local
> time, not GMT. It should be compatible with datetime.datetime, which
> it extends, and hence can be (and does get) used in place of in client
> code. For mine, this is a PyXB bug, and is causing us problems in our
> code at the moment.
I believe this is fixed.
> 3. It's debatable what to do with format 3, but at least PyXB and JAXB
> read it in the same. I think the current decision on handling format 3
> in PyXB is probably the right one. My personal opinion is that JAXB
> printing out in format 1 and not format 3 in the case of format 3
> input is a mistake, but I can see arguments both ways. The argument
> that "it's because JAXB always outputs format 1" is actually wrong,
> because it doesn't do that for format 2 input.
>
>
> So, there are two things I think need to be looked at for changing:
>
> 1. For interoperability, it would be nice if PyXB could print out in
> format 1. Whether it be by default, or outputting in format 1 as a
> configurable option. Those interoperating with JAXB code will want to
> have PyXB output format 1, for sure.
"Print out" in this case means what comes out of str(v) where v is an
instance of the class? Because AFAICT that's just deferring to the
underlying Python datetime.datetime.implementation which PyXB's
dateTime class inherits. I can't really change that.
The other issue is that http://www.w3.org/TR/xmlschema-2/#dateTime is
extremely clear:
All timezoned times are Coordinated Universal Time (UTC, sometimes called
"Greenwich Mean Time"). Other timezones indicated in lexical representations
are converted to UTC during conversion of literals to values.
As such, there's nothing that will retain the +10:00 from the input
once it's made into a value. PyXB retains fact-of timezone by having
a non-None tzinfo attribute, but the fact that the offset is zero
inhibits Python from printing it.
If you use my test program and add:
showdt(xsd.dateTime.today().aslocal())
you can see behavior more like what you may want:
class <type 'datetime.datetime'>
timezone <pyxb.utils.utility.LocalTimeZone object at 0x260db50>
str 2012-05-15 02:02:27.455888-05:00
Note that you have to use today() because now() produces a
non-timezoned time, and the addition of the timezone screws things up.
I'm not sure why I made PyXB's today() overload explicitly include a
timezone. At the moment, that seems like a mistake, and I should have
used a different name for that, since the documentation for
datetime.today() explicitly says the tzinfo in the result is None.
http://docs.python.org/library/datetime.html#datetime-objects
> 2. pyxb.binding.datatypes.dateTime should represent itself to client
> code in a compatible way to datetime.datetime such that it can be a
> drop in replacement (particularly given it subclasses it). That means
> it should represent itself to client code as local time, not GMT.
>
What I tried up at the top suggests that PyXB does do this. It may be
that some locale information in your environment is affecting the
underlying Python behavior, though, causing PyXB's results to have a
timezone attached. (I'm using Python 2.7.3 on Linux.)
The other conflict is that I suspect Python is not displaying the
effect of any timezone information, at least in str(), which is why
things look like local time even when there is a timezone attached.
> Peter - are you ok with me opening two Trac tickets for the above two
> suggestions? Both are issues that will cause us problems for our July
> production release I referenced in another email earlier today.
>From my tests I believe PyXB's doing the overall right thing. If you
still have problems when using the pre-release 1.1.4 material in git,
let's keep discussing this.
One possibility I would entertain is an option to prevent PyXB from
performing the normalize-to-UTC step when processing timezoned times.
In that case, the tzinfo information would be present and not have a
zero offset, and the underlying Python's datetime class should provide
the preserved timezone when being converted via str().
I'm not absolutely convinced that this is a great idea, though, since
it's so incredibly easy to screw things up when dealing with
timezones.
> The Python code was in a previous posting - I'm providing the
> Java/JAXB code below for completeness / mailing list archiving
> purposes.
>
> Regards,
> Nathan.
>
> --
>
> import generated.*;
> import javax.xml.*;
> import javax.xml.bind.*;
> import javax.xml.datatype.*;
> import javax.xml.validation.*;
> import java.io.*;
> import java.net.*;
> import java.util.*;
>
> /*
> $ xjc -d . -p generated schema.xsd
> $ javac Test2.java
> $ java Test2
>
> schema.xsd is:
> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema";
> targetNamespace="http://test.com/test";
> xmlns:zenkai="http://test.com/test";>
> <xsd:element name="timestamp" type="xsd:dateTime"/>
> </xsd:schema>
> */
>
> public class Test2 {
> private static final String[] XML_LIST = {
> "<?xml version=\"1.0\" ?><ns1:timestamp
> xmlns:ns1=\"http://test.com/test\";>2012-05-14T16:36:02.157+10:00</ns1:timestamp>",
> "<?xml version=\"1.0\" ?><ns1:timestamp
> xmlns:ns1=\"http://test.com/test\";>2012-05-14T06:36:02.157Z</ns1:timestamp>",
> "<?xml version=\"1.0\" ?><ns1:timestamp
> xmlns:ns1=\"http://test.com/test\";>2012-05-14T16:36:02.157</ns1:timestamp>"
> };
>
> public static void main(String args[]) throws Exception {
> Calendar c = Calendar.getInstance();
> System.out.format("Calendar.getInstance() prints: %1$tF
> %1$tR:%1$tS.%1$tL", c);
> System.out.println("");
>
> ObjectFactory of = new ObjectFactory();
> JAXBContext jc = JAXBContext.newInstance("generated");
> Unmarshaller u = jc.createUnmarshaller();
> Marshaller m = jc.createMarshaller();
> m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.FALSE);
>
> URL url = new
> Test2().getClass().getClassLoader().getResource("schema.xsd");
> SchemaFactory sf =
> SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
> Schema schema = sf.newSchema(url);
> u.setSchema(schema);
>
> for (String xml: XML_LIST) {
> System.out.print(xml);
>
> GregorianCalendar timestampIn = ((XMLGregorianCalendar)
> ((JAXBElement<?>) u.unmarshal(new
> ByteArrayInputStream(xml.getBytes()))).getValue()).toGregorianCalendar();
> System.out.format(" --> %1$tF %1$tR:%1$tS.%1$tL --> ",
> timestampIn);
>
> StringWriter sw = new StringWriter();
>
> m.marshal(of.createTimestamp(DatatypeFactory.newInstance().newXMLGregorianCalendar(timestampIn)),
> sw);
> System.out.println(sw.toString());
> }
> }
> }
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
pyxb-users mailing list
pyxb-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pyxb-users
