Rob,
I won't be able to do a fix one this soon. I'm leaving on a
two-week vacation in two days, and this issue will require a bit
more work than I'll have time for before then. I'm hoping you can
make do with what you've got, at least temporarily..
In the meantime, here are a few notes on my analysis of the problem.
Maybe it will help remind me of what to look at when I get back.
Or, perhaps will help someone solve this.
A few preliminaries:
- Namespace prefixes can be arbitrarily chosen and modified, if done
consistently.
- A single schema can be defined in multiple files/docs, some of
which define different namespaces. The same name (a top level
`xs:element` or `xs:complexType`) can be defined in multiple
namespaces. The directive `xs:import` is used to pull these
separate documents/namespaces into a single schema.
- A file or document can define a single namespace. The
`targetNamespace` attribute is used to specify that namespace.
And, the `xmlns:xxx` attribute is used to associate a namespace
prefix with that (target) namespace.
- `generateDS.py` uses `process_includes.py` to pull the separate
documents that define a schema into a single (Lxml) element tree.
It does this by traversing each `xs:import` and `xs:include`
directive, recursively.
Looking for what to fix and how to fix it:
1. What is the problem? -- Some elements are being exported with the
wrong namespace prefix. It's the namespace prefix of the
enclosing element.
2. Where does this happen? -- The code that causes the error is
generated by `generateExportChildren` and several functions that
it calls: possiblly `generateExportFn_1`, `generateExportFn_2`, and
`generateExportFn_3`.
3. When I use `ipdb`/`pdb` and put a break-point in those functions,
I find that the information we need about the namespace is not
there. (Note that `generateDS.py` maintains a dictionary that
maps namespaces to namespace prefixes; so, if we had the
namespace, we could look up the prefix.)
4. Possible places where we'd need to do a fix:
- Maybe the method `startElement` in class `XschemaElement` is
not collecting the namespace information we need.
- And, perhaps `process_includes.py` is not saving the
information we need. Note that, since `process_includes.py`
mashes all those documents into a single document and element
tree, the `targetNamespace` (which is implied for each schema
document) is lost. Perhaps `process_includes.py` could be
enhanced so that it collects information (a dictionary?) that
associates the target namespace with the (top level) elements
(`xs:element` and `xs:complexType`) defined within it.
And, we should keep in mind that, since XML schema namespaces enable
you to define the same name differently in multiple namespaces and
since `generateDS.py` generates code that define classes in a
*single* Python namespace, we still will not be able to handle those
duplicate (top-level type names).
More on this next year. In the meantime, hope you have a great
holiday. And, thanks much for your work and help with this.
Dave
On Thu, Dec 14, 2017 at 04:44:34PM -0800, Rob Calvert wrote:
> v2.29.4 allowed me to remove some of my hand-edits, and it
> definitely seems to have fixed the problem that required me to set
> original_tagname_. Thanks! (I see what you mean about how the
> passed “name_” parameter was the cause of the inapproprately-named
> tags.)
>
> I still find that I'm forced to hand-edit the generated
> oadr_20b.py in a few places, though:
>
> The PayloadFloat class exports tags in the “oadr” namespace; it
> should export tags in the “ei” namespace. The PowerReal class
> exports tags in the “oadr” namespace; it should export tags in the
> “power” namespace.
> The intervals class lacks a “strm” namespacedef.
>
> Those first two classes participate in the schema’s polymorphism
> (they’re concrete subclasses attached to variables named for the
> superclass), so I’m guessing that this wrong-namespace issue is
> also caused by the polymorphism, but is perhaps only marginally
> related to the problem that you just solved.
>
> — Rob
>
> > On Dec 14, 2017, at 3:07 PM, Dave Kuhlman <dkuhl...@davekuhlman.org> wrote:
> >
> > Rob,
> >
> > I've made a fix for the export of child elements declared as
> > instances of an abstract type. The new code is here:
> >
> > https://bitbucket.org/dkuhlman/generateds
> >
> > Could you give it a try and see if it fixes this issue. I
> > eye-balled the generated code. That seems to be what we want. And,
> > I also adapted the oadr_test.py file that you sent so that I could
> > run it with code generated before and after this fix. That seems to
> > produce what we want.
> >
> > So, it seems to fix our problem and it does not seem to create new
> > problems.
> >
> > A few more comments follow ...
> >
> > Thanks for your notes, below. They helped focus me on where I
> > believe the problem is.
> >
> > You are correct that we should not have to use original_tagname_.
> > We are using polymorphism here, and the concrete (non-abstract)
> > instance knows its class and knows how it should export itself.
> >
> > The problem, I believe, is that the containing class (IntervalType,
> > for example), when it asks its child to export itself, is passing in
> > its own value for the `name_` parameter and, by doing so, is
> > overriding the correct value, which is the default for the `name_`
> > parameter.
> >
> > In the generated export methods (`export` and `exportChildren`, in
> > particular), the `original_tagname_` class variable, if not None, is
> > used to override the value of `name_`. That's why your temporary
> > fix works. So, if I can straighten out the value of `name_`, which
> > is being passed to the child, perhaps we'll have a fix.
> >
> > Sometimes, the container needs to pass the `name_` parameter.
> > But, if the (static) type of the child is an abstract class/type,
> > then, as you mention below, the container can't know the correct
> > value. I'll focus on that.
> >
> > Dave
> >
> > On Wed, Dec 13, 2017 at 09:35:32PM -0800, Rob Calvert wrote:
> >> Thanks for your efforts in trying to unravel why this problem occurred.
> >>
> >> I think you ended up saying this already, but let me try to
> >> re-state things just for clarity. On the sending end, I
> >> instantiate concrete subclasses (oadrReportPayload, PayloadFloat,
> >> oadrPayloadResourceStatus), but I have to attach those instances
> >> to their containers using variables (streamPayloadBase,
> >> payloadBase) that are named for the abstract superclass -- those
> >> are the variable names that have been provided in the generated
> >> oadr_20b.py. It makes sense to me that those are the variable
> >> names; the variables can’t be named for the concrete classes,
> >> since due to the polymorphism, we can’t predict in advance which
> >> concrete class instance will be sent in each specific case.
> >>
> >> On the receiving end, I’d expect build() to apply some sort of
> >> hint when deciding which concrete class to instantiate.
> >> original_tagname_ provides that hint, and gives me working code,
> >> but as you said below, it seemed somewhat odd that I was required
> >> to set original_tagname_ in the instance before exporting it; in a
> >> perfect world I’d have expected oadr_20b.py to “know” that hint
> >> already by other means.
> >
> >>
> >> Rob
> >>
> >>> On Dec 13, 2017, at 8:06 PM, Dave Kuhlman <dkuhl...@davekuhlman.org>
> >>> wrote:
> >>>
> >>> Rob,
> >>>
> >>> Wait, I think I have a clue. Of course, I thought I had a different
> >>> clue 3 hours ago.
> >>>
> >>> I do not think you should have to use original_tagname_, but I need
> >>> to investigate more. Thanks for that hint.
> >>>
> >>> Ignore the rest. It's just me trying to grope through the problem.
> >>>
> >>> More tomorrow.
> >>>
> >>> Dave
> >>>
> >>> I'm trying to localize the following problem:
> >>>
> >>>> The XSD uses polymorphism, reflected in the generated code as class
> >>>> hierarchies.
> >>>>
> >>>> In some cases, exported XML nodes are getting named based on their
> >>>> abstract superclasses:
> >>>>
> >>>> - streamPayloadBase
> >>>> - payloadBase
> >>>>
> >>>> instead of based on the concrete subclasses
> >>>>
> >>>> - oadrReportPayload
> >>>> - PayloadFloat
> >>>> - oadrPayloadResourceStatus
> >>>>
> >>>> As you can see in oadr_test.py, I can avoid this issue by setting
> >>>> the Python instance’s “original_tagname_”. I don't know, though,
> >>>> whether that's the recommended solution — I couldn’t find
> >>>> documentation about the purpose of original_tagname_. Should I have
> >>>> expected the XML nodes to be named correctly without calling
> >>>> original_tagname_? Is there something else that I should have done
> >>>> to avoid the incorrect names?
> >>>
> >>> Is this problem occuring in `IntervalType`? `IntervalType` is
> >>> defined as follows:
> >>>
> >>> <xs:complexType name="IntervalType">
> >>> <xs:sequence>
> >>> <xs:element ref="xcal:dtstart" minOccurs="0"/>
> >>> <xs:element ref="xcal:duration" minOccurs="0"/>
> >>> <xs:element ref="xcal:uid" minOccurs="0"/>
> >>> <xs:element ref="strm:streamPayloadBase" maxOccurs="unbounded"/>
> >>> </xs:sequence>
> >>> </xs:complexType>
> >>>
> >>> But, as you mention, `streamPayloadBase` is an abstract type.
> >>> So, the above erroneous. We should not be able to define an
> >>> instance of an abstract type. After all, how would we know what
> >>> it's concrete type is? Which subclass is it an instance of?
> >>>
> >>> It seems to me that in xs:complexType `IntervalType`, each child
> >>> needs to be defined as a concrete (i.e. not an abstract) type.
> >>>
> >>> So, in oadr_test.py, when you create an instance of the
> >>> `streamPayloadBase` class and add that instance to the instance of
> >>> the `IntervalType`, there is no way for the `IntervalType` to know
> >>> the (concrete) type of the `streamPayloadBase` instance.
> >>>
> >>> But, wait, you are creating an instance of the
> >>> `oadrReportPayloadType` class, which *is* a concrete type.
> >>>
> >>> So, our problem is that the `IntervalType` should not be telling
> >>> that instance its name.
> >>>
> >>> It might be that there is another way around this using xsi:type in
> >>> the XML instance document, but I don't really understand how that is
> >>> supposed to work.
> >>>
> >>> Or, am I looking at the wrong definition and class?
> >>>
> >>> I'll keep looking. I do have a tendency to try to blame others for
> >>> my own problems.
> >>>
> >>>
> >>> Dave
> >>>
> >>> On Tue, Dec 12, 2017 at 12:38:18AM +0000,
> >>> generateds-users-ow...@lists.sourceforge.net wrote:
> >>>> As list administrator, your authorization is requested for the
> >>>> following mailing list posting:
> >>>>
> >>>> List: generateds-users@lists.sourceforge.net
> >>>> From: r...@kisensum.com
> >>>> Subject: Fwd: original_tagname_, etc.
> >>>> Reason: Message body is too big: 2353427 bytes with a limit of 40 KB
> >>>>
> >>>> At your convenience, visit:
> >>>>
> >>>> https://lists.sourceforge.net/lists/admindb/generateds-users
> >>>>
> >>>> to approve or deny the request.
> >>>
> >>>> Date: Mon, 11 Dec 2017 16:38:02 -0800
> >>>> From: Rob Calvert <r...@kisensum.com>
> >>>> To: generateds-users@lists.sourceforge.net
> >>>> Subject: Fwd: original_tagname_, etc.
> >>>> X-Mailer: Apple Mail (2.3273)
> >>>>
> >>>> Dave, I quickly tried out your new 2.29.3 version of generateDS,
> >>>> wondering whether it might clear up some things I’m currently doing to
> >>>> work around generated-code issues. It hasn’t changed what I’m seeing,
> >>>> though, so I’m seeking your advice.
> >>>>
> >>>> Before describing the current problem, let me thank you for helping my
> >>>> colleague Bob Barcklay last month. The 2.29.0 version of generateDS
> >>>> solved some of what we’d been hitting, allowing us to meet a deliverable
> >>>> deadline. So, on to the current set of issues...
> >>>>
> >>>> I've implemented an XML protocol that uses the OpenADR v2.0b schema. (It
> >>>> can be downloaded from here
> >>>> <http://www.openadr.org/specification-download>; I’d send it as an
> >>>> attachment, but it’s large and encompasses many files, and SourceForge
> >>>> doesn’t let me attach a .zip file.) After using generateDS v2.29.3 to
> >>>> generate the Python file, I’m finding that I need to implement a few
> >>>> workarounds in order to get working code:
> >>>>
> >>>> The XSD uses polymorphism, reflected in the generated code as class
> >>>> hierarchies. In some cases, exported XML nodes are getting named based
> >>>> on their abstract superclasses (streamPayloadBase, payloadBase) instead
> >>>> of based on the concrete subclasses (oadrReportPayload, PayloadFloat,
> >>>> oadrPayloadResourceStatus). As you can see in oadr_test.py, I can avoid
> >>>> this issue by setting the Python instance’s “original_tagname_”. I don't
> >>>> know, though, whether that's the recommended solution — I couldn’t find
> >>>> documentation about the purpose of original_tagname_. Should I have
> >>>> expected the XML nodes to be named correctly without calling
> >>>> original_tagname_? Is there something else that I should have done to
> >>>> avoid the incorrect names?
> >>>> If I export XML using the original, untweaked version of oadr_20b.py,
> >>>> the export fails because the “intervals” class lacks a “strm"
> >>>> namespacedef. (You can see this yourself if you switch which oadr import
> >>>> statement is commented out in oadr_test.py.)
> >>>> If I export XML using the original, untweaked version of oadr_20b.py,
> >>>> the namespace of PayloadFloat is “oadr", which is incorrect — it should
> >>>> be "ei”.
> >>>>
> >>>> If you’re able to shed any light on these issues, I’d greatly appreciate
> >>>> it.
> >>>>
> >>>> Rob Calvert
> >>>> (510) 759-1838
> >>>> www.kisensum.com <http://www.kisensum.com/>
> >>>>
> >>>> Kisensum | 344 Thomas Berkley Way #260 | Oakland, CA, 94612
> >>>>
> >>>> Attachments:
> >>>> oadr_20b_v2_29_3.py (original generateDS output)
> >>>> oadr_20b.py (tweaked version of generateDS output containing some
> >>>> workarounds)
> >>>> oadr_test.py (test code)
> >>>>
> >>>>
> >>>>
> >>>>> Dave, I quickly tried out your new 2.29.3 version of generateDS,
> >>>>> wondering whether it might clear up some things I’m currently doing to
> >>>>> work around generated-code issues. It hasn’t changed what I’m seeing,
> >>>>> though, so it’s time for me to ask your advice.
> >>>>>
> >>>>> Before describing the current issues, let me thank you for helping my
> >>>>> colleague Bob Barcklay last month. The 2.29.0 version of generateDS
> >>>>> solved some of the problems we’d been hitting, allowing us to meet a
> >>>>> deliverable deadline. So, on to the current set of issues...
> >>>>>
> >>>>> I've implemented an XML protocol that uses the OpenADR v2.0b schema
> >>>>> (the schema can be downloaded from here (I’d send it to you as an
> >>>>> attachment, but SourceForge doesn't let me attach a .zip, and the
> >>>>> schema is large and encompasses many files). After using generateDS
> >>>>> v2.29.3 to generate a Python file for that schema, I’m finding that I
> >>>>> need to implement a few workarounds in order to get working code:
> >>>>>
> >>>>> The XSD uses polymorphism, reflected in the generated code as class
> >>>>> hierarchies. In some cases, exported XML nodes are getting named based
> >>>>> on their abstract superclasses (streamPayloadBase, payloadBase) instead
> >>>>> of based on the concrete subclasses (oadrReportPayload, PayloadFloat,
> >>>>> oadrPayloadResourceStatus). As you can see in oadr_test.py, I can avoid
> >>>>> this issue by setting the Python instance’s “original_tagname_”. I
> >>>>> don't know, though, whether that's the recommended solution — I
> >>>>> couldn’t find documentation about the purpose of original_tagname_.
> >>>>> Should I have expected the XML nodes to be named correctly without
> >>>>> calling original_tagname_? Is there something else that I should have
> >>>>> done to avoid the incorrect names?
> >>>>> If I export XML using the original, untweaked version of oadr_20b.py,
> >>>>> the export fails because the “intervals” class lacks a “strm"
> >>>>> namespacedef. (You can see this yourself if you switch which oadr
> >>>>> import statement is commented out in oadr_test.py.)
> >>>>> If I export XML using the original, untweaked version of oadr_20b.py,
> >>>>> the namespace of PayloadFloatType is "oadr", which is incorrect — it
> >>>>> should be "ei”.
> >>>>>
> >>>>> If you’re able to shed any light on these issues, I’d greatly
> >>>>> appreciate it.
> >>>>>
> >>>>> Rob Calvert
> >>>>> (510) 759-1838
> >>>>> www.kisensum.com <http://www.kisensum.com/>
> >>>>>
> >>>>> Kisensum | 344 Thomas Berkley Way #260 | Oakland, CA, 94612
> >>>>>
> >>>>> Attachments:
> >>>>> oadr_20b_v2_29_3.py (original generateDS output)
> >>>>> oadr_20b.py (tweaked version of generateDS output containing some
> >>>>> workarounds)
> >>>>> oadr_test.py (test code)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>> Date: Tue, 12 Dec 2017 00:38:18 +0000
> >>>> From: generateds-users-requ...@lists.sourceforge.net
> >>>> Subject: confirm 24bf5689ce6295e0d6d499e87b69e3433e2a9d93
> >>>>
> >>>> If you reply to this message, keeping the Subject: header intact,
> >>>> Mailman will discard the held message. Do this if the message is
> >>>> spam. If you reply to this message and include an Approved: header
> >>>> with the list password in it, the message will be approved for posting
> >>>> to the list. The Approved: header can also appear in the first line
> >>>> of the body of the reply.
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Dave Kuhlman
> >>> http://www.davekuhlman.org
> >>
> >
> > --
> >
> > Dave Kuhlman
> > http://www.davekuhlman.org
>
--
Dave Kuhlman
http://www.davekuhlman.org
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
generateds-users mailing list
generateds-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/generateds-users
