Hi Dave,

YES!!! That worked!! So, so happy.

Thank you so much for all your hard work!

Warm wishes from a very snowy Cambridge, UK.
Sanja

On 27/02/2018 22:29, Dave Kuhlman wrote:
Sanja,

I did a bit more work on this issue.  I'd tested what I had last
week, but somehow my tests yesterday morning produced another
unicode error.

Now I believe that I have something that works.  At the least, all
the tests I've run indicate that it does what we want.

By the way, the use of command line option --cleanup-name-list is
not needed with your schema.  I was confused about that.  It is only
needed, for example, if you have a multi-byte character in an
element name/tag.

I ran generateDS.py with the following shell script:

        #!/bin/bash -v

        ./generateDS.py \
                -f \
                -o $1 \
                emdb.xsd

I ran it with the following command line:

     $ ./run-emdb.sh emdb_py2.py        # Python 2
     $ ./run-emdb.sh emdb_py3.py        # Python 3

I was able to run generateDS.py with both Python 2 and 3.  I was
able to import the generated module with either Python 2 or 3.

I've attached the fixed version of generateDS and the generated
modules (so that you can see whether you are able to produce
something equivalent).  I've attached them in separate messages, in
case of email size limitations.

Please let me know your results.  I apologize for being slow on
this.

Dave

On Tue, Feb 27, 2018 at 04:27:32PM +0000, Sanja Abbott wrote:
Hi Dave,

I just wondered if you have managed to find time to make generateDS work
with the schema I've sent you?

I've tried all your other suggestions (python3 and
--cleanup-name-list="[('µ', 'm')]") but haven't been successful.

Thank you,
Sanja

On 23/02/2018 11:33, Sanja Abbott wrote:
Hi Dave,

I should have send you our schema earlier but here it is, attached. I
hope that helps.

Thank you,
Sanja

On 23/02/2018 00:02, Dave Kuhlman wrote:
Sanja,

In order to reproduce your non-ascii character error message, I
created a schema containing an attribute declaration with a name
containing a non-ascii character.  Example:

          <xs:attribute name="İstanbul" type="xs:string"
              fixed="a fascinating city" use="required" />

Is that where your non-ascii characters were?  Note that the İ in
İstanbul has a dot on top.

I believe that there already is a way to handle this with
generateDS.py, but it only seems to work with Python 3.  You can try
using the --cleanup-name-list command line option.  In your case it
might be:

      ./generateDS.py --cleanup-name-list="[('µ', 'm')]" -f -o
tmp01sup.py test05-01.xsd

As I said, it only seems to work with Python 3, not Python 2. I'll
look into fixing that tomorrow.

The module that you generate, by the way, should be usable with
either Python 2 or Python3.

Thanks for your kind words.  And, I'm gratified that your group is
getting use out of generateds.

Dave

On Thu, Feb 22, 2018 at 12:01:14PM +0000, Sanja Abbott wrote:
Hi Dave,

please do not apologise, we really appreciate all your effort;
our whole
group uses your software and the fact you can be easily
contacted and act
promptly to our requests is really valuable to us.

Thank you for sending me a new distribution file. It worked (bar
for some
characters, more bellow) and indeed has solved the previous problem and
produces the requested fixed attribute correctly:
def __init__(self, emdb_id=None, version='3.0.0.0', admin=None,
crossreferences=None, sample=None, ...

My last request is to enable the newest generateDS.py to
translate non-ascii
characters. My schema xsd file contains these:
1. µ ( u'\xb5') and
2. Å (u'\u212b')

When I run my schema with these characters against generateDS.py I get,
e.g., this error message:

sanja-ml:generateDS-2.29.7 sanja$ python generateDS.py
--root-element="emd"
-f -o "emdb.py" --external-encoding='utf-8'
../workspaceSelenium/v3/DA_schema/emdb.xsd
Traceback (most recent call last):
    File "generateDS.py", line 7438, in <module>
      main()
    File "generateDS.py", line 7426, in main
      superModule=superModule)
    File "generateDS.py", line 6914, in parseAndGenerate
      prefix, root, options, args, superModule)
    File "generateDS.py", line 6724, in generate
      generateFromTree(wrt, prefix, elements, processed)
    File "generateDS.py", line 6639, in generateFromTree
      generateClasses(wrt, prefix, element, 0, nameSpacesDef)
    File "generateDS.py", line 4994, in generateClasses
      generateCtor(wrt, prefix, element)
    File "generateDS.py", line 4133, in generateCtor
      wrt('    def __init__(self%s):\n' % s2)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u212b' in
position 30: ordinal not in range(128)

However, when I substitute 'µ' with 'm' and 'Å' with 'A' no errors were
reported.

Additionally, previous versions of generateDS could also
translate non-ascii
characters without a problem.

I think the distribution file worked really well and I would give it a
preference when adding new features and testing fixes in future.

Thank you again.
Best wishes,
Sanja

On 21/02/2018 21:13, Dave Kuhlman wrote:
Sanja,

I apologize.  I did not realize that another file
(process_includes.py), imported by generateDS.py, had also changed.
Actually, that new version of process_includes.py was at Bitbucket,
but there was no way for you to know that.

So, I've created a new distribution file containing this fix.  You
should be able to use it as follows:

       $ tar xf generateDS-2.29.7.tar.gz
       $ cd generateDS-2.29.7
       $ python generateDS.py -o tmp01sup.py ../test04-01.xsd
       $ diff -u ../generateDS-2.29.5/tmp01sup.py tmp01sup.py

By the way, there is no need to do an install in order to run the
above from within that directory.

The last line above (diff -u ...) should produce the diff file that
I've attached (in a separate email).

The above produces the change that we have been trying to achieve, I
believe.  Please let me know about that.

In a separate email, I've attached that new distribution file and
the diff file, too.

Again, I apologize for wasting your cycles.

I like to put off pushing a change to Bitbucket and pypi.python.org
until I have some confidence that we want to keep it.  I need to
come up with a more reliable way to pass along a temporary change
for testing.  Maybe this (producing a distribution file) is that
better way.  Let me know what you think about that, too, please.

Dave

On Wed, Feb 21, 2018 at 03:47:41PM +0000, Sanja Abbott wrote:
Hi Dave,

Thank you for sending me generateDS.py (version 2.29.7), a schema for
testing the fixed attributes (test04-01.xsd) and the output file
(tmp07sup.py) you get by running this script.

I have tried to replicate your results but I get this error:

sanja-ml:generateDS-2.29.5 sanja$ python generateDS.py -o "test.py"
../../Desktop/test04-01.xsd
Traceback (most recent call last):
     File "generateDS.py", line 7438, in <module>
       main()
     File "generateDS.py", line 7426, in main
       superModule=superModule)
     File "generateDS.py", line 6904, in parseAndGenerate
       no_redefine_groups=noRedefineGroups,
TypeError: 'lxml.etree._ElementTree' object is not iterable

To explain how I got there. I've downloaded your latest
available package
generateDS-2.29.5.tar.gz. Ran the following commands:

tar xzvf generateDS-2.29.5.tar.gz
cd generateDS-2.29.5
python setup.py build
python setup.py install

Then I renamed generateDS.py to generateDS_original.py and copied
generateDS.py version '2.29.7' into the
generateDS-2.29.5 folder. You can
see that the two generateDS scripts are in the same
directory and they are
different:

sanja-ml:generateDS-2.29.5 sanja$ diff generateDS.py
generateDS_original.py
232c232
< VERSION = '2.29.7'
---
VERSION = '2.29.5'
264d263
< SchemaNamespaceDict = {}
1434,1435c1433
<             default=None,
<             fixed=None):
---
                default=None):
1440,1442d1437
<         # treat `fixed` the same as `default`.
<         if fixed is not None:
<             self.default = fixed
1702,1705d1696
<             if 'fixed' in attrs:
<                 fixed = attrs['fixed']
<             else:
<                 fixed = None
1708,1709c1699
<                 attribute = XschemaAttribute(
<                     name, data_type, use, default, fixed)
---
                    attribute =
XschemaAttribute(name, data_type, use,
default)
1713,1714c1703
<                 attribute = XschemaAttribute(
<                     name, data_type, use, default, fixed)
---
                    attribute =
XschemaAttribute(name, data_type, use,
default)
2824,2829d2812
<     ns_prefix = SchemaNamespaceDict.get(name)
<     if ns_prefix is not None and ns_prefix[0] is not None:
<         namespace = ns_prefix[0] + ':'
<         ns_def = 'xmlns:{}'.format(ns_prefix[0])
<         if ns_def not in nameSpacesDef:
<             nameSpacesDef += ' {}="{}"'.format(ns_def,
ns_prefix[1])
4365c4348
<             pats2 = [u'^{}$'.format(replaceVbars(p1))
for p1 in pats1]
---
                pats2 =
['^{}$'.format(replaceVbars(p1)) for p1 in pats1]
4561,4562d4543
<             # Check special case: simpletype that
restricts xs:simpletype.
<             # Prevent infinite recursion.
6870,6871c6851
<         SchemaLxmlTree, ModuleSuffix, UseSourceFileAsModuleName, \
<         SchemaNamespaceDict
---
            SchemaLxmlTree, ModuleSuffix, UseSourceFileAsModuleName
6898c6878
<             doc, SchemaNamespaceDict =
process_includes.process_include_files(
---
                doc = process_includes.process_include_files(
6935c6915
<             doc, SchemaNamespaceDict =
process_includes.process_include_files(
---
                doc = process_includes.process_include_files(
If I ran generateDS_original.py on your schema
(test04-01.xsd) and compare
the result with the tmp07sup.py file you've provided I get:

sanja-ml:generateDS-2.29.5 sanja$ python
generateDS_original.py -o "test.py"
../../Desktop/test04-01.xsd
sanja-ml:generateDS-2.29.5 sanja$ diff test.py
../../Desktop/tmp07sup.py
5,6c5,6
< # Generated Wed Feb 21 15:32:42 2018 by generateDS.py
version 2.29.5.
< # Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015,
12:54:16)  [GCC 4.2.1
(Apple Inc. build 5666) (dot 3)]
---
# Generated Mon Feb 19 14:13:30 2018 by
generateDS.py version 2.29.7.
# Python 3.6.3 (default, Oct  3 2017, 21:45:48)  [GCC 7.2.0]
9c9,14
< #   ('-o', 'test.py')
---
#   ('-f', '')
#   ('-o', 'tmp07sup.py')
#   ('-s', 'tmp07sub.py')
#   ('--super', 'tmp07sup')
#   ('--member-specs', 'dict')
#   ('--export', 'write')
12c17
< #   ../../Desktop/test04-01.xsd
---
#   test04-01.xsd
15c20
< #   generateDS_original.py -o "test.py" ../../Desktop/test04-01.xsd
---
#   ./generateDS.py -f -o "tmp07sup.py" -s "tmp07sub.py"
--super="tmp07sup" --member-specs="dict"
--export="write" test04-01.xsd
18c23
< #   generateDS-2.29.5
---
#   Test01
448c453
< ExternalEncoding = 'ascii'
---
ExternalEncoding = 'utf-8'
718a724,728
        member_data_items_ = {
            'garden_name': MemberSpec_('garden_name',
'xs:string', 0, 0,
{'use': 'required'}),
            'version': MemberSpec_('version',
'xs:token', 0, 0, {'use':
'required'}),
            'vegetable': MemberSpec_('vegetable',
'xs:string', 1, 1, {'name':
'vegetable', 'type': 'xs:string', 'minOccurs': '0', 'maxOccurs':
'unbounded'}, None),
        }
721c731
<     def __init__(self, garden_name=None, version=None,
vegetable=None):
---
        def __init__(self, garden_name='dave\'s
backyard', version='3.0.0.0',
vegetable=None):
778c788
<         if self.garden_name is not None and 'garden_name' not in
already_processed:
---
            if self.garden_name != "dave's backyard"
and 'garden_name' not in
already_processed:
781c791
<         if self.version is not None and 'version' not in
already_processed:
---
            if self.version != "3.0.0.0" and 'version' not in
already_processed:
924,925c934,935
<         sys.stdout.write('#from test import *\n\n')
<         sys.stdout.write('import test as model_\n\n')
---
            sys.stdout.write('#from tmp07sup import *\n\n')
            sys.stdout.write('import tmp07sup as model_\n\n')
As expected, my test.py was created with the original
generateDS script and
it's different to yours, mine used version '2.29.5',
yours the intended
version '2.29.7'.

However, if I repeat the same with your new
generateDS.py version '2.29.7'
script I get the following:
sanja-ml:generateDS-2.29.5 sanja$ python generateDS.py -o "test.py"
../../Desktop/test04-01.xsd
Traceback (most recent call last):
     File "generateDS.py", line 7438, in <module>
       main()
     File "generateDS.py", line 7426, in main
       superModule=superModule)
     File "generateDS.py", line 6904, in parseAndGenerate
       no_redefine_groups=noRedefineGroups,
TypeError: 'lxml.etree._ElementTree' object is not iterable

I'm not sure what I could do next. I would be very
grateful if you could
suggest what that might be.

Thank you,
Sanja

On 19/02/2018 22:36, Dave Kuhlman wrote:
Sanja,

If I understand you correctly, we are getting different results.

I have the following in my test XML schema:

        <xs:attribute name="garden_name" type="xs:string"
            fixed="dave's backyard" use="required" />
        <xs:attribute name="version" type="xs:token" use="required"
            fixed="3.0.0.0"/>

And, the constructor that is generated by generateDS.py is the
following:

        def __init__(self, garden_name='dave\'s
backyard', version='3.0.0.0', vegetable=None):

Am I right that this is what you want generated?

If so, and if you are not getting that result, is it possible that
another version of generateDS.py on your machine is being used by
mistake.

Or, maybe there is something that I don't understand correctly?

I'll send the complete files (schema, generated module, and
generateDS.py) in a separate email so as not to put too much bulk in
the generateds-users email list.

Let me know if there is something else I need to test.

About, the generateds-users email list -- Yes, it would be good to
reply to or CC that list.  For one thing, that gives me an archive
of comments, requests, fixes, etc.  And, also, there may be other
users who are interested, have suggestions, etc.

Thanks for help with this.

Dave

On Mon, Feb 19, 2018 at 12:30:20PM +0000, Sanja Abbott wrote:
Dear Dave,

thank you for sending me the fix for the issue of 'fixed values for
attributes'. I've tested your solution and it
seems it's not producing the
result I expected.

I'd like to have this attribute:
<xs:attribute name="version" type="xs:token" use="required"
fixed="3.0.0.0"/>

For my purpose it's essential that the attribute
is 'required'. If I remove
use="required" and set default="3.0.0.0"
(default can't be required):
<xs:attribute name="version" type="xs:token" default="3.0.0.0"/>

I get exactly what I need in the python script using generateDS.py:

def __init__(self, emdb_id=None, version='3.0.0.0', admin=None,
crossreferences=None, sample=None, ...

However, the attribute I require creates:
def __init__(self, emdb_id=None, version=None, admin=None,
crossreferences=None, sample=None, ...


Is there anything that can be done so that I can
use the intended format for
my attribute.

Since it's my first time contacting you I wasn't
sure if I should cc
generateds-users as well. Please let me know.

Thank you for your help,
Sanja


On 15/02/2018 21:20, Dave Kuhlman wrote:
Sanja,

I've been thinking about this issue a bit more.

     From what I can learn, an attribute with
a `fixed` value is the same
as one with a `default` value except that a
validating parser should
enforce the `fixed` restriction.  That means that if the value of
that attribute is different from the value specified by `fixed` in
the schema, then the parser should report
that the XML document does
not validate.

Here is a bit of information, if you are interested:

- https://www.w3schools.com/xml/el_attribute.asp

- https://msdn.microsoft.com/en-us/library/ms256143(v=vs.110).aspx

The parsers generated by generateDS.py are
*not* validating parsers.
So, I believe that we're good by treating
`fixed` attributes in the
same way as `default` attributes.

With this change, you will be able to get that fixed value even
when the XML instance document omits it.

If the user (you, for example) wants to validate an XML instance
document and wants to enforce the `fixed`
restriction, then s/he can
run `xmllint` (or some other XML validation
program) against the XML
document and the schema.

I've attached a modified version of generateDS.py containing this
fix to a separate email.

Let me know what you think and whether this solves the problem for
you.

And, thank you for suggesting this enhancement.

Dave

On Fri, Feb 09, 2018 at 11:42:22AM +0000, Sanja Abbott wrote:
Hi,

I wonder if you could implement 'fixed'
values for attributes in your
generateDS. My schema has an attribute called 'version':

However, the python script that
generateDS creates has no means to use
this
fixed value.

Thank you,
Sanja Abbott

--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300

------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
generateds-users mailing list
generateds-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/generateds-users
--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300

--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300

--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300

--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300


--
_____________________________________________

Sanja Abbott, PhD
Bioinformatician and Software Infrastructure Manager
Cellular Structures and 3D Bioimaging
European Bioinformatics Institute
Welcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK
http://www.ebi.ac.uk
Tel: 01223 49 4300


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
generateds-users mailing list
generateds-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/generateds-users

Reply via email to