Denis, Thank you for the SQLAlchemy adaptation. I'm looking into it. It looks good. And, your notes (below) are very helpful.
I'm also working on capturing XML instance doc and loading it into the Django DB/ORM. That is helping me to learn more about the Django database support. I'm trying to adapt the existing Django support so that I can write a generateExportDjango method (in generateDS.py) that generates exportDjango methods that load instance data into the Django DB. By the way, I'm also learning that we need something like the work you did in gends_generate_sqlalchemy.py to create backrefs_map and then use that in generatedssuper.py to create relationships. I'll see what I can learn from your code. When I've made some progress on this, I'll begin work with your SQLAlchemy support. Hopefully, I'll be able to do something analogous in the way of capturing instance data and loading it into an SQLAlchemy database. Whew. This is going to take quite a bit of work. I'll try to keep you informed on my progress, when and if I make some. Thanks again for help. Dave On Fri, Jul 13, 2018 at 03:54:53PM +0300, jabber wrote: > Hello, Dave. > > >That would be great. Please send them. I'll take a look, and then > >might ask a few questions. > > Ok, let's try. > I attached archive with my modifications. This is archived folder > "sqlalchemy". It's similar to your folder "django". > I'll try to describe modifications I made. > > 0. All modifications I made on the base of *generateDS-2.29.14*. > > 1. At first I changed the way that "generateDS" and some its modules use to > output files. I added *encoding='utf-8' *when opening the output files for > write. Here is list of changes: > > - module "gends_extract_simple_types" - line 204, > - module "generateDS" - lines 6807, 6811, 6908, 6946 > > I made so because *gends_generate_django *(and also my module > gends_generate_sa) could not import library code, generated by generateDS > when the schema files were in utf-8 encoding and annotation tags, which you > use to generate comments for data representation classes, were in russian > (error: SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xc0 > in position 0: invalid start byte). Sample schemas are attached > (instinfo.xsd) > > 2. Also I modified *Simple_type_table* in *generatedssuper.py* - added > appropriate sqlalchemy types for builtin types. You didn't use values in > this dict, so this won't interfere with your code. I use this code in > *gends_generate_sqlalchemy.py*. > > 3. *gends_run_gen_sa.py* - this is analogue of your *gends_run_gen_django.py. > *It invokes *generateDS.py*, *gends_extract_simple_types.py*, and than - > *gends_generate_sqlalchemy.py*, which generates sqlalchemy models. > > 4. *gends_generate_sqlalchemy.py* - it' also similar to your > *gends_generate_django.py*. But it generates only models file ( > *models_sa.py*). I modified only *generate_models* function - removed code > for *admin.py* and *forms.py* generation. > Before models generation I populate *simpletypes_set*, which is used to > make import list for sqlalchemy field types (Integer, String, etc). > Also I populate *backrefs_map*. I use it to get for each given model a list > of models that refer to it. I need this to generate in parent model > backrefs to child models (one-to-many relationship). > Then I generate sqlalchemy models for each class ( this code is similar to > your code in * gends_generate_django.py*). I use *generate_sa_model_* > method of *GeneratedsSuper* class in *generatedssuper.py*. I pass an extra > parameter into this function - previously populated *backrefs_map*. > > 5. Also I modified *generatedssuper.py* to add class method > *generate_sa_model_* to *GeneratedsSuper* class. This method (like > *generate_model_* method for django) generates model definition. It > generates class preamble, *__tablename__ *definition and integer *id > *definition, > than it traverses all data items of class and generates fields definitions > for each column, including columns with foreign keys. > After columns processing it generates relationships for children classes. > > That is all. Don't kick me hard:) > > ---------------------------------------- > My best regards. > Denis Sutyagin aka jabber. > ----------------------------------------- > dJabber(AT)gmail.com > > 2018-07-11 0:41 GMT+03:00 Dave Kuhlman <dkuhl...@davekuhlman.org>: > > > Denis, > > > > > May be I could send you my modifications with some comments? > > > > That would be great. Please send them. I'll take a look, and then > > might ask a few questions. > > > > > I'm using django 2.0.4. I think you should use > > "on_delete=models.CASCADE" > > > > Thanks. So, I'll keep that change. > > > > I have a question about models.ForeignKey -- If, for example, we > > have a schema defining a mailbox that contains messages > > > > <?xml version="1.0"?> > > <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema";> > > <xs:element name="mailbox" type="mailboxType"/> > > <xs:complexType name="mailboxType"> > > <xs:sequence> > > <xs:element name="owner" type="xs:string"/> > > <xs:element name="description" type="xs:string"/> > > <xs:element name="messages" type="messageType" > > minOccurs="0" maxOccurs="unbounded"/> > > </xs:sequence> > > </xs:complexType> > > <xs:complexType name="messageType"> > > <xs:sequence> > > <xs:element name="from" type="xs:string"/> > > <xs:element name="to" type="xs:string"/> > > <xs:element name="subject" type="xs:string"/> > > <xs:element name="body" type="xs:string"/> > > </xs:sequence> > > </xs:complexType> > > </xs:schema> > > > > We currently generate the following: > > > > from django.db import models > > > > class mailboxType_model(models.Model): > > owner = models.CharField(max_length=1000, ) > > description = models.CharField(max_length=1000, ) > > messages = models.ForeignKey( > > "messageType_model", > > on_delete=models.CASCADE, > > related_name="mailboxType_messages_messageType", > > blank=True, null=True, > > ) > > > > class messageType_model(models.Model): > > from_x = models.CharField(max_length=1000, ) > > to = models.CharField(max_length=1000, ) > > subject = models.CharField(max_length=1000, ) > > body = models.CharField(max_length=1000, ) > > > > Whereas the Django documentation says, I believe, that the > > ForeignKey should be defined on the other end of the relation, and > > so we should actually generate something like the following: > > > > from django.db import models > > > > class mailboxType_model(models.Model): > > owner = models.CharField(max_length=1000, ) > > description = models.CharField(max_length=1000, ) > > > > class messageType_model(models.Model): > > from_x = models.CharField(max_length=1000, ) > > to = models.CharField(max_length=1000, ) > > subject = models.CharField(max_length=1000, ) > > body = models.CharField(max_length=1000, ) > > messages = models.ForeignKey( > > "mailboxType_model", > > on_delete=models.CASCADE, > > related_name="mailboxType_messages_messageType", > > blank=True, null=True, > > ) > > > > Do you have any guidance? I do not want to change this if the > > current method works. I seem to remember that, sometime in the > > past, someone suggested that the ForeignKey should be reversed and > > that it should be defined in the child rather than the container, > > making it a many-to-one relationship rather than a one-to-many > > relation. I do not understand it well enough to make an informed > > decision on this. > > > > Any hints and suggestions will be appreciated. > > > > Thanks for help. > > > > Dave > > > > On Mon, Jul 09, 2018 at 03:24:18PM +0300, jabber wrote: > > > Hello, Dave. > > > > > > >Would you be willing to contribute your enhancement. If so, I'd be > > > willing to do a little clean-up work if needed. > > > > > > I would be happy, but I never contributed to any open project... > > > May be I could send you my modifications with some comments? > > > And if you decide to include it into distribution of generateDS, you may > > do > > > with this code all you need :) > > > > > > > By the way, I've started working on loading instance data into a Django > > > database. > > > > > > I'm happy to hear about! > > > > > > >on_delete=models.CASCADE > > > > > > I'm using django 2.0.4. I think you should use > > "on_delete=models.CASCADE" > > > because of django migration engine. Migration angine allows downgrade > > > database to the previous version. And during downgrade process it should > > > delete previously created tables. If the table depends on another table > > > (the second table refers to first using foreign key), migration engine > > > could not delete first table. But when you write " > > on_delete=models.CASCADE > > > " - django could delete tables with their child tables. > > > > > > > > > > > > ---------------------------------------- > > > My best regards. > > > Denis Sutyagin aka jabber. > > > ----------------------------------------- > > > dJabber(AT)gmail.com > > > > > > 2018-07-05 1:44 GMT+03:00 Dave Kuhlman <dkuhl...@davekuhlman.org>: > > > > > > > Denis, > > > > > > > > Please pardon a possible repeat message, but my previous message did > > > > not show up in the generateds-users mailbox. > > > > > > > > By the way, I've started working on loading instance data into a > > > > Django database. So far, I've been able to generate the models for > > > > a sample XML schema, and then do: > > > > > > > > $ python manage.py makemigrations myapp > > > > $ python manage.py migrate > > > > > > > > In order to do the above, I had to modify the django generator so > > > > that it adds: > > > > > > > > on_delete=models.CASCADE, > > > > > > > > I'm not sure why. Am I using a different version of django? My > > > > version is django 2.0.5. > > > > > > > > Also, I had to change the line generated in admin.py that reads: > > > > > > > > from models import \ > > > > > > > > to this: > > > > > > > > from .models import \ > > > > > > > > I believe that is because I'm using Python 3.6.6. So, I'll need to > > > > see about making that work with both Python 2.7 and 3.6. Perhaps, > > > > it's because I generated the project and app using Python 3.6.6? > > > > > > > > My next step is to design and investigate the use of a method in the > > > > classes generated by generateDS.py (let's call it exportDjango) to > > > > capture XML instance data and load it into a django DB. I still > > > > have not gotten far enough to know whether doing that is feasible. > > > > > > > > But, as Ringo Starr told us: > > > > > > > > "Got to pay your dues if you wanna sing the blues, > > > > And you know it don't come easy." > > > > > > > > More later. > > > > > > > > Dave > > > > > > > > In and earlier message, dave wrote: > > > > > > > > Denis, > > > > > > > > > I'm using generateDS to create Django models from xml schemas. It's > > > > great. > > > > > I even modified some code to generate SQLAlchemy models. > > > > > > > > Cool. Do you think that your modifications could be added to the > > > > generateDS distribution? Would you be willing to contribute them? > > > > If so, I'd be willing to do a little clean-up work if needed. > > > > > > > > > But I think the next step could be loading XML instance data into > > > > generated > > > > > models. Did you think about this case? > > > > > > > > Yes, I think that would be a valuable enhancement. And, yes, I've > > > > thought about it. Of course, just thinking does not get us there; > > > > I'd have to *do* something. > > > > > > > > In the meantime, here are a few thoughts: > > > > > > > > It might make sense to add support for loading instance data into a > > > > Django model as another form of export. generateDS already has > > > > three of those. From the help/usage message: > > > > > > > > --export=<export-list> Specifies export functions to be > > generated. > > > > Value is a whitespace separated list of > > > > any of the following: > > > > write -- write XML to file > > > > literal -- write out python code > > > > etree -- build element tree (can > > serialize > > > > to XML) > > > > Example: "write etree" > > > > Default: "write" > > > > > > > > There is a corresponding function in the `generateDS.py` module for > > > > each of these: `generateExportFn`, `generateToEtree`, and > > > > `generateExportLiteralFn`. I suppose that we could consider adding > > > > `generateExportDjangoFn` (or something like that), which would > > > > generate exportDjango. And, while working on that, it might be a > > > > good idea to change `generateExportLiteralFn` to > > > > `generateExportJSONFn` (or maybe create a new function to do that). > > > > And, since you are apparently interested in the SQLAlchemy ORM, > > > > perhaps we also have to consider `generateExportSQLAlchemy`. Whew! > > > > > > > > Effectively, we would be implementing a new function > > > > (`generateExportDjangoFn`) that would generate a new method in each > > > > class generated for each XML element type. And, that new method > > > > would (like the other export methods) would perform a tree walk of > > > > the element type classes and while doing so would insert instance > > > > data into the Django model. > > > > > > > > Assumptions and cautions: > > > > > > > > - The model (definition) is one that has been generated by > > > > generateDS. > > > > > > > > - The model definition) was generated by the same XML schema that > > > > was used to generate the module containing the exportDjango > > > > methods used to populate it. Perhaps we even need to implement a > > > > way to verify this (possibly using an shasum somehow?). > > > > > > > > Requirements: > > > > > > > > - There is a way to give each export method (say, `exportDjango`) > > > > a target, e.g. a Django database (and other info). > > > > > > > > - We have to enable the export process to run within the Django > > > > environment. Right? Is that possible? Need to check. Run > > > > inside the Django `dbshell`? > > > > > > > > It's be awhile since I've used Django, so I suppose I should > > > > consider this as an opportunity to relearn what I've forgotten and > > > > what I have yet to learn. > > > > > > > > By the way, just recently I ran across the term ETL (extract > > > > transform load, see https://en.wikipedia.org/wiki/ > > Extract,_transform,_load > > > > ), > > > > although I believe that it's a rather old term. It seems that we > > > > should be trying to think of generateDS as a possible part of an ETL > > > > tool chain. And, related to ETL, I found notes about extracting > > > > *text* documents and storing them in a database, and while doing so, > > > > also using some of the Python data science tools (e.g. > > > > scikit-learn) to help make an intelligent search capability. That > > > > sounds cool. Perhaps thinking about ETL and intelligent search will > > > > help motivate me. > > > > > > > > Give me a little time to think about it and to try to get myself > > > > motivated. > > > > > > > > More later. > > > > > > > > Dave > > > > > > > > On Fri, Jun 29, 2018 at 08:57:33AM +0300, jabber wrote: > > > > > Hello Dave, > > > > > > > > > > I'm using generateDS to create Django models from xml schemas. It's > > > > great. > > > > > I even modified some code to generate SQLAlchemy models. > > > > > But I think the next step could be loading XML instance data into > > > > generated > > > > > models. Did you think about this case? > > > > > > > > > > Thank you. > > > > > Denis > > > > > > > > > ------------------------------------------------------------ > > > > ------------------ > > > > > Check out the vibrant tech community on one of the world's most > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > > > > > > _______________________________________________ > > > > > generateds-users mailing list > > > > > generateds-users@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/generateds-users > > > > > > > > > > > > -- > > > > > > > > Dave Kuhlman > > > > http://www.davekuhlman.org > > > > > > > > -- > > > > Dave Kuhlman > > http://www.davekuhlman.org > > -- Dave Kuhlman http://www.davekuhlman.org ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ generateds-users mailing list generateds-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/generateds-users
