Re: Lets not solve Schema Evolution but reframe the problem

[Russell Keith-Magee]

On Wed, Apr 29, 2009 at 10:42 AM, DavidP  wrote:
>
> A  solution to the schema evolution problem is to reframe it and solve
> a different problem.

While I'm glad you are enthusiastic about the schema evolution
problem, now is not the best time to be discussing it. We are trying
(desperately) to get v1.1 out the door, and design discussions that
aren't explicitly on the v1.1 path only serve to distract the
community. If you could hold off on discussions such as these until we
are in the v1.2 feature discussion period, we would be much obliged.

I would also note that code speaks much louder than words. If you
think that you have a magical solution for schema evolution, code up a
proof of concept. Then you will be in a much stronger position to
argue your case.

> The "problem" with schema evolution is caused by a violation of the
> DRY principle. There is redundancy between the db schema and
> django's .py files.   The DRY violation may be removed by coding an
> application definition as per-application meta data annotations of the
> database itself. The meta data must be guaranteed to survive arbitrary
> changes to the schema by ALTER statements and disappear when a column
> or table is DROP'ed.
>
> In mySQL, one way of anchoring meta data to the schema is to store a
> GUID in the table and column comment fields in the database
> information_schema.  The data in comment field survives ALTER
> statements and will disappear after DROP statements and may be used as
> the keys for linking meta data to the schema.
>
> With these GUID’s in place, an application's specific django
> annotations (e.g. verbose_name=”", whether or not a field is visible
> in the application etc) may be stored in an a per-application
> definition table and the django models.py, admin.py, ... files may be
> regenerated at any time based on introspection of the database
> information_schema + this application's annotations.
>
> Bingo.

While I'm sure you are convinced of the efficacy of your solution, you
need to keep the following in mind:

 * Your solution needs to be database independent. Any solution you
propose needs to work with databases other than MySQL.

 * Django currently treats the Python model as canonical. If you're
proposing that we change this approach, you're going to need to show
how database changes can be reflected in Python. It isn't clear to me
how your proposal will maintain Python model logic, such as validation
and other utilities, when the database is reflected back into Python
code.

 * It isn't clear to me how your proposal will be able to handle
manual database changes. If a database manager adds a new field or
modifies an existing field on a Django model, but doesn't update the
metadata, how does this whole process work?

Schema evolution is a hard problem. People proposing easy solutions
are going to be treated with skepticism. This doesn't mean your
solution is wrong, but you need to be prepared to defend your
proposals - preferably when the rest of the community isn't trying to
push a release out the door.

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Lets not solve Schema Evolution but reframe the problem

[DavidP]

A  solution to the schema evolution problem is to reframe it and solve
a different problem.

The "problem" with schema evolution is caused by a violation of the
DRY principle. There is redundancy between the db schema and
django's .py files.   The DRY violation may be removed by coding an
application definition as per-application meta data annotations of the
database itself. The meta data must be guaranteed to survive arbitrary
changes to the schema by ALTER statements and disappear when a column
or table is DROP'ed.

In mySQL, one way of anchoring meta data to the schema is to store a
GUID in the table and column comment fields in the database
information_schema.  The data in comment field survives ALTER
statements and will disappear after DROP statements and may be used as
the keys for linking meta data to the schema.

With these GUID’s in place, an application's specific django
annotations (e.g. verbose_name=”", whether or not a field is visible
in the application etc) may be stored in an a per-application
definition table and the django models.py, admin.py, ... files may be
regenerated at any time based on introspection of the database
information_schema + this application's annotations.

Bingo.

David

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Schema Evolution

[andreas]

On Apr 27, 10:49 pm, Malcolm Tredinnick 
wrote:
>
> Permit me to correct that impression: you are mistaken.

Good! That's what I want'ed to be.

>
> That's also not a really valid assumption. There are still significant
> differences and different limitations and advantages with each approach
> to database migration. A bit more convergence and experience -- which is
> happening -- is already going on and will continue to do so. Look at all
> the active projects in this space. They are all changing their
> approaches as time goes by, based on feedback, maintenance experience
> and general whims.

This applies to django, webdevelopment and software in general aswell
and it's a good thing imo, "magic removal" in django
is a good example of how django not changed the values for what it
stands for
but slightly changed it's approach and became more explicit.



--

I will accept the fact that you guys don't want to discuss this now.
I didnt want to make anyone upset. I was kind of thinking about this
when I
mentioned that "everyone is probably fed up with this topic"
So Im hereby withdrawing from this thread.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: SQL Anywhere database backend

[Malcolm Tredinnick]

On Tue, 2009-04-28 at 14:38 -0400, Nathan Auch [Sybase] wrote:
> Hi Russ,
> 
> Thanks for taking the time to respond. We will finish off the SQL 
> Anywhere work in Django and then follow the procedure you outlined. It 
> sounds like starting with a externally maintained module and seeing 
> where that takes us is the way to go.

I would modify Russell's checklist very slightly... creating a patch in
Trac isn't going to be particularly helpful initially (I'm not quite
sure why he made that point #2 ahead of point #3 -- when the latter is
generally more important for an external feature -- but there may be
something I'm misunderstanding). This is only because an external module
very much is the best way to go in the initial phase. No modifications
to Django should be necessary for this and you don't need to live in the
django.db.backends.* namespace (in fact, avoid it as an external
module), as Russell noted.

A patch/ticket only really makes things a bit confusing for people
looking for this backend, as they won't know whether to use the patch or
some external project. By being purely an external project for a while,
there's no confusion. Anybody asking about an SQL Anywhere backend can
be pointed to the project website, download the latest version from
there, report bugs there and so on (it doesn't make sense for us to
handle bug reports for this backend, so, again, we want to make that
clear to potential users). There's also the short- to medium-term
problem of not really being any way to directly resolve such a ticket
and tickets should ideally have a way to be closed by somebody doing
some work on them ("waiting a while to see if it will be incorporated"
isn't an active strategy)

Obviously there are many ways to make this available and maybe you
already plan to place it under sybase.com/developer/opensource/
(although requiring membership before download is going to restrict the
number of people using this). One comparable case study here that might
be of interest, particularly for something that will hopefully be
updated regularly based on feedback, is IBM's db2 wrapper for Python.
They created a Google code project for it and then would transfer any
reported bugs back to their internal system for resolution. It's pretty
clunky, but partly required because they had a very restrictive policy
in place for that particular internal division regarding external
contributions (which appears to have been drastically relaxed and
improved lately, thank goodness). The advantage of what they did,
though, was it provided a subversion repository and publicly available
bug reporting path that was already familiar and available to a few tens
of thousands potential users and contributers.

Finally, one thing to consider before an initial release, given that
your ultimate goal might be inclusion in Django: the license. Django's
core and contrib applications are all BSD license, so, at a minimum
you'd have to be compatible with that before inclusion. This becomes a
consideration once you make an open source release because there's the
chance you'll be accepting contributions from the broader community. So
a later license change is very difficult (logistically) unless you go
for copyright assignment as part of contributing (which will restrict
the contributions). If I'm appearing to be teaching you to suck eggs
here, that's not the intention. Maybe this is all very familiar and well
planned on your side. I can't read any of the license information or
usage policies on existing stuff at the Sybase site without creating an
account and I'm not going to do that at the moment. A bit of forward
planning at this point (or working out if your corporate policies
conflict irrevocably with Django's ) might smooth things out a bit going
forwards. After all, a more restrictive license than the BSD license is
entirely possible, providing your end-game isn't inclusion in Django
core. We support external database modules as first-class citizens (with
the proviso that they're responsible for keeping up with any necessary
changes we have to make. The backend dataabaseAPI isn't part of our
forwards- or backwards-compatibility policy), so that's certainly a
supportable strategy as well.

Regards,
Malcolm


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: SQL Anywhere database backend

[Alex Gaynor]

On Tue, Apr 28, 2009 at 2:38 PM, Nathan Auch [Sybase] wrote:

>
> Hi Russ,
>
> Thanks for taking the time to respond. We will finish off the SQL
> Anywhere work in Django and then follow the procedure you outlined. It
> sounds like starting with a externally maintained module and seeing
> where that takes us is the way to go. I'll be reading this list going
> forward so I'll make sure to keep my eyes peeled for anything that could
> affect the SQL Anywhere db backend. Thanks for the heads up about
> multi-database support and key-value stores.
>
> We've done the port mainly to support the internal use of Review Board
> (http://www.review-board.org/). What do you want to see to show that we
> have a working back end? Should we just be trying to pass the included
> unit tests or is there a specific set of Django applications with which
> we should be testing?
>
> Thanks!
>
> Nathan Auch
> Sybase iAnywhere R
>
> Russell Keith-Magee wrote:
> > On Tue, Apr 28, 2009 at 10:17 PM, Nathan Auch [Sybase] 
> wrote:
> >
> >> I'm a developer on the core engine team of the SQL Anywhere database
> >> server (http://www.sybase.com/products/databasemanagement/sqlanywhere).
> >> We're in the final stages of adding a SQL Anywhere database backend to
> >> Django and would like to share our work with the community. What is the
> >> process for getting our new database backend into the main Django
> >> repository?
> >>
> >
> > Hi Nathan,
> >
> > Thanks for going to the effort of supporting Django! The more backends
> > the merrier!
> >
> > The process of getting your backend into Django trunk goes a little
> > something like this:
> >
> >  1) You open a new ticket describing the new feature (SQLAnywhere
> support)
> >  2) You upload your backend as a patch against the Django trunk
> >  3) You demonstrate over an extended period of time (6-12 months
> > minimum) that you are committed to keeping your backend up to date.
> >  4) We accept the backend, and give the developers of that backend
> > limited commit access to keep the backend up to date.
> >
> > Step 3 is the big hurdle here. It requires that you keep up to date
> > with any changes to the Django core - if we add a new function to the
> > backend interface, or a new feature to queries (e.g., aggregates), you
> > would be expected to port those changes in a timely fashion. It also
> > means that someone (and preferable multiple someones) from Sybase's
> > engineering team needs to get involved with the Django community,
> > engaging with design discussions that affect database backends, etc.
> > As a heads-up, there will be a few of these discussions in the near
> > future. Multiple DB support has been accepted as a Google Summer of
> > Code project - you can bet this will affect backends. Backend
> > modifications required to support column/key-value stores
> > (CouchDB/BigTable etc) may also crop up in the near future.
> >
> > This is a deliberately lengthy process. When we add a database backend
> > to Django, we (the Django Core) are implicitly committing to keeping
> > that backend up to date. Our old MSSQL backend went stale simply
> > because the original contributor didn't maintain the code, and the
> > core team didn't have ready access (or, for that matter, the personal
> > inclination) to support the backend.
> >
> > We don't want to see a repeat of this, so we will err on the side of
> > caution before adding a new backend. We're not going to add anything
> > until we know that we're going to get a couple of committers that are
> > both willing and able to maintain the backend for the long term. If
> > any licenses are going to be required to test your backend, we'll
> > probably hit you up for a few complimentary copies so the core team
> > can independently test your code.
> >
> > We're also going to need to see some sort of demonstrated demand
> > before we add a new backend to trunk. Prior to the addition of the
> > Oracle backend, the Django mailing lists had regular requests for
> > Oracle support. We still have regular requests for MSSQL support.
> > Without wanting to squash egos, I can only find 2 requests in the
> > Django-users archive for SQL Anywhere support. I appreciate there is a
> > certain amount of "build it and they will come" involved here, but you
> > will need to convince us that they actually will come if we build it
> > :-)
> >
> > The good news is that from a purely technical perspective, your code
> > doesn't need to be in the Django trunk in order for your userbase to
> > use Django. Django database backends can be referenced as an external
> > library. In your DATABASE_BACKEND setting, in addition to
> > 'postgresql', 'sqlite3', and the other official backends, you can
> > provide a python module name, and Django will load and use that
> > backend.
> >
> > Supporting your backend as an external module will be the preferred
> > approach for at least the short term. It's not a complete loss, 

Re: SQL Anywhere database backend

[Nathan Auch [Sybase]]

Hi Russ,

Thanks for taking the time to respond. We will finish off the SQL 
Anywhere work in Django and then follow the procedure you outlined. It 
sounds like starting with a externally maintained module and seeing 
where that takes us is the way to go. I'll be reading this list going 
forward so I'll make sure to keep my eyes peeled for anything that could 
affect the SQL Anywhere db backend. Thanks for the heads up about 
multi-database support and key-value stores.

We've done the port mainly to support the internal use of Review Board 
(http://www.review-board.org/). What do you want to see to show that we 
have a working back end? Should we just be trying to pass the included 
unit tests or is there a specific set of Django applications with which 
we should be testing?

Thanks!

Nathan Auch
Sybase iAnywhere R

Russell Keith-Magee wrote:
> On Tue, Apr 28, 2009 at 10:17 PM, Nathan Auch [Sybase]  
> wrote:
>   
>> I'm a developer on the core engine team of the SQL Anywhere database
>> server (http://www.sybase.com/products/databasemanagement/sqlanywhere).
>> We're in the final stages of adding a SQL Anywhere database backend to
>> Django and would like to share our work with the community. What is the
>> process for getting our new database backend into the main Django
>> repository?
>> 
>
> Hi Nathan,
>
> Thanks for going to the effort of supporting Django! The more backends
> the merrier!
>
> The process of getting your backend into Django trunk goes a little
> something like this:
>
>  1) You open a new ticket describing the new feature (SQLAnywhere support)
>  2) You upload your backend as a patch against the Django trunk
>  3) You demonstrate over an extended period of time (6-12 months
> minimum) that you are committed to keeping your backend up to date.
>  4) We accept the backend, and give the developers of that backend
> limited commit access to keep the backend up to date.
>
> Step 3 is the big hurdle here. It requires that you keep up to date
> with any changes to the Django core - if we add a new function to the
> backend interface, or a new feature to queries (e.g., aggregates), you
> would be expected to port those changes in a timely fashion. It also
> means that someone (and preferable multiple someones) from Sybase's
> engineering team needs to get involved with the Django community,
> engaging with design discussions that affect database backends, etc.
> As a heads-up, there will be a few of these discussions in the near
> future. Multiple DB support has been accepted as a Google Summer of
> Code project - you can bet this will affect backends. Backend
> modifications required to support column/key-value stores
> (CouchDB/BigTable etc) may also crop up in the near future.
>
> This is a deliberately lengthy process. When we add a database backend
> to Django, we (the Django Core) are implicitly committing to keeping
> that backend up to date. Our old MSSQL backend went stale simply
> because the original contributor didn't maintain the code, and the
> core team didn't have ready access (or, for that matter, the personal
> inclination) to support the backend.
>
> We don't want to see a repeat of this, so we will err on the side of
> caution before adding a new backend. We're not going to add anything
> until we know that we're going to get a couple of committers that are
> both willing and able to maintain the backend for the long term. If
> any licenses are going to be required to test your backend, we'll
> probably hit you up for a few complimentary copies so the core team
> can independently test your code.
>
> We're also going to need to see some sort of demonstrated demand
> before we add a new backend to trunk. Prior to the addition of the
> Oracle backend, the Django mailing lists had regular requests for
> Oracle support. We still have regular requests for MSSQL support.
> Without wanting to squash egos, I can only find 2 requests in the
> Django-users archive for SQL Anywhere support. I appreciate there is a
> certain amount of "build it and they will come" involved here, but you
> will need to convince us that they actually will come if we build it
> :-)
>
> The good news is that from a purely technical perspective, your code
> doesn't need to be in the Django trunk in order for your userbase to
> use Django. Django database backends can be referenced as an external
> library. In your DATABASE_BACKEND setting, in addition to
> 'postgresql', 'sqlite3', and the other official backends, you can
> provide a python module name, and Django will load and use that
> backend.
>
> Supporting your backend as an external module will be the preferred
> approach for at least the short term. It's not a complete loss, though
> - you can use this external support as part of the process of proving
> you are committed to Django support. Download stats for that backend
> could also be used to demonstrate the level of user demand.
>
> As an interim measure, if you can show 

Re: Bug or feature: pre_save used on FileField/ImageField doesn't work as in Django 1.0.2

[Lior Gradstein]

About the initial problem that started this thread, the associated bug
number is #10948.


On Apr 6, 10:41 pm, Karen Tracey  wrote:
> On Mon, Apr 6, 2009 at 1:56 PM, Marty Alchin  wrote:
>
> > On Mon, Apr 6, 2009 at 1:16 PM, Karen Tracey  wrote:
> > > I feel like I'm going around in circles thinking about this one -- is
> > there
> > > a way out that someone else sees that I'm blind to?
>
> > Welcome to my world! :( I spent a long time on issues like this prior
> > to getting the new file storage system in place at first, and
> > apparently skipped much of that step when doing r9766. At this point,
> > I think the most useful approach is to take a step back and just look
> > at the high-level options we have available. Trying to determine the
> > name in advance is a no-go, because that would just make the existing
> > known race conditions much more prominent. I see two options left:
>
> > 1. Write something to disk (maybe the whole file, as it was before,
> > maybe just a placeholder, as the _save() does now) when the file is
> > first assigned, so that we have a full, proper filename early on.
> > Then, if the model doesn't get saved, somehow roll that back so we
> > don't leave stuff lingering on the filesystem. This is currently
> > something of a "then some magic happens" approach, since I'm not yet
> > sure if there's a reasonable, reliable way to roll back a file save.
> > But I'd like to keep an open mind, so there it is. This would also be
> > preferable if we can detect transaction rollbacks, because there is
> > also #6456 to consider.
>
> I'm not seeing where exactly we have a _save() that writes just a
> placeholder?  The _save() in django.core.files.storage.FileSystemStorage
> writes the whole content, and I don't see any other _save as opposed to
> save() methods in the FileField area?
>
>
>
>
>
> > 2. Save the file all at once, as soon as possible, and blatantly
> > document this behavior when model validation goes in. Then, if people
> > need to validate a model that has a file on it, they have two options:
> > validate the model *before* saving the file to it, so the model is
> > known to be valid and can all be saved at once, or (if they need to
> > validate something in the file, such as its name or contents), add
> > their own code to delete the file if validation fails. Or, I suppose a
> > third option is to ignore the lingering files until they suck up too
> > much space, requiring a manual purge.
>
> > I obviously hate to go with number 2, but if we can't come up with
> > something solid, I think it's the better approach, at least for now.
> > Documenting unfortunate behavior is certainly preferable to coding
> > even more unfortunate behavior.
>
> What about:
>
> 3. Revert the removal of the FileField save_form_data override that was part
> of r9766.  It is that removal that is causing save() on a ModelForm to no
> longer save the file to disk.  If we restore it, names will get assigned for
> code using ModelForms just as they used to be.
>
> I am missing why the save_form_data override had to be removed in order to
> support the direct assignment of files to FileFields that is needed for
> model validation to work without the side-effect of model validation saving
> the file to disk.  I can see that it might be preferable, in general, to
> delay saving as long as possible so you don't wind up with files written to
> disk for models that are ultimately not saved to the DB.  But we didn't have
> that behavior in 1.0 -- model form save (even with commit=False), in 1.0.X,
> saves files to disk.  Trying to move the save later at this point, for the
> model form usage scenario, leads to backwards-compatibility problems.
>
> It's kind of ugly to have one form of model creation/assignment ( via model
> forms) save data early and a 2nd (assignment of files to FileFields) save
> data late, so I can see we might not want to do this.  But I thought I'd
> throw it out there.
>
> Reverting just that bit of r9766 also doesn't fix #10249 or #10300 which are
> resulting form the mixing in of FieldFile in the class bases.  It also
> doesn't fix #10404 in the case where someone uses the new
> assign-file-to-FileField without saving the field explicitly before the
> model.  So they'd still need fixing, but they seem more easily solved than
> this file name issue.
>
> Karen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---