Re: Test suite for session backend

[Andrei Antoukh]

2012/5/2 jgc31416 

> Hi all,
>
> I am new to Django development and I would like to know if there is a test
> for the session backend, I am writting a "yetAnOtherOne" session backend
> for MongoDB ( https://github.com/jgc31416/DJango14_MongoSessionStorage )
> and I would like to see if it passes the tests.
>
> JG
>

Hi!

What you're programming, already implemented in mongoengine:

https://github.com/MongoEngine/mongoengine/tree/master/mongoengine/django

Andrei.


-- 
http://twitter.com/niwibe
http://www.freebsd.org/
http://www.postgresql.org/
http://www.python.org/
http://www.djangoproject.com/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django git guidelines

[Andrei Antoukh]

2012/5/18 Anssi Kääriäinen <anssi.kaariai...@thl.fi>

> A heads up: I am working on Git and Github usage guidelines. There is
> a ticket https://code.djangoproject.com/ticket/18307, and I have a
> github branch with some initial work
> https://github.com/akaariai/django/tree/django_git_guidelines
> (or for changeset
> https://github.com/akaariai/django/compare/django_git_guidelines)
>
> The guidelines in short:
>  - For trivial patches use pull requests directly
>  - For non-trivial patches, create a trac ticket, announce your work
> by linking to your github branch, and when your work is ready to be
> pulled in, only then do a pull request
>  - Aim for logically contained commits, commit messages of 50 char
> summary line, 72 char paragraphs thereafter.
>  - When upstream has changed use git rebase instead of git pull
>  - When you do additional fixes to your work, use git rebase -i so
> that your work still fullfills the logical commits requirement.
>
> Lots of more details in the WIP branch. All feedback welcomed. Lets
> keep the discussion of any high-level issues here on django-
> developers, as the choices made impact the whole community.
>
> The biggest issue is how we aim to use the pull requests. My take on
> this is that pull requests should be only used for work ready for
> committer. That is, the original author feels the work is ready, or he
> doesn't know how to do anything more. If the pull requests are used
> for feature requests or work-in-progress patches we risk having lots
> of open tickets and lots of open pull requests.
>
> I have tried to gather pieces of wisdom around the net. I am not too
> experienced with Git, so if you have experience with the above
> mentioned and/or other Git workflows feedback is appreciated.



Looks excellent. Great work!

-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

About: Feature request: Comments in Django models saved to database schema

[Andrei Antoukh]

Hello!

I am writing about the issues:
https://code.djangoproject.com/ticket/13867
https://code.djangoproject.com/ticket/18468

I am not the author of these issues, but I think this feature is
interesting to have in the ORM.
I agree that we should not do things just because we can make!

In the source code often put comments to clarify things. These comments for
the same project
developer or someone who looks at the source code are often useful to know
how
something works or understand something that is not entirely clear.

In SQL's absolutely the same case. Sometimes you need to make comments on
both tables to clarify
the purpose and functionality, and in columns. All assume that if you want
to understand something,
 look at the source code, but this may not always be so.

Right now, the behavior to make comments can be emulated using sql
fixtures. And it has these drawbacks:

* Is not portable between different database backends, 3 files need to
write redundant for this in mysql,
postgresql and sqlite (sqlite does not natively support comments, but you
can emulate sql
http://stackoverflow.com/questions/7426205/sqlite comments
-adding-comments-to-tables-and-columns )

* It is more tedious to write SQL statements and you end up not making the
comments.

Is a low priority feature (obviously) but I think it's something that could
facilitate the ORM.

My first proposal is:
Comments to model tables (for mysql, postgresql and sqlite3) as the first
patch.
Without any kind of magic by django, a simple attribute "comment" on object
"Meta".

My initial proposal for postgresql implementation:
https://github.com/niwibe/django/tree/issue_18468

Andrei Antoukh

PD: sorry for my poor inglish

-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: About: Feature request: Comments in Django models saved to database schema

[Andrei Antoukh]

2012/7/9 Anssi Kääriäinen <anssi.kaariai...@thl.fi>

> On 7 heinä, 22:25, Andrei Antoukh <n...@niwi.be> wrote:
> > Hello!
> >
> > I am writing about the issues:
> https://code.djangoproject.com/ticket/13867https://code.djangoproject.com/ticket/18468
> >
> > I am not the author of these issues, but I think this feature is
> > interesting to have in the ORM.
> > I agree that we should not do things just because we can make!
> >
> > In the source code often put comments to clarify things. These comments
> for
> > the same project
> > developer or someone who looks at the source code are often useful to
> know
> > how
> > something works or understand something that is not entirely clear.
> >
> > In SQL's absolutely the same case. Sometimes you need to make comments on
> > both tables to clarify
> > the purpose and functionality, and in columns. All assume that if you
> want
> > to understand something,
> >  look at the source code, but this may not always be so.
> >
> > Right now, the behavior to make comments can be emulated using sql
> > fixtures. And it has these drawbacks:
> >
> > * Is not portable between different database backends, 3 files need to
> > write redundant for this in mysql,
> > postgresql and sqlite (sqlite does not natively support comments, but you
> > can emulate sqlhttp://stackoverflow.com/questions/7426205/sqlitecomments
> > -adding-comments-to-tables-and-columns )
> >
> > * It is more tedious to write SQL statements and you end up not making
> the
> > comments.
> >
> > Is a low priority feature (obviously) but I think it's something that
> could
> > facilitate the ORM.
> >
> > My first proposal is:
> > Comments to model tables (for mysql, postgresql and sqlite3) as the first
> > patch.
> > Without any kind of magic by django, a simple attribute "comment" on
> object
> > "Meta".
> >
> > My initial proposal for postgresql implementation:
> https://github.com/niwibe/django/tree/issue_18468
>
> I am not sure this is something we want to support. To me a generic
> "get_post_sync_sql(connection)" is a better approach. In addition to
> comments you could add indexes, constraints and so on in the hook. It
> should be called even for non-managed tables. That way you could use
> views using the hook. Trivial example:
>
> class UserPostCount(models.Model):
> user = models.OneToOneKey(User, primary_key=True,
> db_column="user_id", on_delete=models.DO_NOTHING,
> related_name='post_count')
> post_count = models.IntegerField()
>
> class Meta:
> db_table = 'user_post_count_view'
> managed = False
>
> def get_post_sync_sql(connection):
> if connection.vendor != 'postgresql':
> raise NotImplementedError
> return [
> """
> CREATE OR REPLACE VIEW user_post_count_view AS (
> SELECT user.id AS user_id, count(*) AS post_count
>   FROM user
>   JOIN user_posts ON user.id = user_posts.user_id
> );""",
> """
> COMMENT ON VIEW user_post_count_view IS 'A trivial view returning
> post count for users.';
> """]
>
> There could also be a get_post_flush_sql(connection) hook.
>
> It seems best if the hooks are called in a second pass - that is, the
> whole schema is already installed into the DB before running any of
> the post_sync SQL.
>
> Why not use the initial SQL files we already have? They are ran every
> time after flush - which means they are useless for defining views for
> example - and at least I like to keep the SQL together with model
> definition. In addition (IIRC) there are some problems with backends
> which do not support executing multiple statements in one go. Still,
> one can do logic in the hook, while the initial SQL files are flat
> files.
>
>  - Anssi
>
>
Hi Anssi!

I had submitted a proposal very simplistic. But your idea seems great! In
fact I love it! And I would solve much more. As you've said: indexes,
constrains, etc..
I will present a proposal for implementation with this in mind.

Andrei Antoukh

-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5 <http://www.kaleidos.net/A5694F/>--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this gro

Re: About: Feature request: Comments in Django models saved to database schema

[Andrei Antoukh]

Hello!

I started working on what had been proposed. In the commit
https://github.com/niwibe/django/commit/af887029integeintege614ca53d8bdab23f0a40ee7420b9bc20<https://github.com/niwibe/django/commit/af887029614ca53d8bdab23f0a40ee7420b9bc20>
is
the initial implementation. It is based on the Anssi idea.

I have some doubts about how to make things better. The proposed method
should return a list of sql statements:
["COMMENT ON TABLE fooapp_sampletable IS 'Foo comment';"]

or a list of tuples with sql statements with possible arguments:

[("COMMENT ON TABLE fooapp_sampletable IS %s", ["Foo comment"])]

Currently, the initial implementation, is a list of tuples, but I'm
wondering if the best option. I think that this is not usabe for commands
such as sqlall because the returned sql is not complete, if only partial,
failing to place them arguments.

Thank you very much!
Andrey Antukh



2012/7/9 Andrei Antoukh <n...@niwi.be>

> 2012/7/9 Anssi Kääriäinen <anssi.kaariai...@thl.fi>
>
>> On 7 heinä, 22:25, Andrei Antoukh <n...@niwi.be> wrote:
>> > Hello!
>> >
>> > I am writing about the issues:
>> https://code.djangoproject.com/ticket/13867https://code.djangoproject.com/ticket/18468
>> >
>> > I am not the author of these issues, but I think this feature is
>> > interesting to have in the ORM.
>> > I agree that we should not do things just because we can make!
>> >
>> > In the source code often put comments to clarify things. These comments
>> for
>> > the same project
>> > developer or someone who looks at the source code are often useful to
>> know
>> > how
>> > something works or understand something that is not entirely clear.
>> >
>> > In SQL's absolutely the same case. Sometimes you need to make comments
>> on
>> > both tables to clarify
>> > the purpose and functionality, and in columns. All assume that if you
>> want
>> > to understand something,
>> >  look at the source code, but this may not always be so.
>> >
>> > Right now, the behavior to make comments can be emulated using sql
>> > fixtures. And it has these drawbacks:
>> >
>> > * Is not portable between different database backends, 3 files need to
>> > write redundant for this in mysql,
>> > postgresql and sqlite (sqlite does not natively support comments, but
>> you
>> > can emulate sqlhttp://
>> stackoverflow.com/questions/7426205/sqlitecomments
>> > -adding-comments-to-tables-and-columns )
>> >
>> > * It is more tedious to write SQL statements and you end up not making
>> the
>> > comments.
>> >
>> > Is a low priority feature (obviously) but I think it's something that
>> could
>> > facilitate the ORM.
>> >
>> > My first proposal is:
>> > Comments to model tables (for mysql, postgresql and sqlite3) as the
>> first
>> > patch.
>> > Without any kind of magic by django, a simple attribute "comment" on
>> object
>> > "Meta".
>> >
>> > My initial proposal for postgresql implementation:
>> https://github.com/niwibe/django/tree/issue_18468
>>
>> I am not sure this is something we want to support. To me a generic
>> "get_post_sync_sql(connection)" is a better approach. In addition to
>> comments you could add indexes, constraints and so on in the hook. It
>> should be called even for non-managed tables. That way you could use
>> views using the hook. Trivial example:
>>
>> class UserPostCount(models.Model):
>> user = models.OneToOneKey(User, primary_key=True,
>> db_column="user_id", on_delete=models.DO_NOTHING,
>> related_name='post_count')
>> post_count = models.IntegerField()
>>
>> class Meta:
>> db_table = 'user_post_count_view'
>> managed = False
>>
>> def get_post_sync_sql(connection):
>> if connection.vendor != 'postgresql':
>> raise NotImplementedError
>> return [
>> """
>> CREATE OR REPLACE VIEW user_post_count_view AS (
>> SELECT user.id AS user_id, count(*) AS post_count
>>   FROM user
>>   JOIN user_posts ON user.id = user_posts.user_id
>> );""",
>> """
>> COMMENT ON VIEW user_post_count_view IS 'A trivial view returning
>> post count for users.';
>> """]
>>
>> There could also be a get_post_flush_sql(connection) hook.
>>
>>

Re: About: Feature request: Comments in Django models saved to database schema

[Andrei Antoukh]

2012/7/24 Anssi Kääriäinen <anssi.kaariai...@thl.fi>

> On 22 heinä, 23:01, Andrei Antoukh <n...@niwi.be> wrote:
> > Hello!
>

Hello!


> >
> > I started working on what had been proposed. In the commithttps://
> github.com/niwibe/django/commit/af887029integeintege614ca53d8...<
> https://github.com/niwibe/django/commit/af887029614ca53d8bdab23f0a40e...>
> > is
> > the initial implementation. It is based on the Anssi idea.
> >
> > I have some doubts about how to make things better. The proposed method
> > should return a list of sql statements:
> > ["COMMENT ON TABLE fooapp_sampletable IS 'Foo comment';"]
> >
> > or a list of tuples with sql statements with possible arguments:
> >
> > [("COMMENT ON TABLE fooapp_sampletable IS %s", ["Foo comment"])]
> >
> > Currently, the initial implementation, is a list of tuples, but I'm
> > wondering if the best option. I think that this is not usabe for commands
> > such as sqlall because the returned sql is not complete, if only partial,
> > failing to place them arguments.
>
> I do think the sqlall output should be usable as is. That is, the
> return value should be list of strings. This might mean that one has
> to create the SQL in a way that is susceptible to SQL injection
> attacks, but I don't see that as a problem. The post_sync method's SQL
> can't be ran or controlled by users, so there is no attack vector.
>

OK!



>
> I have a couple of questions:
>   1) How does this work in model inheritance situations? If the base
> model defines post_sync_sql(), wouldn't that be called again for the
> child models? Maybe it is possible to run the method only if it is
> defined in the current class directly. This should be possible by
> inspecting the model's __dict__ for the method instead of using
> hasattr.
>

Good observation!


>   2) What is the order in which the post_sync_sql methods are called?
> This could be important for some use cases (one view depends on
> another view for example). I guess the order is app for app in
> INSTALLED_APPS, models in the order they are found from app's
> models.py? The order should be tested and documented.
>

I'll think about it. Currently uses the same order as syncdb, even if not
the most appropriate for this.
We could think of a way to define the order of execution by the user.

I created the asociated ticket for this feature:
https://code.djangoproject.com/tidcket/18672<https://code.djangoproject.com/ticket/18672>

Andrey.



>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django should load custom SQL when testing

[Andrei Antoukh]

2012/7/26 Anssi Kääriäinen <anssi.kaariai...@thl.fi>

> On 26 heinä, 15:44, Danilo Bargen <gez...@gmail.com> wrote:
> > As the Tickethttps://code.djangoproject.com/ticket/16550is closed, I
> will
> > continue the discussion here on the mailing list.
> >
> > The reason for the ticket is that the Django test runner does not
> provide a
> > way to run custom SQL. This makes sense when users want to load custom
> SQL
> > *data*, but not when modifying the way the database works.
> >
> > Here's my comment from the issue comment thread:
> >
> > I agree very much with the comment above, in that there should be a way
> to
> > execute custom SQL before and after running syncdb. This should not be
> used
> > to load custom data, which is truncated anyways, but to alter the schema,
> > create new types or load extensions.
> >
> > In my case, I have to load the Postgres citext extension:
> >
> > CREATE EXTENSION IF NOT EXISTS citext;
> >
> > Because I have to create a database manually before running syncdb, I can
> > run my custom SQL in my database and then use it normally with Django.
> But
> > because the extensions are database specific, when running the tests
> (which
> > create their own database) they fail because the extension is not
> present.
> > This means that contrary to the comment by russellm, the *test database*
> > introduces an inconsistency between the way tables work during a test
> case
> > and the way they work in production.
> >
> > Another widely used extension is the hstore extension.
> >
> > As a workaround, I loaded the extension in the default postgres template.
> >
> > psql -d template1 -c 'CREATE EXTENSION citext;'
> >
> > But that means that any new database I create will contain the citext
> > extension, which might not be what I want.
> > Therefore I suggest that you add some pre syncdb and post syncdb hooks to
> > run custom SQL, and to mention in the docs that this should *not* be used
> > to load custom data, but to create new types and alter the schema, so
> that
> > it *matches the production database*. The data is truncated anyways.
> >
> > I vote for reopen.
> >
> > What do you think about this issue? Does anyone else agree that there
> > should be a way to run custom SQL in order not to create inconsistencies
> > between the test and production database?
>
> I agree, the ability to run raw SQL on syncdb would be useful to me.
>
> There is an idea for allowing post_sync SQL for models (see
>
> http://groups.google.com/group/django-developers/browse_thread/thread/2a1b068f56faae75
> ).
> It is clear one might want to define something else than post-sync SQL
> per model. The full set of hooks seems to be:
>  - per project (pre + post)
>  - per app (pre + post).
>  - per model (per + post)
> I wonder if we want to support all of those...
>

I think all three are useful, but I think the most useful: per model and
per project.


>
> The main gripe against any of the above is that once you start doing
> schema migrations, then you need to define the raw SQL in multiple
> places. Both in the app and in the migrations files. This raises the
> question if migration files is the right place to do this to begin
> with. The syntax could be something like:
>
> class Migration:
> apply_at = 'pre_sync'/'post_sync'/'migrate'
>
> Where 'migrate' is the default and means the migration is only applied
> on "manage.py migrate". pre_sync and post_sync would be applied on
> syncdb, and also on migrate if they are not applied before to the DB.
> The idea is that you could define the raw SQL only once, and it would
> be applied both to testdb on sync, and to production db on migrate.
>
> Still another idea: maybe we want to take the easy route and add
> special "pre_sync/post_sync" initial SQL file names. Document that
> these files should not contain data, just schema definition. Once we
> get the migrations framework in, then deprecate the whole "initial SQL
> in a directory" concept.
>

Having the hooks as a method or function, I find it much more interesting
than having them in files. Above that allows logic in these methods, which
with flat files is not possible.


>
>  - Anssi


Andrey.

-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django should load custom SQL when testing

[Andrei Antoukh]

2012/7/27 Anssi Kääriäinen <anssi.kaariai...@thl.fi>

> On 27 heinä, 12:01, Andrew Godwin <and...@aeracode.org> wrote:
> > Chiming in here on the migration front, my proposal for the next-gen
> > migration solution that'll potentially land in Django includes support
> > for just running chunks of raw SQL, so there's potential to auto-include
> > those in an initial migration.
> >
> > However, as you quite rightly point out, detecting changes is a pain.
> > Still, South sends the post_syncdb signal after model creation anyway,
> > so there's potential here for people who are lazy and don't want to port
> > their SQL across to migrations to have it Just Work. (and then Just Fail
> > when they want to change it). It should be good enough until I get the
> > migration stuff in, I'd say.
>
> The problem with post_syncdb is that it is fired also after db flush.
> Thus you get the SQL applied after every transactional test case, and
> that means you can't have DDL commands in the post sync SQL. This
> should be easy enough to fix by introducing a new kwarg which tells
> when the signal is sent (flush, sync, migrate).
>
> In addition we could add post_sync without sender for whole project,
> and post_sync with an App as sender when the app-loading refactor gets
> in.
>

I hate to use the signal post_syncdb for everything (project, app, model),
I think it is to use a feature for more things you should.


> It seems we would also need a pre-sync signal, too.
>

OK


>
> The only downside I can see is that this way you can't show the SQL as
> part of sqlall. Maybe we should just forget about this limitation, and
> continue with the signals.
>

The sql that runs on the handlers of signals, is the only one who can not
see part of the output of sqlall. But we can provide alternatives. See:
https://gist.github.com/3198204


>
> Once the migrations get in, we can make sure we have a nice framework
> for initial SQL, and then recommend not using the sync signals at all.
>
> So, maybe this would work:
>   - add a new signal for pre sync.
>   - add a new kwarg to pre/post sync which tells if the signal is
> fired on flush or on sync.
>   - add pre/post sync for the whole project.
>   - add pre/post sync for apps once the app-loading refactor gets in.
>
>  - Anssi


Now in my spare time I am working on a prototype. I hope suggestions and
improvements!
I think they should make a small refactor on the order of execution of the
command "syncdb". And I have doubts as to how to deal with the signals for
this.

Andrey.



> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Andrei Antoukh - <n...@niwi.be>
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/

"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.