Re: how to scale

[Andrey Khavryuchenko]

James,

 JB> On 10/3/07, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> The only thing django app should know is that there are two different
 >> *kinds* of servers: read-only and whole access.  It doesn't matter how many
 >> servers you have - you may well have pool of masters and pool of slaves, it
 >> doesn't matter.

 JB> For *your* case. 

In my case I need no replication nor clustering: project is fairly small
now.  My blog post and question here where to that point just a mind
excercise.

 JB> But remember that Django is expected to be generic and broadly usable,
 JB> which means that actually the decorators would have to support, say,
 JB> people who are sharding and want to have this view write to that
 JB> shard, and that view read from another, etc., etc., which means that
 JB> your decorator scheme basically comes down, in the end, to hard-coding
 JB> database or cluster or shard names alongside each view.

Does django orm support sharding now?  No, as far as I know.  So let's
solve problems as they come.

 JB> And again, I'd argue that this isn't something the application layer
 JB> needs to know or should have to know.

 >> And, completely unrelated to above, some companies made *business* decision
 >> to use MySQL Replication widely.  So if django doesn't support it, no big
 >> projects using django.

 JB> You mean no projects using MySQL Replication, 

Did I?  I think I've said exactly opposite.

 JB> which is very, very, VERY far from "no big projects" -- there are
 JB> plenty of big projects using Django, they're just using smarter DB
 JB> setups. Personally, I think MySQL's standard attitude of "well, just
 JB> completely rewrite your application when you need more than one
 JB> database" is the thing that's not suitable for big projects (where you
 JB> need to be thinking about, and planning for, configurable scaling
 JB> across multiple machines from the very start, not pausing and
 JB> rewriting when it's necessary); anybody who paid a consultant to get
 JB> that recommendation should ask for their money back.

Perhaps.  The company I've had in mind is not my company, instead it is a
leading search engine player in other country.  

And I won't like to discuss *that* stuff further.  

I am interested in technical discussion, and not "you shouldn't use this
(mysql replication), you should use that (mysql clustering)" - I know at
least two people that are interested specifically in using mysql
replication in django projects.

-- 
Andrey V Khavryuchenko - http://a.khavr.com/
Django NewGate -  http://www.kds.com.ua/djiggit/
Chytach - http://www.chytach.com/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: how to scale

[Andrey Khavryuchenko]

 AA> Am 03.10.2007 um 12:36 schrieb Andrey Khavryuchenko:

 AA> It's possible to use circular replication together with some free  
 AA> load balancing software. 

As I've told earlier, I know perfectly about circular replication.

 AA> Then there's MySQL Proxy that allows you to manipulate queries on
 AA> their way to the server to implement load balancing and, in the near
 AA> future, connection pooling. With the embedded lua engine, it should be
 AA> even possible to do query redirection/filtering but I haven't
 AA> investigated it yet. So I think that in the middle/long term there's
 AA> no need to put more logic in this regard into Django.

Reread what you've wrote:  MySQL Proxy manipulates *queries*, not
transactions.

Imagine something like:

sql: BEGIN TRANSACTION;

sql: SELECT * FROM something WHERE id=198273;

py: if queryset.count()==0:
py:MyObject.objects.create(**data)

sql: END TRANSACTION;

MySQL proxy or any other query-level solution has no way to know that first
select should go to the master server, so it would be routed to a slave,
that, what a coinsidence, is a second behind the master and the master
*just* got that specific record.  Voila, duplicate records and you can't
reproduce it unless you already know the cause.

Transaction-level solution requires some effort from developer to tell
which transactions modify data and which don't - exactly the stuff I wrote
in the post [1].

Footnotes: 
[1]  http://a.khavr.com/2007/10/01/how-to-use-mysql-replication-in-django/

-- 
Andrey V Khavryuchenko - http://a.khavr.com/
Django NewGate -  http://www.kds.com.ua/djiggit/
Chytach - http://www.chytach.com/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: how to scale

[Andrey Khavryuchenko]

 JB> On 10/2/07, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> Shouldn't it?  In case of MySQL replication (which is quite widely used)
 >> and transactions someone somehow *has* to decide where the whole
 >> transaction is going - to a master or to a slave.

 JB> In the case of MySQL, use MySQL Cluster, which designates a "SQL Node"
 JB> to receive the queries and route them through the cluster.

I know about MySQL Cluster.  The obvious disadvantage is that the whole
database should be in main memory.  Kiss goodbye to a bunch of cheap
servers sitting close to actual users.

There are also replication tweaks like multi-master replication, but they
are what they are - the tweaks.  If they break, you keep both pieces of
your db.

 >> The most obvious candidate for "someone" is the developer and the
 >> housekeeping stuff could be hidden in decorators.  I wrote a bit more on
 >> this at [1] (no working code yet, sorry).

 JB> So when you change the number of databases in your cluster, do you
 JB> also have to update *every* view in your application? 

Why?

The only thing django app should know is that there are two different
*kinds* of servers: read-only and whole access.  It doesn't matter how many
servers you have - you may well have pool of masters and pool of slaves, it
doesn't matter.

And, completely unrelated to above, some companies made *business* decision
to use MySQL Replication widely.  So if django doesn't support it, no big
projects using django.

-- 
Andrey V Khavryuchenko - http://a.khavr.com/
Django NewGate -  http://www.kds.com.ua/djiggit/
Chytach - http://www.chytach.com/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: how to scale

[Andrey Khavryuchenko]

 JB> As for database clustering, there's a philosophical issue here: Django
 JB> shouldn't need to know whether there's one database server behind it,
 JB> or five, or a hundred.

Shouldn't it?  In case of MySQL replication (which is quite widely used)
and transactions someone somehow *has* to decide where the whole
transaction is going - to a master or to a slave.

The most obvious candidate for "someone" is the developer and the
housekeeping stuff could be hidden in decorators.  I wrote a bit more on
this at [1] (no working code yet, sorry).

Footnotes: 
[1]  http://a.khavr.com/2007/10/01/how-to-use-mysql-replication-in-django/

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Chytach - http://www.chytach.com/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django Email Failure

[Andrey Khavryuchenko]

 k> I am trying to use the send_email facility from django.core.mail with
 k> the following
 k> in my settings.py

 k> EMAIL_HOST = 'smtp-server.cfl.rr.com'
 k> EMAIL_PORT = '25'
 k> EMAIL_HOST_USER = 'username'
 k> EMAIL_HOST_PASSWORD = 'password'

 k> I get the error message:

 k> SMTP AUTH extension not supported by server.

 k> I am sure the username and password are correct and am using
 k> RoadRunner in the central Florida area:

Are you sure that this login/password are for *sending*?  Try commenting
out EMAIL_HOST_USER and EMAIL_HOST_PASSWORD.


-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Chytach - http://www.chytach.com/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Data truncated for column 'text' at row 1

[Andrey Khavryuchenko]

 SD> Data truncated for column 'text' at row 1

[...]

 SD> The POSTing works 'cause when I return to the site I can see the post.
 SD> I guess this has something to do with unicode or something, but I
 SD> don't really know where to look.

 SD> Any hint is really appreciated!

This means exactly what's written there: data was truncated.  Check how
long 'text' column is and, probably, alter table to increase its size.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Using a Single Sign-on Server with Django

[Andrey Khavryuchenko]

 NAA> Does anyone know of a single sign-on server that can be used with
 NAA> Django? Similar to the approach taken by Cosign 
 NAA> or Pubcookie  but without the fluff that is
 NAA> LDAP and Kerberos. Just a simple database of users and passwords to
 NAA> authenticate against, but allows multiple Django-based sites hosted on
 NAA> the same server to connect to authenticate against it. Each site will
 NAA> have to keep a separate database for its user profile data if needed
 NAA> but if there exists a system that has profile management then that
 NAA> would be nice too.

I'd go with django-openid as a client and an openid server.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Capisdjango

[Andrey Khavryuchenko]

Chris,

 CH> Sure. Anyone to join me?

Show me a code and I'll join, since have a nagging need for such tool.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Using multiples version of Django on the same server

[Andrey Khavryuchenko]

 FS> I'm setting up a new server to host my Django projects and they don't
 FS> each use the same version of Django (0.95, 0.96 and svn). Is their a
 FS> way to configure each virtualhost to use their own version of Django ?

Put each version in it's own directory (say, django-0.95 and django-0.96)
and use PYTHONPATH to point each virtualhost's mod_python to the correct
version. 

-- 
Andrey V Khavryuchenko
Development - http://www.kds.com.ua 

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: need help for transaction with postGreSql....

[Andrey Khavryuchenko]

 s> I just wanna know how  to do the  trasnsaction with postgresql. If
 s> anyone has any clue / document, or has some working code on this
 s> issue, just send me the related excerpts.

Quick google search on "django transaction" reveals

http://www.djangoproject.com/documentation/transactions/

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Unable to get Django to output variables when calling from a different app

[Andrey Khavryuchenko]

 C> I'm curious, is there a way to do this without regenerating all of
 C> the code inside of the views.py file?

 C> I've been reading about setting up templatetags, would this work for
 C> what I want to do, and if so would it be better?

Regenerating?

The polls or latest_poll_list should somehow be put in the rendered
Context, it won't appear automagically.  And the easiest way to do this, is
to pass it explicitly.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: database permissions

[Andrey Khavryuchenko]

Steve,

 SB> Perhaps I am wrong, or just going about this the wrong way, but I have
 SB> been looking the Django documentation and asking around in #django and
 SB> nobody can (or do not want to) give me a straight answer and the minimum
 SB> database permissions Django needs to operate.  So I am assuming that
 SB> everybody is just giving Django free rein over the database, which I
 SB> think is a bad idea.

 SB> Is there any documentation anywhere that describes setting up correct
 SB> database permissions for use with Django.

It should strongly depend on the queries you're doing with django.  You may
inspect them with 
manage.py sql
manage.py sqlall
and similar commands together with simply looking into your db queries log
or django.db.connection.queries in DEBUG mode.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: db mock for unit testing

[Andrey Khavryuchenko]

Russ,

 RK> On 7/29/07, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> 
 >> I'm not using django's testing framework for several reasons:
 >> - I'm using django from 0.91 days and wasn't following django-users all
 >> this way
 >> - I use nose and twill for testing and they have no ready-to-use plugs
 >> into django testing (or I haven't been able to find it)

 RK> There aren't any built-in mechanisms for supporting nose or twill
 RK> tests. There are so many 'alternative' testing frameworks - we're not
 RK> going to add support for every single one of them.

I know and accept that.

 RK> However, it is very simple to add support for an external testing
 RK> mechanism - this is one of the original design considerations. See the
 RK> following for details:

 RK> 
http://www.djangoproject.com/documentation/testing/#using-a-different-testing-framework

I know that section exists.  When I mentioned 'ready-to-use plugs' above
I've meant exactly these hooks.  Someday I'll write nosetests plugin.

 >> But sqlite has no concat function and straightforward approach will lead to
 >> an exception.  Thus
 >> sqlite_conn.connection.create_function('concat', 2,
 >>lambda *args: ''.join(args))
 >> in DbMock setup.

 RK> This is a workaround required by your specific SQL requirements.
 RK> Similar problems would exist if you used Postgres TSearch2 extensions
 RK> in your queries, or any other DB-specific extension. It's not really
 RK> Django's responsibility to normalize every possible SQL query across
 RK> every possible backend.

I know.

 RK> However, it probably is Django's responsibility to provide a generic
 RK> hook so that you (as an end user) can add whatever normalizations your
 RK> application may require. Any suggestions on how to approach this
 RK> problem would be greatfully accepted.

Well, if I'm using sqlite as a test db backend, I can add nearly any
function I need in the manner described above.

What other usecases might be?

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: db mock for unit testing

[Andrey Khavryuchenko]

Russ,

 RK> On 7/29/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:
 >> 
 >> So what advantages are there to the mocking approach over just replacing
 >> the setting?

 RK> Genuine mocking (as opposed to this proposal) has one really big
 RK> advantage - it's lightning fast. All the db-calls get faked using a
 RK> cache-like setup, so it takes pretty much no time to run any db query.
 RK> The cost comes in keeping the mock data source up to date.

I first thought on logging sql queries with result and feed them via mock.
But the cost of supporting this in test-first development seems to high for
me.  Having to think out everything down to sql level kills most
development benefits of Django ORM for me.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: db mock for unit testing

[Andrey Khavryuchenko]

Malcolm,

 MT> That isn't an answer to the question Russell asked, though. You can get
 MT> exactly the same end-effect (using in-memory SQLite) if you specify
 MT> SQLite as the database engine in the settings file you use for testing.
 MT> Deriving your testing settings file from the main settings file is just
 MT> a matter of importing and replacing the right variable.

 MT> The advantage of specifying SQLite directly (and the drawback to trying
 MT> to "fake it") is that Django won't inadvertently use any MySQL- or
 MT> PostgreSQL-specific SQL constructs under the covers when calling the
 MT> database. There are places we consider the value of
 MT> settings.DATABASE_ENGINE when constructing the SQL and we may leverage
 MT> that more in the future.

 MT> So what advantages are there to the mocking approach over just replacing
 MT> the setting?

Ok, I'll be more wordy.

I'm not using django's testing framework for several reasons:
 - I'm using django from 0.91 days and wasn't following django-users all
   this way
 - I use nose and twill for testing and they have no ready-to-use plugs
   into django testing (or I haven't been able to find it)

Also simply overriding DATABASE_ENGINE to sqlite leads to problems.
E.g. my code has 
   profiles = models.User.objects.extra(where=[
"'%s'" % identity_url + "like concat(login, '%%')"])
But sqlite has no concat function and straightforward approach will lead to
an exception.  Thus 
sqlite_conn.connection.create_function('concat', 2,
   lambda *args: ''.join(args))
in DbMock setup.

So, back to business..  

My intention was to speedup unit tests and make initial test data setup
easier.  I am not proficient in django testing framework and couldn't
utilize it with nosetests.  DbMock solves my issue.

Sure, I may be wrong and the proper way is to write nose test runner
and file patches that provide more mysql or postgresql compatability for
testing.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: db mock for unit testing

[Andrey Khavryuchenko]

Russell,

 RK> On 7/27/07, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> 
 >> Then I've paused and wrote DbMock class for django that uses some black
 >> magic to steal django db connection and substitute it with temporary sqlite
 >> in-memory db.

 RK> How is this different to the default Django behavior if you specify
 RK> SQLite as your database? Can't you get exactly the same behavior by
 RK> creating a test_settings.py file that contains:

 RK> from settings import *
 RK> DATABASE_BACKEND='sqlite'

 RK> and then run:

 RK> ./manage.py --settings=test_settings test

 RK> ?

Just replied on django-developers: I need database mock to speedup tests.
Mocking them into in-memory sqlite was the simplest way to reach this w/o
losing flexibility.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

db mock for unit testing

[Andrey Khavryuchenko]

I do hardcore test-driven development and hate when tests hit my mysql
database (even local one).  Things get only worse when testcases start
demanding radically different datasets.

Then I've paused and wrote DbMock class for django that uses some black
magic to steal django db connection and substitute it with temporary sqlite
in-memory db.

Here it is: http://www.djangosnippets.org/snippets/345/

So far, it works w/o any issues on my pet project.  Will be glad to hear
any comments and, especially, bugs found :)

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Can Django call cscript.exe to run vbscripts

[Andrey Khavryuchenko]

 b> Does someone have a solution ?

Have you tried subprocess module mentioned before?  What was your
experience? 

BTW, it's nothing django-specific in calling external software from your
python app...

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: transaction commit

[Andrey Khavryuchenko]

Nimrod,

 NAA> On 7/25/07, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> >> Yes, I read carefuly your question and thought the answer was
 >> >> straighforward.   I don't understand why you don't want decorators, but 
 >> >> you
 >> >> could just check the decorator definition to read what it does and copy
 >> >> it's code.  All you need is decorator name and grep over django sources 
 >> >> :)
 >> 
 NAA> Aren't decorators usable only in views?
 >> 
 >> Decorators are python feature.  Django views are regular functions, just as
 >> any other.

 NAA> Sorry I wasn't very clear on that last message, I meant the Django
 NAA> db.transactions decorator functions. I am thinking they only work when
 NAA> you use them in a view. When you try to use them elsewhere, e.g. a
 NAA> Django-based command line app, see below...

No, you're wrong.  Transaction decorators work not only in view functions.
They work anywhere in the web app, provided transaction middleware is
included.

 NAA> At least that's what I think.  I have a project that required a
 NAA> separately running process (running as daemon) that needed access to
 NAA> the Django models. Since the code in the daemon is not run in the
 NAA> context of a view (no triggering of TransactionMiddleware),
 NAA> transactions decorators do not work. I get:
 >> 
 NAA> TransactionManagementError: This code isn't under transaction management
 >> 
 >> Not having read transaction code, can't say what haven't worked in your
 >> case.  Definitely, something wasn't initialized :)

 NAA> You're right. The app I am talking about is not a web app, rather a
 NAA> command line app that I made as a supplement to the web app I made
 NAA> using Django. Since it makes no sense to reinvent the wheel, I just
 NAA> used Django's ORM system in the command line app. When I tried using
 NAA> transactions using the transaction decorator functions I get the error
 NAA> above. Likely cause is that the command line app is not using
 NAA> middleware, ergo not using TransactionMiddleware, then I needed a way
 NAA> to initialize and manage transactions manually.

A ticket in my queue is similar, so I understand you perfectly.

Quick glance on django.middleware.transaction shows:

def process_request(self, request):
"""Enters transaction management"""
transaction.enter_transaction_management()
transaction.managed(True)

def process_exception(self, request, exception):
"""Rolls back the database and leaves transaction management"""
if transaction.is_dirty():
transaction.rollback()
transaction.leave_transaction_management()

def process_response(self, request, response):
"""Commits and leaves transaction management."""
if transaction.is_managed():
if transaction.is_dirty():
transaction.commit()
transaction.leave_transaction_management()
return response

So in cmd line app you have to call 
   transaction.enter_transaction_management()
   transaction.managed(True)
in initialization and 
   transaction.leave_transaction_management()
in the end.  Not tested, but should work.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Auth framework - initials et of users/groups ?

[Andrey Khavryuchenko]

Przemyslaw,

 PW> I'm just trying to do is using django.contrib.auth framework.

 PW> What I'd need is the possibility to create initial set of groups, users
 PW> and user-group assignements, when 'syncdb' is performed.

 PW> One option I see is to add custom statements to one of SQL files used to
 PW> initialize my application's model. Yet it's a bit ugly, isn't it ?

Create this data in console or in the script and then use 
   manage.py dumpdata
to save then in json format.   Use 
   manage.py loaddata 
as you need

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: transaction commit

[Andrey Khavryuchenko]

Michal,

 MK> I suppose you read my question attentively and therefore you know that
 MK> I searched the web (and django documentation of course including the
 MK> transaction page). And I suppose you know that I was looking for
 MK> example how to use transactions without decorators.

 MK> The problem is that there's no example how to use transactions without
 MK> decorators in the documentation, there's no example here in this
 MK> mailing list... I found the example in
 MK> django.db.transaction.py

Yes, I read carefuly your question and thought the answer was
straighforward.   I don't understand why you don't want decorators, but you
could just check the decorator definition to read what it does and copy
it's code.  All you need is decorator name and grep over django sources :)

 MK> But thanks anyway ;-)

Luck!

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: transaction commit

[Andrey Khavryuchenko]

 MK> Hi,
 MK> I would like to use transactions (in MySQL and InnoDB). I know there
 MK> are decorators but I don't want to use them now. Is there any example
 MK> how to use transactions without decorators? I couldn't find anything
 MK> on django website, this user-group...

Quick google on "django transactions" shows
http://www.djangoproject.com/documentation/transactions/ as the first
result.

And it has a cut down example of manual commit usage.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django Blocking/ Background Jobs

[Andrey Khavryuchenko]

Paul,

 PB> On Jul 23, 3:50 pm, Andrey Khavryuchenko <[EMAIL PROTECTED]> wrote:
 >> Paul,
 >> 
 PB> Are there any better approaches? I'm sure someone has needed
 PB> background tasks liked spidering, querying whois, downloading
 PB> images,... before, without stalling the whole process.
 >> 
 >> I'm using plain cron jobs that do regular jobs.  There are plenty of
 >> example snippets describing how to set up necessary environment.

 PB> How do you get data to and from them ? I don't need scheduled tasks,
 PB> but background tasks.

Pretty everything these scripts need is in the db.  So I use regular Django
ORM and the rest of data is passed via cmdline params.

 PB> The jobs are running just a few seconds and i need quick response

How you're going to serve the results of these jobs to a user?

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django Blocking/ Background Jobs

[Andrey Khavryuchenko]

Paul,

 PB> Are there any better approaches? I'm sure someone has needed
 PB> background tasks liked spidering, querying whois, downloading
 PB> images,... before, without stalling the whole process.

I'm using plain cron jobs that do regular jobs.  There are plenty of
example snippets describing how to set up necessary environment.  

Basically, make sure that your project is in your PYTHONPATH and
DJANGO_SETTINGS_MODULE env variable is pointing to your project's
settings.py 

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

suggestion for sql error handling

[Andrey Khavryuchenko]

Hi!

Sometimes we all get sql errors from our code.  Nevertheless the traceback
kind of 
  File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 242, 
in count
cursor.execute("SELECT COUNT(*)" + sql, params)
  File "/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 19, 
in execute
return self.cursor.execute(sql, params)
  File "/usr/lib/python2.4/site-packages/django/db/backends/sqlite3/base.py", 
line 95, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: near "?": syntax error

isn't quite informative.

What I would like is to have sql clause and params in the exception I get.

The only thing that needs to be modified is *CursorWrapper.execute for each
backend.  Just like this for SQLiteCursorWrapper:

def execute(self, query, params=()):
query = self.convert_query(query, len(params))
try:
return Database.Cursor.execute(self, query, params)
except Exception, e:
e.query = query
e.params = params
raise

Any reason for *not* doing this?  
Would be such featurelet desireable?
If yes, I'll prepare a patch against the trunk.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Can Django call cscript.exe to run vbscripts

[Andrey Khavryuchenko]

 b> I'm an Windows System Administrator and for my job i often use
 b> vbscripts to automate my tasks. I was wondering if Django can call
 b> cscript.exe to execute a vbscript file (Offcourse django is assumed to
 b> run under Windows)? I know that i can also script wmi with python but
 b> i'm allready mastering vbscript and have a huge library of function
 b> vbscripts.
 b> Can someone help me pls ?

Django app is a plain python script.  So check python doc for starting
external applications (module os).

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua 
Call akhavr1975 on www.gizmoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

custom order_by?

[Andrey Khavryuchenko]

Folks,

In spirit of recent question regarding custom 'where'...  Is there a way to
put a custom ORDER BY into QuerySet?  I have an sql function that does all
the math on db side and would like to use it at order_by clause

I quickly skimmed _get_sql_clause and haven't found any way to incorporate
something like "ORDER BY calc_post_rating(id)".

Am I wrong or is the only way to have this to override/hack
_QuerySet._get_sql_clause?

TIA!

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/
Development - http://www.kds.com.ua OR skype://akhavr

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: trunk, sqlite backend bug with date handlig

[Andrey Khavryuchenko]

Malcolm,

Thanks, your answers helped to trace down the problem.  It appeared that
just sqlite connection was opened without proper 'detect_types' parameter. 

 MT> On Sat, 2007-07-07 at 23:04 +0300, Andrey Khavryuchenko wrote:
 >> 
 MT> On Sat, 2007-07-07 at 20:57 +0300, Andrey Khavryuchenko wrote:
 >> >> While I'm digging in sqlite backend sources, may anyone share his
 >> >> experience?
 >> 
 MT> I just tried this with some models I have here using sqlite and they all
 MT> loaded DateTime fields as Python datetime instances. Saving and
 MT> reloading didn't seem to change the type, either.
 >> 
 MT> So this doesn't seem to be a universal failure. Would be interesting to
 MT> know what's going on, though.
 >> 
 >> I'm observing this in unit tests where I do some black magic to substitute
 >> usual mysql db backend to temporary pysqlite2 in-memory db.

 MT> That's almost certainly the problem. In the sqlite backend, we install
 MT> various converters, including on to create datetime objects.

 MT> Why not just use a settings file that has DATABASE_ENGINE="sqlite3" and
 MT> set the name to ":memory:"? Telling Django that the engine is MySQL and
 MT> then using something else is going to lead to no end of problems, I
 MT> suspect.

Because app in usual mode uses mysql and some tests rely on initial mysql
data.  

And anyway, playing with settings.py file and it's content within unit
tests smell me more than substituting db connection in setup and restoring
it in teardown.

 >> So, the question to help to clarify what I've done wrong in that magic:
 >> 
 >> When the object is created in django.db.models.query there's a call around
 >> line 205
 >> 
 >> obj = self.model(*row[:index_end])
 >> 
 >> Should the  data be already converted to proper data types or not?

 MT> Yes, it should already be using the correct Python types at this point
 MT> (e.g. datetime).

Exactly.  That solved the issue.

Thanks again.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: trunk, sqlite backend bug with date handlig

[Andrey Khavryuchenko]

 MT> On Sat, 2007-07-07 at 20:57 +0300, Andrey Khavryuchenko wrote:
 >> While I'm digging in sqlite backend sources, may anyone share his
 >> experience?

 MT> I just tried this with some models I have here using sqlite and they all
 MT> loaded DateTime fields as Python datetime instances. Saving and
 MT> reloading didn't seem to change the type, either.

 MT> So this doesn't seem to be a universal failure. Would be interesting to
 MT> know what's going on, though.

I'm observing this in unit tests where I do some black magic to substitute
usual mysql db backend to temporary pysqlite2 in-memory db.

So, the question to help to clarify what I've done wrong in that magic:

When the object is created in django.db.models.query there's a call around
line 205

obj = self.model(*row[:index_end])

Should the  data be already converted to proper data types or not?

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

trunk, sqlite backend bug with date handlig

[Andrey Khavryuchenko]

Hi!

Had anyone noticed that loading DateField and DateTimeField from sqlite
in-memory db results in getting plain strings instead of datetime objects?

Code:

f = models.Feed(feed_url='no such url', is_active=True)
f.save()
post = models.Post(feed = f, title='test post', link=self.url)
post.save()

assert models.Feed.objects.count() == 1, 'should be single feed'
assert models.Post.objects.count() == 1, 'should be single post'

print '!!!', type(post.date_created), \
  models.Post.objects.all()[0].date_created, \
  type(models.Post.objects.all()[0].date_created)
assert type(models.Post.objects.all()[0].date_created) == \
   type(post.date_created)

It prints the following and fails on last assert:

 >> begin captured stdout << -
!!!  2007-07-07 

- >> end captured stdout << --

While I'm digging in sqlite backend sources, may anyone share his
experience?

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: proper and easy way to urlencode unicode string?

[Andrey Khavryuchenko]

 MT> I've just this minute committed a fix in [5461]. So let me know if there
 MT> are any further problems there, but it should at least not explode now.

Thanks, see the change, will check how it performs now.

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: proper and easy way to urlencode unicode string?

[Andrey Khavryuchenko]

 MT> On Mon, 2007-06-11 at 05:03 +0300, Andrey Khavryuchenko wrote:
 >> Malcolm,
 >> 
 MT> On Tue, 2007-06-05 at 17:47 +0300, Andrey Khavryuchenko wrote:
 >> >> Well, I think here you've missed the point.  Python fails *before*
 >> >> iriencode - during urlencode.  Check the traceback again:
 >> 
 MT> Ah, I see. For some reason I forgot to switch the urlencode filter over
 MT> to use django.utils.http.urlquote(). I'll fix that in the morning.
 >> 
 >> Was the change made?  I feel I still miss it in the svn..

 MT> No. I was sick last week and didn't get to it. 

Ouch.  Hopefully it's ok now..

 MT> And, since there was no ticket for this, I forgot about it (we keep
 MT> saying this happens; it's true).

 MT> Thanks for the reminder. I'll do it tonight.

Post a ticket for that?

-- 
Andrey V Khavryuchenko
Django NewGate -  http://www.kds.com.ua/djiggit/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: proper and easy way to urlencode unicode string?

[Andrey Khavryuchenko]

 MT> Have a look at the iriencode filter (only in the unicode branch, so
 MT> you'll need to read docs/templates.txt from the source). This handles
 MT> the last stage of encoding to ASCII.

Thanks for pointer.

 MT> You cannot necessarily skip the urlencode portion, but you may be able
 MT> to. The reason is that the IRI -> URI conversion algorithm does _not_
 MT> encode things like '%', so it is safe to apply to something that has
 MT> already been partially encoded. However, if your proto-URL string
 MT> contains a '%' that should be encoded (e.g. "100%-guaranteed"), you will
 MT> need to pass it through urlencode first.

 MT> So {{ tagname|urlencode|iriencode }} is completely safe, although
 MT> possibly redundant (and even incorrect if tagname was something that had
 MT> already been URL encoded).

Well, I think here you've missed the point.  Python fails *before*
iriencode - during urlencode.  Check the traceback again:

> {{ tagname|urlencode }}
> 
> But when the 'tagname' contains non-ascii symbols, urlencode barfs:
> 
> KeyError at /
> u'\u0420'
> Request Method:   GET
> Request URL:  http://localhost:8088/
> Exception Type:   KeyError
> Exception Value:  u'\u0420'
> Exception Location:   /usr/lib/python2.4/urllib.py in quote, line 1117
> Template error


 MT> You might also want to read the section call "URI and IRI handling" in
 MT> docs/unicode.txt because urllib.quote() is not the most bullet-proof
 MT> choice when working with unicode strings. Reading your email, I just
 MT> realised I have forgotten to mention the urlencode and iriencode filters
 MT> in that section, but that will be fixed when I next sit down to commit
 MT> bug fixes.

Yes, I've read this document (again) but 

  {{ tagname | urlencode | iriencode }}

won't work due to the same error.

-- 
Andrey V Khavryuchenko
http://www.kds.com.ua

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

proper and easy way to urlencode unicode string?

[Andrey Khavryuchenko]

Folks,

I'm working on multiligual app using unicode branch and hit an interesting
bit yesterday.

I'm trying to present a (cyrillic) tag in url-ready form.  Usually this is
solved by

{{ tagname|urlencode }}

But when the 'tagname' contains non-ascii symbols, urlencode barfs:

KeyError at /
u'\u0420'
Request Method: GET
Request URL:http://localhost:8088/
Exception Type: KeyError
Exception Value:u'\u0420'
Exception Location: /usr/lib/python2.4/urllib.py in quote, line 1117
Template error

My current (temporary) solution is to encode this manually, in python:

urllib.quote(tagname.encode('utf8')

But this is not DRY and makes me think there's a better way.

Or it's time to add a custom filter?

-- 
Andrey V Khavryuchenko
http://www.kds.com.ua

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---