I can't figure it out: "manage.py dbshell" gives "Access denied for user 'ODBC'@'localhost'"

[Jumpfroggy]

Hey All,
I've had this problem for a while and still can't figure it out.  I
have django setup and working on my windows machine.  I can run
manage.py syncdb and other commands, but for some reason the dbshell
command doesn't work.  It just gives this error:

> manage.py dbshell
> ERROR 1045 (28000): Access denied for user 'ODBC'@'localhost' (using 
> password: NO)

I modified django/db/backends/mysql/client.py like another poster did
to see what credentials were being passed, and I got this:

client.py:
...
print "self.executable_name:", self.executable_name
print "args:", args
os.execvp(self.executable_name, args)
output:
> self.executable_name: mysql
> args: ['', '--user=test', '--password=test', 'test']
> ERROR 1045 (28000): Access denied for user 'ODBC'@'localhost' (using 
> password: NO)

I also tried the following lines, with the same results:
print "os.execlp:"
os.execlp(self.executable_name, "-?")

print "os.execvp:"
os.execvp(self.executable_name, ["-?"])

These lines should simple call "mysql.exe" and print out the help
file, but instead it seems like mysql.exe is always running without
any arguments.

Does anyone have a clue why mysql.exe isn't seeing the arguments?  I'd
love to figure this out and be able to use dbshell from now on.
Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: I can't figure it out: "manage.py dbshell" gives "Access denied for user 'ODBC'@'localhost'"

[Jumpfroggy]

Alright, after some more searching I figured out my answer.  First
off, the test statements I posted were wrong.  The first argument
passed to execlp is the executable name, so:
> os.execlp(self.executable_name, "-?")
Should be
> os.execlp(self.executable_name, self.executable_name, "-?")

That works now.  So I applied the same thing to the original client.py
file:
def runshell(self):
-args = ['']
+args = [self.executable_name]
db = settings.DATABASE_OPTIONS.get('db',
settings.DATABASE_NAME)
user = settings.DATABASE_OPTIONS.get('user',
settings.DATABASE_USER)
...
print "self.executable_name:", self.executable_name
print "args:", args
os.execvp(self.executable_name, args)
And I get this:
> self.executable_name: mysql
> args: ['mysql', '--user=test', '--password=test', 'test']
>
> Welcome to the MySQL monitor.  Commands end with ; or \g.
> Your MySQL connection id is 212
> Server version: 5.0.51a-community-nt-log MySQL Community Edition (GPL)
>
> Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
>
> mysql>

So it looks like passing '' as the first argument to execvp() on
Windows for mysql doesn't work.  If using self.executable_name as the
first arg works on other platforms, it looks like this should be
changed to fix this bug and make client.py more cross-platform.

Can anyone see if it works on unix?  Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

ManyToManyField() causes bugs when using 'ModelName' instead of 'self'

[Jumpfroggy]

Summary: django should throw an exception when a model has its own
name as the 'to' field of a ManyToMany() insead of 'self'.

I had a field like this:

def Item(Model):
name = CharField(max_length=100)
related_items = ManyToManyField('Item', blank=True)

But this caused some very strange bugs.  For example:

item1 = Item.objects.create(name='First')
item1.save()

item2 = Item.objects.create(name='Second', related_items=[item1])
item2.save()

print item2.related_items
> []
print len(item2.related_items)
> 0

The correct way, as stated in the docs, is to use 'self' for a
recursive relationship.

def Item(Model):
name = CharField(max_length=100)
related_items = ManyToManyField('self', blank=True)

(NOTE: This is pseudo/test code, may contain typos)

When I use this method, it works fine.  However, if the first method
is incorrect and produces functionally invalid results, then there
should be a model validation check for this and an exception if it's
found.  Basically, if a model has a ManyToManyField() with its own
name as the 'to' parameter, an exception should be raised.

I understsand that the current functoinality & docs are working as
they're supposed to.  However, it would also be nice for django to
raies an exception instead of silently failing and producing invalid
results.

Have I missed anything?  I may be misunderstanding the problem here,
but I have managed to fix my problem and figure this could save others
from the same trap.  I'd like to get some input before submitting a
ticket.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Figuring out my memory usage for django + wsgi + apache prefork

[Jumpfroggy]

I'm running a bunch of django apps on my shared host with an 80mb
memory limit.  I have a bunch of very low-traffic sites I want to keep
running, but use as little memory as possible, so my goal is to
minimize the memory usage for each of these very small sites.  I'm
just starting to learn about apache/python/django memory usage, so I
figure there's a lot I just don't know right now.

I'm running on webfaction with:
Apache 2.2.3 with the Prefork MPM
Python 2.5.2
Django 1.0.2 (as stated in webfaction's automatic setup, though django/
__init__.py shows VERSION = (1, 1, 0, 'beta', 1))

I'm measuring against the RSS.  A sample of one of my apps:

RSSCommand
3624   .../apache2/bin/httpd -f .../apache2/conf/httpd.conf -k start
15716  .../apache2/bin/httpd -f .../apache2/conf/httpd.conf -k start

In this case, I've got one apache supervisor process using a small
amount of memory (3.5MB) and a single worker process using a good
amount (15.3MB).  When I first start apache, the worker process starts
small (<4MB), but increases with the first page access (>12MB) and
stays there until the next restart.

Ideally, I'd like for the worker threads to release all the memory
they're using when idle to get back down to their initial memory
usage.  I realize that there will be some performance tradeoffs for
this, but in this case these sites have very low requirements so it
seems worth the tradeoff.

I've already played with apache's httpd.conf settings:
ServerLimit 1
StartServers 1
MinSpareServers 1
In an effort to remove the second worker thread (so I have one process
using 13mb instead of two).  However, at this point apache still
spawns another process when usage spikes slightly, and then the second
thread remains until restart.

I've tried investigating apache MPM's, apache settings, and python
memory profilers, but haven't made much progress.  Some things I'd
like to try to figure out:
-What is using memory in my apache worker threads?  Is that the
baseline django memory usage, or are there ways to decrease this
minimum?
-What python techniques could I be using to decrease the long-term
memory usage? (not leaks, just persistent memory usage)
-What apache/wsgi config changes should I be looking into?  Different
MPM?

Any comments/suggestions are appreciated, I know I'm probably missing
some obvious things here.  Thanks.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Using the @login_required() decorator with #hash url's

[Jumpfroggy]

Currently, the @login_required() decorator does not preserve any #hash
part of the URL.  So if I go to /page#section1 without being logged
in, I'm redirected to /login?next=/page#section1.  After logging in,
I'm redirected to /page (without the hash part).

I read in another thread here that the hash is not sent as part of the
request headers, so the django server doesn't have this information.
However, I still need to be able to preserve the hash part of URL's
when using the login feature, since the hash can alter the page in
AJAX apps.

My question: does anyone have any ideas how to go about implementing
this?  My first thought is to modify the @login_required() decorator,
so that it spits out some javascript to 1) capture the hash of the url
and then 2) redirect to the login page.  This may break the back
button in browsers.

Does anyone else have ideas of how to do something like this?  Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: New to unit testing, need advice

[Jumpfroggy]

> besides the testing issues (which are certainly a heated debate!), i have to 
> say that my Django projects became far better organized and a lot more 
> flexible when i learned to put most of the code on the models, and not on the 
> views.

This is a pretty interesting topic.  Coming from a PHP background, I
started out with very messy code (each .php file contained it's own
program logic at the top).  From there I went to Javabeans (tomcat)
and got a crash course in MVC & OO web apps, almost to an extreme I
wouldn't want to duplicate.  Now that I'm in django, I have a nice
middle ground.

There are certain ideals (ie. programming paradigms) that might make
unit testing easier.  One is separating the different parts of MVC
(model, view, controller).  Controller in our case is mostly the
django code (if I'm not wrong).  The views are our views.py
functions.  The models are our models.py file.  But there's the
unspoken part of business logic.  By default, most django apps seem to
have very dumb models.py files (the classes are simply wrappers around
database rows & tables), and all the logic is in views.py.  However,
the business logic should really be pulled out of views.py, whether
that goes into heavy models.py objects (like Javier has), or a
separate business_logic.py file.

It helps to keep asking yourself, what is this line of code doing?

For example, in views.py:

def handle_add_item_form(request):
form = AddItemForm(request.POST)
if form.is_valid():
item = Item.objects.create(
name=form.cleaned_data['name'],
price=form.cleaned_data['price'],
status=form.cleaned_data['status'],
)
if item.status == 'out of stock':
# Order 100 more items for later
order_more_items(item, 100)
email_admin_about_oos(settings.ADMINS)

This handles a simple form to create an item.  However, there's
clearly business logic here... the whole "status == 'out of stock'"
sticks out like a sore thumb.  It'd be better like this:

in views.py:

def handle_add_item_form(request):
form = AddItemForm(request.POST)
if form.is_valid():
item = ItemManager.create(
name=form.cleaned_data['name'],
price=form.cleaned_data['price'],
status=form.cleaned_data['status'],
)

and in managers.py:

class ItemManager:
def create(name, price, status):
item = Item.objects.create(
name=form.cleaned_data['name'],
price=form.cleaned_data['price'],
status=form.cleaned_data['status'],
)
if item.status == 'out of stock':
# Order 100 more items for later
order_more_items(item, 100)
email_admin_about_oos(settings.ADMINS)

You have to type a bit more (to create all those managers, or service
objects, or whatever you call them), but the benefit is that business
logic is now in one place.  Views.py should only be the dumb glue that
connects the HTTP requests with the service objects, and returns the
results.

Like Daniel said, models.py are usually so simple/dumb that you don't
need to test them much.  Your views.py should be the same way.
They're simply glue, so the only things that will go wrong are simple
logic errors and typos.  All the complicated stuff is in the business
logic objects (the managers.py here), and that's what you spend a lot
of time unit testing.

In this example, you can use the ItemManager.create() function
anywhere you create items... adding new ones, copying existing ones,
importing items from a file, etc.  In the old example, you'd have to
duplicate the views.py code in each view that handles this situation
and test each one.  You could put all these functions into the
models.py file if you wanted, you'd just end up with huge models.py
objects, and you might also run into cases where there is not an
obvious place to put a bit of business logic.  Not every business
logic function corresponds exactly to one model.

Another great change... with all your logic separated like this, if
you have to change something very complicated (like the way the whole
system handles out of stock, ordering more inventory, permissions,
etc), you have one place to change & test.  It's great.

So do I use this?  No, all my code is in wrapped up in my views.py and
I have no unit testing.  I try to break out duplicated functionality
into their own functions, but that's more for convenience than true
separation.  I'd recommend starting with the "keep them separated"
mentality, since it gets harder and harder as a project goes on.

Sorry for the long rant, but it touched on something I've been
observing in my own code.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to 

Re: Using the @login_required() decorator with #hash url's

[Jumpfroggy]

On Jul 16, 11:05 am, Javier Guerra  wrote:
> maybe urlencode()'ing the '/page#section1' parameter?

The first problem is that the django server only receives the '/page'
part of the URL.  The browser itself holds onto the '#hash' part and
doesn't transmit that to the django server at all, so the
login_required() decorator calls request.get_full_path() and gets '/
page', so that's what it uses.  It doesn't look like there's a simple
way to get the '#hash' from django.

Once I get it, yes I can use urlencode(full_path_with_hash) and use
that instead.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Using the @login_required() decorator with #hash url's

[Jumpfroggy]

> The only way I know around this involves leaning on JavaScript to
> get the URL, split off the #hash bit from it, and sneak it in as
> a hidden element on the login form.

Yeah, I ended up doing this on the /login page.  Basically:


$(function() {
$('#url-hash').attr('value', window.location.hash);
});




...


This works for firefox.  The bigger problem?  Internet Explorer does
not preserve the hash when following redirects!!!  So in this case,
you have *no opportunity* to save the hash via javascript on the login
page.  So instead of sending the 'location' header to redirect, you'd
have to send something like this HTML page:


window.location = '/login?next=' + escape(window.location);


Besides being totally annoying and hackish, it breaks the back
button.  You can't save the hash with javascript on the login form
with IE, because to get to the login form you had to use a redirect,
and when you use a redirect the hash is lost... this means the login
page never sees the hash in any way when using a redirect.  It's
amazing that something seemingly simple and fundamental ends up being
such a hack.

So besides my horrible back-button-breaking hack, does anyone else see
a way to do this?

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

How do I minimize memory usage with WSGI and Apache?

[Jumpfroggy]

I'm hosting a bunch of django apps on a shared host with 80MB of
application memory (Webfaction in this case).  I've got a bunch of
apps that are very infrequently used, but need to be online.  I've
already changed the httpd.conf:
ServerLimit 1
Instead of the default "ServerLimit 2".  With the default, each app
spawns 3 apache processes; one manager, 2 worker processes.  With
ServerLimit 1, there is only a manager and a single worker.  This cuts
down on a lot for low-traffic sites.

However, I'd like to see what the bare minimum memory usage can be.
I've made sure DEBUG = False.  I also separated the static media to a
separate server with MEDIA_ROOT and MEDIA_URL.  When I first start
apache, the manager and worker processes use about 3MB (RSS) each.
After the first access with a browser, the manager stays the same but
the worker uses 16MB and stays there until apache is restarted.

Ideally, I'd love it if the memory usage go back to the initial state
(3MB) after a period of inactivity.  I'm curious; what is being stored
in memory?  Are there other things I can do to reduce memory usage?

The one thing I've seen is the "MaxRequestsPerChild" apache setting.
As far as I can tell, it'll recreate the worker process after X
requests.  But this seems like it might adversely affect performance,
since I need something time-based, and not # requests based.  With
MaxRequestsPerChild, I'm guessing if I set it to 10 but only had 9
requests, it would sit there using up memory until the tenth request,
which is not ideal.  With MaxRequestsPerChild set to 1, I'd be
guaranteed low memory usage but would also recreate the worker process
for each request, which seems like a bad idea.

My goal is to have many apps running at bare-minimum memory usage, and
only the frequently accessed sites taking up ~16MB memory each.  And
when the active sites are dormant (late at night, etc), I'd like those
to lower their memory use as well.  Is there any way to do this?

Thanks! (and apologies if I've missed some obvious documentation...
I've gone through a few guides, but nothing seems to have helped).
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Jumpfroggy]

Wow, lots of good feedback here.  Thanks!

On Aug 9, 6:17 pm, Graham Dumpleton 
wrote:
> Replace use of mod_python with mod_wsgi.

I'm using mod_wsgi for everything already, forgot to say.  When I
researched, it seemed like the newer/better way to do django.

> Ensure you use prefork MPM
> for Apache and delegate Django instances to run in mod_wsgi daemon
> mode process.
> Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
> PREFORK. Hmmm, that was a bad stuff up. :-)

I just started using the worker MPM for one of my apps.  It didn't
seem to give any performance or memory savings, since my app is so
small/simple.  I'll keep an eye on it for later.

Using daemon mode gives me auto-reloading by touching the .wsgi file
(yay), but it uses more memory.  Instead of supervisor (3MB) and
apache worker/WSGI/Django (16MB) processes, now I have supervisor
(3MB), apache worker (3MB), and wsgi/django (15MB).  Not a huge gain,
but just a bit more.  I'm willing to live with that for the auto-
reloading benefit, however.

> Having done that, you can use inactivity-timeout option
> for mod_wsgi daemon processes to have the process restarted and
> returned to low memory footprint idle process after no use for a
> while.

This is exactly what I wanted!!!  It's funny how when I finally read
the description for something, it matches exactly what I'm looking
for.  I just didn't know where to look.  I just tried it, and yes my
memory baloons up to 16MB, then comes back down later.  The only
strange thing is that I set it to 10s (very short) to test out, but it
takes a while for the process to actually be reclaimed.  If I put
inactivity-timeout=10, then load the page, then close the browser,
memory usage should be reclaimed in 10 seconds, right?  It seems more
like 1 minute wait, which is fine but strange to me.  Maybe there's
only polling at certain intervals (checks every 1 minute), so the
resolution isn't that fine.  Maybe it's dependent on the deadlock
timeout?  Don't know.

> Note, just ensure you don't keep loading mod_python after having
> switched to mod_wsgi as mod_python will still consume some extra
> memory when not in use.

Yeah, never touched mod_python, so no problem here.

On Aug 10, 6:44 am, Dj Gilcrease  wrote:
> I just went though this process on Webfaction as well, so here is my
> apache conf, it might help...

Thanks for the post, good to see another webfaction user here.  It
always helps to see other people's real-world configs, since examples
usually leave out the key bits.

On Aug 10, 10:25 am, Dj Gilcrease  wrote:
> >> MaxRequestsPerChild 5000
> > Don't see much point for MaxRequestsPerChild.
> This just restarts the child after 5k requests, which helps clear out
> some memory on occasion (I am still trying to get rrd or something run
> on on Webfaction so I can actually map memory usage over time, and
> start tuning Apache to my actual usage needs better)

I experimented with using this as a dumb alternative to inactivity-
timeout.  Set MaxRequestsPerChild to 1, means the WSGI process will
restart after each HTTP request.  Very slow, but gives immediate
memory savings.  Only viable if the MEDIA_URL is set to a separate,
static server (which webfaction recommends and makes easy).  However,
inactivity-timeout is still better.

> >> WSGIPythonOptimize 1
> > Am dubious that turning on optimisation helps and can cause problems
> > if set to '2' as doc strings will be stripped.
> I doubt it is much improvement, but I just turned it on as a test and
> did read that 2 is a bad idea which is why I stuck with 1

I'd love to see some benchmarks for this config with real apps.  The
docs make sense... don't expect magic switches to help if you haven't
optimized your code, so I really shouldn't even worry about this
option yet.

> >> 
> >> ServerName demo.digitalxero.net
> >> DocumentRoot /home/digitalxero/webapps/wsgi/demo
> >> WSGIDaemonProcess demo.digitalxero.net threads=5
> >> maximum-requests=1 inactivity-timeout=3600
> >> display-name=wsgi:demo_site
>
> > That is 1 hour for inactivity timeout. They possibly want something
> > less than that.
>
> Ya, I picked an hour for starters and will drop it down or increase it
> as my needs dictate (one I get something to monitory usage better then
> guessing)

Their default DJango installs are either mod_python or mod_wsgi.  The
mod_wsgi is embedded mode, and has no inactivity-timeout and is
StartServers 2, so by default one Django app takes 3MB + 16MB + 16MB
of memory (35MB) out of your allotted 80MB.  That means only 2.5
django apps before you run out of memory.  They have a "how to save
memory with django" howto, but it doesn't have any of these settings.
I'll try to send it updates, they'll probably appreciate it.

> > If you application does leak Python objects due to inability to break
> > reference count cycles, there shouldn't really be a need for maximum
> > 

Re: How do I minimize memory usage with WSGI and Apache?

[Jumpfroggy]

On Aug 10, 2:24 pm, dartdog  wrote:
> Big help!! Had just started getting using too much memory notices from
> Webfaction yesterday evening...

Yeah, I didn't even pay attention till I hit about 120-140MB on my
80MB plan and got everything turned off.  Otherwise I would have just
let it grow, not paying attention.

> The other item is using the crontab -e command from ssh to comment out
> apps you don't need to have running For some reason could not find
> that when I was looking for it and one of my sites just kept
> restarting after .stop, which I understand but did not know how to
> stop the behavior!

Good catch!  I noticed that they restart themselves, didn't bother
finding out why.

Here's my complete writeup and howto:
http://forum.webfaction.com/viewtopic.php?pid=11096

Thanks to those in this thread for helping me make this work.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Django dev server (runserver) takes 5 seconds after reloading new code?

[Jumpfroggy]

I have a strange problem that's popped up just recently, and it's
puzzling to me.

I'm running the "manage.py runserver" dev server for development.
Normally, right after I save a file, the server reloads the new code
and is ready to go right away.  However, recently one of my apps has a
strange problem where it'll reload the code immediately, but then the
server is unresponsive for exactly 5 seconds.

So it goes like this:
-Modify code & save
-Django dev server reloads, says "Validating models...", then "0
errors found" and is ready.
-Attempting to connect via the browser just stalls, then times out.
-5 seconds after the dev server reloads, it starts accepting
connections from the browser.

So there's just a 5 second waiting time between when the dev server
reloads, and when it will accept connections.  It only happens with
one of my apps, and has just recently started.  Does anyone know why
this might be happening?  I just wanted to ask before I start diving
through django source code.  Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Model.objects.values('some_column').distinct() returns the wrong results

[Jumpfroggy]

I have a model with a field, and I wanted to pull all the unique
values of that field.  So I do this:

distinct_values = Model.objects.values('some_column').distinct()
print 'distinct_values.count()', distinct_values.count()
print 'len(distinct_values)', len(distinct_values)

Which gives:

distinct_values.count()4
len(distinct_values)   137

This seems wrong.  If I do this:

print distinct_values

or

for value in distinct_values:
print value

I get the entire list of values from every row in the database.  It
does not print only the distinct values, as I'd assume it would.

The closest I've found is this closed ticket from a year ago:
http://code.djangoproject.com/ticket/2939
But that is ticket is different: 1) it shows the correct results when
printing the results, and the len() and count() results are swapped
from this example.

Is anyone else seeing this?  Maybe I just have my code mixed up.
Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Starting a daemon thread from Django view

[Jumpfroggy]

I'm in the same boat here.  My plan is similar.

>From views.py, record the job in the DB.
Start the daemon thread.
Return a "Processing..." response to the browser.
When the daemon thread is done, it records it in the DB.
The browser page uses AJAX to query the DB and update the status.
When the DB changes, the browser page then shows "Done." and the
results.

One unanswered question is whether updating the DB with the models is
thread safe.  I realize that Django itself is not guaranteed to be
thread safe, but I don't yet see that there would be any problems
inserting & updating models (db rows) so far.  I know I'll have to
worry about things like race conditions, but I suspect that actual
operation of updating a model would not be affected.  Hopefully that's
true and I don't run into crazy threading bugs.

Your method will be safer, since calling a web service to signal the
end of the job is concurrency, but avoids the threading problem.  It's
no different than multiple users calling the server at once.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Unable to filter a query on >1 ManyToMany relationships

[Jumpfroggy]

I have a class like this:

class Node:
name = models.CharField()
parents = models.ManyToManyField('self', symmetrical=False,
related_name='child_nodes', null=True, blank=True)

This allows a basic parent(s)/children hierarchy.  A node can have >=
0 parents.  A parent can have >= 0 children.

Say I have these nodes:

A - parent node
B - parent node
C - child of both A & B

So I can do this:

Node.objects.filter(name='C')
> returns ['C' node].

Node.objects.filter(name='C', parents__name='A')
> returns ['C' node].

Node.objects.filter(name='C', parents__name='B')
> returns ['C' node].

Node.objects.filter(Q(name='C'), Q(parents__name='A'))
> returns ['C' node].

Node.objects.filter(Q(name='C'), Q(parents__name='B'))
> returns ['C' node].

Node.objects.filter(Q(name='C'), Q(parents__name='A'),
Q(parents__name='B'))
> returns []

Node.objects.filter(Q(name='C') & Q(parents__name='A') &
Q(parents__name='B'))
> returns []

Node.objects.filter(Q(parents__name='A'), Q(parents__name='B'))
> returns []


Why do those last two return an empty set?  I can even do this:

Node.objects.filter(Q(name='C'),
Q(parents__name='A')).filter(Q(parents__name='B'))
> returns ['C' node].

That works.  But filtering on two parents (AND, not OR... I want the
child nodes of both parents) returns the empty set.  It doesn't make
sense to me why using two separate filters would work, but combining
them into one statement gives different results.

Anyone have an idea about this?  Is this a bug in the django query
system, or am I looking at this wrong?  Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What does an ideal django workflow setup look like?

[Jumpfroggy]

This is such a great thread!  I'd love to see a wiki page detailing
some good setup ideas.  It's such a leap forward from the nuts & bolts
of "How do I install django" and "How do I run a complex query" to
this... "How do I use django more effectively, end to end?"

I agree - most of these tips will be generic to project dev &
deployment, but it seems there are a few django specific ones.

My thoughts...

- Use VCS.  First step.  I use SVN, but others are probably better.
Even if you're a solo dev, even for small projects, it's worth it.

- Definitely have local, test, and prod servers.  Local is where the
development happens (on your machine).  Testing is where you can test
the latest commits.  Ideally you'd have continuous integration as
well, which would also run on the latest revision.  Prod is the live
site.  If you're a single dev on a small project, you can get away
with local & prod.

- +1 on using "settings.py" and "local_settings.py".  The problem with
keeping local settings files in the VCS is when you have multiple
servers with different settings.  You could store each file as
local_settings_test_server.py and so on.  Then you could have
something like "settings_server_name.py" that's not in VCS, and it
only has:

SERVER_NAME = 'prod'

And in your settings.py:

import * from settings_server_name

if SERVER_NAME == 'prod':
import * from settings_prod_server
elif SERVER_NAME == 'test':
...
etc

That way almost everything is in VCS, and you only have one file to
write for each server.

- Try to develop & test on servers similar to your PROD server.  It
stinks to develop and test, then realize you had python X and PROD
server is python X - 0.1, which means some features/libraries aren't
available (or syntax is wrong).

- Use "south" for database changes!  It takes just a bit to get used
to, but will make database changes (both schema and data) so much
easier to keep track of.  This is basically what Kenneth Gonsalves
described, but in reliable/convenient/automated form.  It's amazing.
Go south!  Use south!  (don't think about dmigrations.  It does not
exist.)

- Use some kind of deployment script.  I wrote my own and it's simple
and awesome.  You can also use something like fabric, which seems
complicated but is very useful.  Add things like "svn update force" to
update a repsitory, "touch *.wsgi" to reload the app, "python
manage.py migrate" to update any DB changes (using south), etc.
Anything you'd do on the server, create scripts for.  Even stuff like
"copy PROD DB to TEST server", so you have real data to play with.

That's what I can think of at the moment.  It'd be nice to have a
"best practices" page on the django site, since a lot of these
suggestions will be repeated for most users.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ManytoMany initial values

[Jumpfroggy]

Giancarlo,

I've run into this trouble myself.  The problem here is the widget
itself.  It does exactly what you described:
-Shows all possible values for the ManyToMany field.
-Highlights the "active" or "selected" values.  These correspond to
items in that list/array/queryset.

For any more than a couple of values, it's horribly confusing.  Even
though I'm used to it, I still find myself assuming that listed values
are selected (especially when you have a few hundred options).  It's
just a bad widget for this kind of data.

Take a look at the Admin control panel widgets.  Specifically, view a
User in the django control panel and look at the permissions widget.
That's what you want.  Something where it shows all available
(unselected) options on one side, all selected options on the other
side, and a way to filter the available options for the ones you
want.  Much better than the simple SELECT MULTIPLE="" that is used by
default.

I haven't looked into it, but there  may be a way to use that admin
widget for your own forms.  This really should be the default widget
anyway.

So bottom line - you need a different widget, maybe one you have to
write yourself.  Good luck!

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ManytoMany initial values

[Jumpfroggy]

On Mon, Oct 25, 2010 at 2:23 PM, Giancarlo Razzolini
 wrote:
> James,
>   First of all thank you for the fast response. I'll take a look into the 
> admin forms widgets. I'm already looking for third party applications field 
> types and widgets to see if I find one that does what I want. I was even 
> considering creating views on the underlying database to accomplish this. I 
> think that django should implement on the forms a better way to show granular 
> output. My django application is completely user dependent, and only let a 
> user to view/add/edit/remove information that belongs to him. Only the 
> superuser can deal with all objects. I was using a custom for to accomplish 
> that, but since I'm using a ManyToMany relation, it was a lot of work to 
> validate it myself and save the relationship. I guess I'll have to turn back 
> to it if I can't use the admin form you mentioned. Thank you again.
> My regards,
> Giancarlo Razzolini

No problem.  For anything user-facing, I'd recommend not using admin
forms.  You might be able to get away with it, but it seems each time
you add some business logic (permissions, validation, etc) it makes it
less and less a fit with the built-in forms.

It's relatively trivial to create your own forms, so it shouldn't be
too hard to ignore the admin pages.  It's just annoying to have to do
the work to create templates, logins, etc.

But depending on your requirements, you'll probably find that it's
easier to just create your own forms.  You'll still have to figure out
the widget issue, but at least you'll have better control over the
validation.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Unable to filter a query on >1 ManyToMany relationships

[Jumpfroggy]

Tom,

Thanks for the link, I read through that and it makes sense.

So if both conditions are in the same filter, they are both applied
simultaneously.  Since I wanted two separate conditions, I needed two
separate filter() statements.

In other words, my method, is "Find a node who's parent is both A and
B".
What I wanted, and what you recommended, is "Find a node who's parents
include A and include B".

Very interesting.  I get it now, and yes it is documented.  For some
reason I have a really hard time navigating the docs, even though I've
read the page you mentioned about a dozen times.  I think a lot of it
has to do with the horrible heading / lack of indents in the docs.

But I digress.  Thanks tom so much, this answers my question!

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

How do I filter/exclude custom queryset fields created by .extra()?

[Jumpfroggy]

I have something like this:

class Node(Model):
widgets = ForeignKey(Widget)

And I want to retrieve all nodes that have >0 widgets.  I have
an .extra() query like this:

nodes = Node.objects.all().extra(
select={
'num_widgets': 'SELECT COUNT(*) FROM appname_node_widgets
WHERE appname_node_widgets.node_id = appname_node.id',
}
)

This works fine, since I can now do this:

print nodes[0].num_widgets

But I can't filter or exclude on this field.  I try this:

nodes = nodes.exclude(num_widgets=0)

Gives a "FieldError: Cannot resolve keyword 'num_widgets' into
field."  I understand that the fields created by the .extra() call are
not "real" fields attached to the model, but they are fields attached
to the QuerySet.  Is there a way to use fields created with .extra()
in subsequent filter/exclude calls?

My next guess is to drop down to custom SQL and write the whole query
out by hand, but I thought I'd check to see if I'm doing something
wrong here.  I thought I remembered being able to use .extra() fields
in later .filter() statements, but I might be imagining that.

Any thoughts?  Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: adding field to query set

[Jumpfroggy]

What's your motive?  Are you worried about performance?

If performance is not an issue, you can just do this:

for project in Project.objects.all():
print project.user.first_name + ' ' + project.user.last_namea

This might speed things up:

projects = Project.objects.all().select_related('user')
for project in projects:
print project.user.first_name + ' ' + project.user.last_namea

If that's not fast enough, you may want to do something using .extra()
or .raw().  I'm not sure how to do this effeciently with .extra().

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: adding field to query set

[Jumpfroggy]

If it were my project, I'd probably add this kind of custom field to
your mapping:

mapping = {
# list the normal columns first... then:
'full_name': ['user__first_name', 'user__last_name'],
}

Then your render code would be something like this:

def render(self):
for row in self._data:
for name, value in self._mapping.items():
print 'name: %s' % name
if hasattr(value, '__iter__'):
final_value = []
for value2 in value:
final_value.append(getattr(row, value2))
final_value = ' '.join(final_value)
else:
final_value = getattr(row, value)
print 'final_value: %s' % final_value

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Aggregates are giving the wrong numbers for .annotate()? What am I doing wrong?

[Jumpfroggy]

Hi all,

I've been working in django for a while now, but am still wrapping my
head around the more complex queryset features.

I have something like this:

class Foo:
name = CharField()
bars = ForeignKey(Bar)
widgets = ForeignKey(Widget)

I can do this:

Foo.objects.extra(select={
'num_bars': 'SELECT COUNT(*) FROM app_bar WHERE app_bar.foo_id =
app_foo.id',
'num_widgets': 'SELECT COUNT(*) FROM app_widget WHERE
app_widget.foo_id = app_foo.id',
})

That gives me what I want... a list of all foo's with counts of how
many bars & widgets it's linked to.

NameNum BarsNum Widgets
First   1   4
Second  2   3

However, I wanted to use annotations.  So I do this:

Foo.objects.annotate(num_bars=Count(bar), num_widgets=Count(widget))

But I get increased counts:

NameNum BarsNum Widgets
First   19  19
Second  5   5

When I spit out the str(queryset.query) var, I get SQL like this:

SELECT
app_foo.name,
COUNT(app_bar.id) AS num_bars,
COUNT(app_widget.id) AS num_widgets
FROM app_foo
LEFT OUTER JOIN app_bar ON ...
LEFT OUTER JOIN app_widget ON ...
WHERE ...
GROUP BY app_food.id

This seems to fail because each LEFT OUTER JOIN gives multiple rows
for each match in the resulting table.  Then when COUNT() goes to
work, it will count each of the duplicate rows as another entry.  Then
all the COUNT() vars end up being a) larger, and b) the same as each
other, since they're all counting the end-result rows.

I've tried .distinct(), but it does not fix the problem.  Am I
misunderstaning .annotate()?  Is this a bug?  Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How do I filter/exclude custom queryset fields created by .extra()?

[Jumpfroggy]

>From a raw SQL standpoint, I've figured out that these constraints
need to be in the HAVING clause, not the WHERE clause.  But it looks
like HAVING is not exposed through the .extra() function, which would
seems logical.  It appears you can create extra fields, but due to
this lack you can't filter on them.  Is there another way to customize
the HAVING field?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Reloading a single model object from .py file... keeps getting cached results?

[Jumpfroggy]

I want to run a script outside of the django server for maintenance
purposes, but it still accesses some of the models.  The problem is
that I can't seem to get the model object to reload it's data from the
DB - it always seems cached.

# Setup django.
import project.settings
from django.core.management import setup_environ
setup_environ(project.settings)

from project import models
import time

while True:
obj = models.Object.objects.get(id=1)
print 'obj.status: %s' % obj.status
time.sleep(1)

Theoretically the line "obj = models.Object.objects.get(id=1)" should
reload a new copy of the object with id=1 each time.  But when I run
this script, the values never change - they are whatever they were
when the object was first loaded, and do not change (regardless of
what's in the DB) until the next time I run the program.

My understanding is that this is how you reload an object after
changing it (after .save(), etc).  But it's clearly not working in my
case.  Am I doing something wrong?  How do I force the object to be
reloaded from the database?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: absolute url in template and url name

[Jumpfroggy]

I always do something like this:

abs_activate_url =
request.build_absolute_uri(reverse('account_activate', args=user.id))

That's an absolute URL, so it's like "http://domain.com/account/
activate/2" instead of the normal reverse() or {% url %} which gives a
relative URL like "/account/activate/2".  Then you can use it just
like others have stated:

from django.template.loader import render_to_string
email_body = render_to_string('email_template.html', {
'abs_activate_url': abs_activate_url,
}

Jason Mayfield's "current site" method seems easier, since you don't
have to make & pass all the absolute URLs first.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: absolute url in template and url name

[Jumpfroggy]

As a followup, I had to do something similar and ended up using this:

# Gets the http://domain.com without the trailing /
base_url = request.build_absolute_uri('/')[:-1]

And in the template I can do this:

...

The benefits:
-You only have to create the base_url in the view, the rest you can do
in the template itself.
-The URLs are absolute & contain the domain (ie. http://www.domain.com/page/1,
https://www.domain.com:8008/view2, etc).
-Includes http & https.
-Simple.

This the method I'm using from now on.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

How do I rename a FileField() that's attached to a model?

[Jumpfroggy]

I've gone through the documentation a ton, been trying different
techniques, and I know I'm just missing something obvious.

I have a model like this:

class MyModel(Model):
file1 = FileField(...)

So I can do this:

model = MyModel.objects.get(id=1)
print mode.file1.name

But I want to rename the file on the filesystem.  I want something
like this:

model.file1.rename(new_filename)

But that doesn't exist.  The closest I can find is somethig like this:

contents = model.file1.read()
model.file1.delete()
model.file1.save(new_filename, contents)

But this gives an exception since the FileField save() method requires
a "contents" parameter that has a ".chunks()" method.  Right now it's
just a string.

I can't use a python tempfile.TemporaryFile() object, since that then
gives the exception "Unable to determine the file's size".

All of this seems like I'm jumping through hoops that don't make
sense.  How do I just rename the file?  Is there an easier way?

Thanks so much!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How do I rename a FileField() that's attached to a model?

[Jumpfroggy]

> you could do it manually. first use os.rename to rename the file on the disk,
> and then update your model instance with the new name and save it.

I think this is what I'm missing.  What's the code for this?  Can I
just do:

os.rename(model.file1.name, new_filename)
model.file1.name = new_filename
model.save()

Is that enough?  I'll go try that now.  The problem is that the
documentation leaves it unclear whether simply changing the name and
saving the model is kosher.  I guess I'd expect a more complicated
method, but if that works then great.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How do I rename a FileField() that's attached to a model?

[Jumpfroggy]

Yes, it was that easy.  I don't know why, but I've been banging my
head against the wall on this for a while and never thought of this
solution.

One thing that would really help a ton is more/better examples about
file-handling in django.  I've spent more hours on this single feature
reading  & rereading the docs trying to figure out how things work.  A
few good examples would have saved me tons of time.

Who do I talk to about possibly contributing to the docs?


On Mar 14, 3:10 am, Jumpfroggy <rocketmonk...@gmail.com> wrote:
> > you could do it manually. first use os.rename to rename the file on the 
> > disk,
> > and then update your model instance with the new name and save it.
>
> I think this is what I'm missing.  What's the code for this?  Can I
> just do:
>
>         os.rename(model.file1.name, new_filename)
>         model.file1.name = new_filename
>         model.save()
>
> Is that enough?  I'll go try that now.  The problem is that the
> documentation leaves it unclear whether simply changing the name and
> saving the model is kosher.  I guess I'd expect a more complicated
> method, but if that works then great.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

form.is_valid() changes a ModelForm's associated instance now?

[Jumpfroggy]

I discovered the recent changes to the "is_valid()" model form method
in 1.2, and I'd love to hear thoughts on the rationale behind this
change.

I have this kind of code:

item = Item.objects.get(id=1)
print 'item.value: %s' % item.value
form = ItemModelForm(request.POST, instance=item)
old_item = item
if form.is_valid():
print 'old_item.value: %s' % old_item.value
print "form.cleaned_data['value']: %s" %
form.cleaned_data['value']
new_item = form.save(commit=False)
print 'new_item.value: %s' % new_item.value

And I'd expect this output:

item.value: old_value
old_item.value: old_value
form.cleaned_data['value']: new_value
new_item.value: new_value

But instead I get this:

item.value: old_value
old_item.value: new_value
form.cleaned_data['value']: new_value
new_item.value: new_value

It seems like the "is_valid()" function not only checks for validation
errors and creates the "form.cleaned_data" members, it also modifies
the instance of the model attached to this form.  I just looked over
the docs, and I can now see that this is implied but not really
explicitly referenced here:

http://docs.djangoproject.com/en/1.2/topics/forms/modelforms/

The first time you call is_valid() or access the errors attribute
of a ModelForm has always triggered form validation, but as of Django
1.2, it will also trigger model validation. This has the side-effect
of cleaning the model you pass to the ModelForm constructor. For
instance, calling is_valid() on your form will convert any date fields
on your model to actual date objects.

The problem is not that the instance is being "cleaned".  It's that
the instance is being modified, then cleaned.  It would help if the
docs specified this as well, so we know what the side effects of
"is_valid()" really are.

Also, doesn't this make this code obsolete?

item = form.save()

And:

item = form.save(commit=False)
item.save()

We could instead just do this:

form = ItemModelForm(request.POST, instance=item)
if form.is_valid():
item.save()

(assuming that we're modifying an existing instance, and not creating
a new one)

Since the original "item" is modified, I could just save that as well,
right?  Is that what we truly want to happen?  Seems like the
modification of the original instance in this case is an unintentional
side effect of needing to clean the model for validation.  It seems
like it would be better if is_valid() created a copy of the instance,
and then modified & cleaned that copy instead of the original.
Otherwise it seems tricky to have a validation method secretly
modifying seemingly unrelated data.

I just want to wrap my head around these new changes, and maybe
understand the rationale behind them.  Thanks!

PS. The solution to my specific problem above is to do this:

item = Item.objects.get(id=1)
print 'item.value: %s' % item.value
form = ItemModelForm(request.POST, instance=item)
# NOTE: This fixes the problem:
old_item = copy.deepcopy(item)
if form.is_valid():
print 'old_item.value: %s' % old_item.value
print "form.cleaned_data['value']: %s" %
form.cleaned_data['value']
new_item = form.save(commit=False)
print 'new_item.value: %s' % new_item.value

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

How do I run test and get results programmatically? (django, south, fixtures, sqlite)

[Jumpfroggy]

I have an app that I'm starting to write tests for.  The app uses
south for all migrations and mysql for the db.

I wrote some tests and also created an initial_data.json fixture to
provide some default data.  If I run "python manage.py test", the
database is created via sqlite, the tests are run, and then the report
is spit out.  This works fine.

However, I want to automate this where I can programmatically run
tests, find the failures, and do some custom logging/saving of the
failures.  How do I run the tests in python?

I've tried a few variations of the below code, but the problem is the
database.  When I run the "python manage.py test" command, it
automatically uses sqlite3 as the backend, creates the database, and
uses it for all tests.  However, when I run my own tests, it uses the
mysql backend.  If I force it to use sqlite, is has problems with the
fixtures because the migrations are not run before installing the
fixtures.

How do I run tests manually using the same database setup that happens
when using the manage.py test command?

My code:

import my_application.settings

# NOTE: Use this to force the testing database to use the sqlite
engine.
my_application.settings.DATABASE_ENGINE = 'sqlite3'

from django.core.management import setup_environ
setup_environ(my_application.settings)

from django.conf import settings
from django.test.utils import get_runner

verbosity = 1
interactive = False
failfast = True
TestRunner = get_runner(settings)

test_labels = ()

test_runner = TestRunner(verbosity=verbosity,
interactive=interactive, failfast=failfast)
failures = test_runner.run_tests(test_labels)

And this is the error I get:

calling TestRunner.run_tests()
  verbosity: 1
  interactive: False
  failfast: True
  test_labels: ()
connection.settings_dict: {'ENGINE': 'django.db.backends.sqlite3',
'TEST_MIRROR': None, 'NAME': 'my_application', 'TEST_CHARSET': None,
'TIME_ZONE': 'America/New_York', 'TEST_COLLATION': None, 'OPTIONS':
{}, 'HOST': '', 'USER': 'my_application', 'TEST_NAME': None,
'PASSWORD': 'my_application', 'PORT': ''}
test_databases: {('', '', 'django.db.backends.sqlite3',
'my_application'): ['default']}
Creating test database 'default'...
Syncing...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table south_migrationhistory
Installing index for admin.LogEntry model
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
No fixtures found.

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > south

Not synced (use migrations):
 - my_application
(use ./manage.py migrate to migrate these)
test_db_name: ':memory:'
Problem installing fixture 'C:\virtualenv-my_application\lib
\site-packages\django\contrib\auth\fixtures\authtestdata.json':
Traceback (most recent call last):
  File "C:\virtualenv-my_application\lib\site-packages\django\core
\management\commands\loaddata.py", line 174, in handle
obj.save(using=using)
  File "C:\virtualenv-my_application\Lib\site-packages\django\core
\serializers\base.py", line 165, in save
models.Model.save_base(self.object, using=using, raw=True)
  File "C:\virtualenv-my_application\Lib\site-packages\django\db
\models\base.py", line 566, in save_base
created=(not record_exists), raw=raw)
  File "C:\virtualenv-my_application\Lib\site-packages\django
\dispatch\dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, **named)
  File "C:\my_application\..\my_application\models.py", line 1175,
in _create_profile_for_user
profile.save()
  File "C:\virtualenv-my_application\Lib\site-packages\django\db
\models\base.py", line 456, in save
self.save_base(using=using, force_insert=force_insert,
force_update=force_update)
  File "C:\virtualenv-my_application\Lib\site-packages\django\db
\models\base.py", line 549, in save_base
result = manager._insert(values, return_id=update_pk,
using=using)
  File "C:\virtualenv-my_application\Lib\site-packages\django\db
\models\manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
  File "C:\virtualenv-my_application\Lib\site-packages\django\db
\models\query.py", line 1518, in insert_query
return 

Re: How do I run test and get results programmatically? (django, south, fixtures, sqlite)

[Jumpfroggy]

@gladys,

While that didn't solve my original question, it did solve another
related problem.  Before, I had issue with migrations failing since
they depended on pre-existing data (which did not exist in the blank
testing database).  But if I added an initial_data.json fixture, it
gets run for every migration - during prod and testing.  This is bad.

So I went with "SOUTH_TESTS_MIGRATE = False", then I manually load the
"test_data" fixture during the first test.  Works great.

I still can't figure out why running "python manage.py test" and
running the above code have such different results.  I'd love to
figure out why, since it would be easier to use the API calls and look
in the "failures" var than it is to run the "python manage.py test"
command and parse through the output.  But for now, the latter gets
things working.

So thanks!  My original question is still open, but I'm un-stuck and
can continue working for now.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

How can I integrate performance measurements with unit tests?

[Jumpfroggy]

I have django testing setup, and it runs a few tests.  I would like to
measure the time each test takes to run, and record that in a data
file I can access later.

How do I record the time each test takes?  If this is not (easily)
possible with unit tests, then is there another easy way to test view
times?

My goal is to be able to run a performance test for each revision of
the code, and be able to track performance improvements / regressions
as development happens.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Just started Django - please help clarify web site configuration strategy

[Jumpfroggy]

> Basically, the question is - when a web
> application starts there is a number of things that needs to be
> available any time any request. Some of them are really static in
> nature - for example, pagination parameters;

If these are static values, a good place to put them is in the
settings.py file.  I put things like APP_NUMBER_POSTS_PER_PAGE = 10 in
settings.py.  As a bonus, they can be overridden with your
local_settings.py file (if you use one).  Alternately, you could have
an "app_settings.py" file where you store a bunch of static values,
and just import that file into anywhere it's needed (like your
views.py).

> some of them are more
> dynamic - for example, menu options.

If these menu options are application wide, but change often, then I'd
create models to represent them, and store them in the database.  If
these are server wide config settings that only change once in a
while, then I'd add them to settings.py (since you probably want those
hardcoded and recorded in your source-control system.  If those
settings are per-user, then they'd either be cookies (browser-based),
session vars (session-based), or a settings stored in the user's
profile (ie. stored in DB as a model, the most permanent of the 3).

> Things get more interesting when such data are read from the database.
> So far what I see is that all that stuff should be places in the
> request context (hopefully cached). It can be done more elegantly if
> custom context processor(s) are created. But we are still rely on
> request contexts. Unless I am missing something?

In django, contexts are tied to requests.  There really isn't an
"application context".  If you have a scenario that doesn't fit into
a) settings.py vars, or b) database-backed settings, give us more
details and we can figure out how best to store them.  Context
processors are convenient ways to automatically run code for each
request, but I normally just use those to automatically add certain
settings.py vars to each context (like MEDIA_URL, current user, etc).

> It seems that although request contexts combined with caching can
> help, it is way too complex for the purposes of initializing the
> application.

What's the caching for?

> Which leads me to another question - is there any callback mechanism
> when Django application starts? I guess what I am looking is a place
> to make one-time queries, store data in a static member - and viola, I
> have my application context.

There is no callback mechanism.  You can add code to any file
(models.py is a good place), and you can do some tricky python-fu to
make sure it only runs once per process, but that's a bit of a hack.
The real question is - do you really need this?  Is there a better
way?

The most helpful thing would be an example - what is a real-world
value you'd like to store in an "application context", and how would
you use it?  Then we can give you some pointers on how to do things
the django way.

As an aside, I know asp (and other web servers) run as if you have a
single process, with a single set of static vars, and you can pretend
that everything is running as a single application.  In django, it's
often better to think about things as "stateless"... ie. there's no
guarantee that there is a single, persistent app running.  There's
just your code that runs when needed.  For example, if you run apache
+ django with multiple processes, then they are separate.  Changes to
static vars in one process will not affect the same var in other
processes.  So forget all about static classes/objects/variables, as
they're probably inappropriate for most use cases.

Also, you might be over-optimizing.  If you're worried about the
performance hit of loading a set of values for each request (such as
the settings.py values above), you should profile / measure it.  The
performance hit of loading such vars would be almost completely
unmeasurable in most cases.  So you could just load up such settings
for each request instead of keeping them in static memory, which
really is an optimization (ie. trading memory space vs. CPU & disk
access).

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.