from:"Matt"

Disregard.  I found my error.  I was trying to pass something I shouldn't 
have.  This seems to be working now.

Thanks again!

On Thursday, December 5, 2013 9:10:13 AM UTC-5, Daniel Roseman wrote:
>
> On Thursday, 5 December 2013 12:23:52 UTC, Matt Lind wrote:
>>
>> Ok, I get what what you're saying, but I seem to be having a heck of a 
>> time getting the randDir passed into the instance to perform the upload.
>>
>> Here is the new code snippets:
>>
>> views.py
>> instance = Upfile( file = file)
>> instance.upload(randDir)
>>
>> models.py
>>
>> @staticmethod
>> def upload(randDir)
>>file = models.FileField( upload_to = MEDIA_ROOT+'/'+randDir)
>>
>> I am using @staticmethod as I found that otherwise I needed to pass 
>> "self" and kept getting a NameError "self" not defined.
>>
>> The error I currently get with the above code is:
>>
>> TypeError: 'file' is an invalid keyword argument for this function
>>
>> When clicking my "upload" button.
>>
>> Thanks
>>
>
>
> You can't do that: model fields need to be defined in the model itself, at 
> declaration time.
>
> But you seem to have missed the bit in the documentation[1] which 
> describes how upload_to can be a callable, which does exactly what you want.
>
> def my_upload_to(instance, filename)
> directory = "".join(random.choice(string.ascii_letters) for n in 
> xrange(12))
> return os.path.join(MEDIA_ROOT, directory, filename)
>
> class UpFile(models.Model):
> file = models.FileField(upload_to=my_upload_to)
>
>
> [1]: 
> https://docs.djangoproject.com/en/1.6/ref/models/fields/#django.db.models.FileField.upload_to
>
> --
> DR.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/04ca88b0-4d89-481a-843b-90d252c21066%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Generate a random directory for files to be uploaded

I did miss that thanks!

However, I now seem to have an issue with the def my_upload_to(instance, 
filename) line.

Apparently "self" nor "instance" works here, and so I am getting the error 
1 argument given when I need "two".

@staticmethod doesn't seem to want to work properly either.

Thanks again!

On Thursday, December 5, 2013 9:10:13 AM UTC-5, Daniel Roseman wrote:
>
> On Thursday, 5 December 2013 12:23:52 UTC, Matt Lind wrote:
>>
>> Ok, I get what what you're saying, but I seem to be having a heck of a 
>> time getting the randDir passed into the instance to perform the upload.
>>
>> Here is the new code snippets:
>>
>> views.py
>> instance = Upfile( file = file)
>> instance.upload(randDir)
>>
>> models.py
>>
>> @staticmethod
>> def upload(randDir)
>>file = models.FileField( upload_to = MEDIA_ROOT+'/'+randDir)
>>
>> I am using @staticmethod as I found that otherwise I needed to pass 
>> "self" and kept getting a NameError "self" not defined.
>>
>> The error I currently get with the above code is:
>>
>> TypeError: 'file' is an invalid keyword argument for this function
>>
>> When clicking my "upload" button.
>>
>> Thanks
>>
>
>
> You can't do that: model fields need to be defined in the model itself, at 
> declaration time.
>
> But you seem to have missed the bit in the documentation[1] which 
> describes how upload_to can be a callable, which does exactly what you want.
>
> def my_upload_to(instance, filename)
> directory = "".join(random.choice(string.ascii_letters) for n in 
> xrange(12))
> return os.path.join(MEDIA_ROOT, directory, filename)
>
> class UpFile(models.Model):
> file = models.FileField(upload_to=my_upload_to)
>
>
> [1]: 
> https://docs.djangoproject.com/en/1.6/ref/models/fields/#django.db.models.FileField.upload_to
>
> --
> DR.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/879ed008-64fd-4901-8ffc-dfa167eaaf91%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Generate a random directory for files to be uploaded

I should also state that further down in the code I make use of snippets 
like:

basename = os.path.basename( instance.file.file.name )

So by not passing and uploading the file right away it seems like those 
snippets don't work anymore

On Sunday, November 24, 2013 10:31:34 AM UTC-5, Michael Manfre wrote:
>
> There are two things that you'll need to do. Set the random directory name 
> on the UpFile instance in the view and define a function to use with 
> upload_to that uses the random directory name on the instance to define the 
> file's target path.
>
>
> https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.FileField.upload_to
>
> Regards,
> Michael Manfre
>
> On Saturday, November 23, 2013 10:33:59 AM UTC-5, Matt Lind wrote:
>>
>> So I am trying to modify django-jfu ( a multi uploader) to send files to 
>> a unique directory for every unique upload session performed.  This is just 
>> due to my application's requirements and to prevent file collisions during 
>> the operations later down the road in my app.
>>
>> Anyway, here is views.py (I think the most relevant portion anyway, I 
>> could be wrong)
>>
>> def upload( request ):
>>
>> # The assumption here is that jQuery File Upload
>> # has been configured to send files one at a time.
>> # If multiple files can be uploaded simulatenously,
>> # 'file' may be a list of files.
>> #Create the file object
>> file = upload_receive( request )
>>
>> #Create an instance of our Uploader Class and pass it the file object
>> instance = UpFile ( file = file )
>> #Save the file object
>> instance.save() 
>>
>> And the "UpFile" class in models.py:
>>
>> class UpFile(models.Model):
>> file = models.FileField( upload_to = 
>> MEDIA_ROOT+'/'+"".join([random.choice(string.ascii_letters) for n in 
>> xrange(12)]))
>>
>> The problem I am having is that it appears that the "random" section at 
>> the tail end of the "upload_to" option in models,py is processed only once 
>> when the server is started (or restarted).
>>
>> This won't work for my particular application, as I need the "top level" 
>> for any uploaded of a set of files to be unique.
>>
>> For example if I have the following "upload sessions"
>>
>> Session 1:  User only uploaded a single file.  A new random/unique 
>> directory should be generated under MEDIA_ROOT, and the file should be 
>> placed in that directory.
>> Session 2:  User uploaded 2 files.  A new random/unique directory should 
>> be generated under MEDIA_ROOT, and BOTH files should be uploaded to that 
>> directory (NOT a different directory per file)
>> Session 3:  User uploaded 100 files.  Like session 2 above, a new 
>> random/unique directory should be generated under MEDIA_ROOT and all 100 
>> files should be placed underneath.
>>
>> Later on I plan on cleaning house, but for now simply getting that logic 
>> down will get me moving again...
>>
>> Thanks for any help you can provide.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8170d555-94a3-4695-bff3-ab070276213f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Generate a random directory for files to be uploaded

Ok, I get what what you're saying, but I seem to be having a heck of a time 
getting the randDir passed into the instance to perform the upload.

Here is the new code snippets:

views.py
instance = Upfile( file = file)
instance.upload(randDir)

models.py

@staticmethod
def upload(randDir)
   file = models.FileField( upload_to = MEDIA_ROOT+'/'+randDir)

I am using @staticmethod as I found that otherwise I needed to pass "self" 
and kept getting a NameError "self" not defined.

The error I currently get with the above code is:

TypeError: 'file' is an invalid keyword argument for this function

When clicking my "upload" button.

Thanks


On Sunday, November 24, 2013 10:31:34 AM UTC-5, Michael Manfre wrote:
>
> There are two things that you'll need to do. Set the random directory name 
> on the UpFile instance in the view and define a function to use with 
> upload_to that uses the random directory name on the instance to define the 
> file's target path.
>
>
> https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.FileField.upload_to
>
> Regards,
> Michael Manfre
>
> On Saturday, November 23, 2013 10:33:59 AM UTC-5, Matt Lind wrote:
>>
>> So I am trying to modify django-jfu ( a multi uploader) to send files to 
>> a unique directory for every unique upload session performed.  This is just 
>> due to my application's requirements and to prevent file collisions during 
>> the operations later down the road in my app.
>>
>> Anyway, here is views.py (I think the most relevant portion anyway, I 
>> could be wrong)
>>
>> def upload( request ):
>>
>> # The assumption here is that jQuery File Upload
>> # has been configured to send files one at a time.
>> # If multiple files can be uploaded simulatenously,
>> # 'file' may be a list of files.
>> #Create the file object
>> file = upload_receive( request )
>>
>> #Create an instance of our Uploader Class and pass it the file object
>> instance = UpFile ( file = file )
>> #Save the file object
>> instance.save() 
>>
>> And the "UpFile" class in models.py:
>>
>> class UpFile(models.Model):
>> file = models.FileField( upload_to = 
>> MEDIA_ROOT+'/'+"".join([random.choice(string.ascii_letters) for n in 
>> xrange(12)]))
>>
>> The problem I am having is that it appears that the "random" section at 
>> the tail end of the "upload_to" option in models,py is processed only once 
>> when the server is started (or restarted).
>>
>> This won't work for my particular application, as I need the "top level" 
>> for any uploaded of a set of files to be unique.
>>
>> For example if I have the following "upload sessions"
>>
>> Session 1:  User only uploaded a single file.  A new random/unique 
>> directory should be generated under MEDIA_ROOT, and the file should be 
>> placed in that directory.
>> Session 2:  User uploaded 2 files.  A new random/unique directory should 
>> be generated under MEDIA_ROOT, and BOTH files should be uploaded to that 
>> directory (NOT a different directory per file)
>> Session 3:  User uploaded 100 files.  Like session 2 above, a new 
>> random/unique directory should be generated under MEDIA_ROOT and all 100 
>> files should be placed underneath.
>>
>> Later on I plan on cleaning house, but for now simply getting that logic 
>> down will get me moving again...
>>
>> Thanks for any help you can provide.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/dfbe4f18-8195-4242-acc7-eb505798f82f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Errors on Django Tests Tutorial 5, how to fix.

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'django',
'USER': 'matt',
'PASSWORD':'***',
'HOST': '127.0.0.1',
'PORT': '5432',
},
'OPTIONS': {
'autocommit': True,
}

I forgot to post DB settings
On Wednesday, December 4, 2013 2:32:14 PM UTC-8, Matt Schloepp wrote:
>
> Django is 1.6 and Python is 2.7.5. On Windows 8.1, no virtual-env, but my 
> actual project will be on a *nix VM. I accidentally download and install a 
> development version, I think removed it before installing 1.6 but no 
> previous django versions were installed on this system. Maybe dropping my 
> db and starting over would fix the issue?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/acdf1883-ce8e-49e5-b48c-94edf6117620%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Errors on Django Tests Tutorial 5, how to fix.

Django is 1.6 and Python is 2.7.5. On Windows 8.1, no virtual-env, but my 
actual project will be on a *nix VM. I accidentally download and install a 
development version, I think removed it before installing 1.6 but no 
previous django versions were installed on this system. Maybe dropping my 
db and starting over would fix the issue?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2272299c-3432-495a-b31f-28b70613ee13%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Errors on Django Tests Tutorial 5, how to fix.

Commenting out my Postgresql DB and putting back the default options for 
the sqlite DB fixed the issue. 

Is there a tests.py variable I could use to create an If..Else that would 
allow this to be automatic and not force me to edit my settings.py every 
time I wanted to run a test.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/16971b55-ed41-4086-ba37-4248b13646b7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Errors on Django Tests Tutorial 5, how to fix.

I'm running Django on Windows with PostgreSQL database. I'm in tutorial 5 
and have copy/pasted the code after it's initial failure. I've been 
following the tutorials to this point, with only a little side 
experimentation, and everything has been working. It looks like it DB 
related since I'm not using SQLite. I'm lost. What do I need to add or 
modify to get the tests to work?

Output:

> ==
>
> ERROR: test_was_published_recently_with_future_poll 
>> (polls.tests.PollMethodTests)
>
> --
>
> Traceback (most recent call last):
>
>   File "C:\python27\lib\site-packages\django\test\testcases.py", line 178, 
>> in __call__
>
> self._pre_setup()
>
>   File "C:\python27\lib\site-packages\django\test\testcases.py", line 749, 
>> in _pre_setup
>
> self._fixture_setup()
>
>   File "C:\python27\lib\site-packages\django\test\testcases.py", line 861, 
>> in _fixture_setup
>
> if not connections_support_transactions():
>
>   File "C:\python27\lib\site-packages\django\test\testcases.py", line 848, 
>> in connections_support_transactions
>
> for conn in connections.all())
>
>   File "C:\python27\lib\site-packages\django\test\testcases.py", line 848, 
>> in 
>
> for conn in connections.all())
>
>   File "C:\python27\lib\site-packages\django\utils\functional.py", line 
>> 49, in __get__
>
> res = instance.__dict__[self.func.__name__] = self.func(instance)
>
>   File "C:\python27\lib\site-packages\django\db\backends\__init__.py", 
>> line 664, in supports_transactions
>
> self.connection.leave_transaction_management()
>
>   File "C:\python27\lib\site-packages\django\db\backends\__init__.py", 
>> line 317, in leave_transaction_management
>
> if managed == self.get_autocommit():
>
>   File "C:\python27\lib\site-packages\django\db\backends\__init__.py", 
>> line 324, in get_autocommit
>
> self.ensure_connection()
>
>   File "C:\python27\lib\site-packages\django\db\backends\__init__.py", 
>> line 124, in ensure_connection
>
> self.connect()
>
>   File "C:\python27\lib\site-packages\django\db\utils.py", line 86, in 
>> __exit__
>
> db_exc_type = getattr(self.wrapper.Database, dj_exc_type.__name__)
>
> AttributeError: 'DatabaseWrapper' object has no attribute 'Database'
>
>
>> --
>
> Ran 0 tests in 0.001s
>
>
>> FAILED (errors=1)
>
>
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b0067a22-b212-4ffa-ad5d-7fab897f9ca7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Generate a random directory for files to be uploaded

2013-11-23 Thread Matt Lind

So I am trying to modify django-jfu ( a multi uploader) to send files to a 
unique directory for every unique upload session performed.  This is just 
due to my application's requirements and to prevent file collisions during 
the operations later down the road in my app.

Anyway, here is views.py (I think the most relevant portion anyway, I could 
be wrong)

def upload( request ):

# The assumption here is that jQuery File Upload
# has been configured to send files one at a time.
# If multiple files can be uploaded simulatenously,
# 'file' may be a list of files.
#Create the file object
file = upload_receive( request )

#Create an instance of our Uploader Class and pass it the file object
instance = UpFile ( file = file )
#Save the file object
instance.save() 

And the "UpFile" class in models.py:

class UpFile(models.Model):
file = models.FileField( upload_to = 
MEDIA_ROOT+'/'+"".join([random.choice(string.ascii_letters) for n in 
xrange(12)]))

The problem I am having is that it appears that the "random" section at the 
tail end of the "upload_to" option in models,py is processed only once when 
the server is started (or restarted).

This won't work for my particular application, as I need the "top level" 
for any uploaded of a set of files to be unique.

For example if I have the following "upload sessions"

Session 1:  User only uploaded a single file.  A new random/unique 
directory should be generated under MEDIA_ROOT, and the file should be 
placed in that directory.
Session 2:  User uploaded 2 files.  A new random/unique directory should be 
generated under MEDIA_ROOT, and BOTH files should be uploaded to that 
directory (NOT a different directory per file)
Session 3:  User uploaded 100 files.  Like session 2 above, a new 
random/unique directory should be generated under MEDIA_ROOT and all 100 
files should be placed underneath.

Later on I plan on cleaning house, but for now simply getting that logic 
down will get me moving again...

Thanks for any help you can provide.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/49b570f3-20d0-46f0-bffe-dff7ad14cdd4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Apache won't allow upload but dev server will

2013-08-09 Thread Matt Lind

I am attempting to setup django-bft on my CentOS 6.4 server.

When running under the dev server I am able to upload the file and it shows 
up on my server.

However, when running through MOD_WSGI everything about the site works, 
except for the upload.  The file never lands on the server and the 
application throws an error:

Invalid JSON: Environment: Request Method: POST Request URL: 
http://172.23.77.10/uploader/process/?X-Progress-ID=69e91467a474e2ae156004bcae3d8c75
 
Django Version: 1.3.7 Python Version: 2.6.6 Installed Applications: 
['django.contrib.auth', 'django.contrib.contenttypes', 
'django.contrib.sessions', 'django.contrib.sites', 
'django.contrib.messages', 'django.contrib.staticfiles', 
'django.contrib.admin', 'bft'] Installed Middleware: 
('django.middleware.common.CommonMiddleware', 
'django.contrib.sessions.middleware.SessionMiddleware', 
'django.middleware.csrf.CsrfViewMiddleware', 
'django.contrib.auth.middleware.AuthenticationMiddleware', 
'django.contrib.messages.middleware.MessageMiddleware') Traceback: File 
"/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in 
get_response 111. response = callback(request, *callback_args, 
**callback_kwargs) File 
"/usr/lib/python2.6/site-packages/django/db/transaction.py" in inner 217. 
res = func(*args, **kwargs) File "/bft/views.py" in 
process_submission 262. upload_html(request, form.instance) File "/bft/views.py" in upload_html 426. upload.save() File "/bft/models.py" in save 91. super(File, self).save(*args, 
**kwargs) File "/usr/lib/python2.6/site-packages/django/db/models/base.py" 
in save 460. self.save_base(using=using, force_insert=force_insert, 
force_update=force_update) File 
"/usr/lib/python2.6/site-packages/django/db/models/base.py" in save_base 
543. for f in meta.local_fields if not isinstance(f, AutoField)] File 
"/usr/lib/python2.6/site-packages/django/db/models/fields/files.py" in 
pre_save 255. file.save(file.name, file, save=False) File 
"/usr/lib/python2.6/site-packages/django/db/models/fields/files.py" in save 
92. self.name = self.storage.save(name, content) File 
"/usr/lib/python2.6/site-packages/django/core/files/storage.py" in save 48. 
name = self.get_available_name(name) File 
"/usr/lib/python2.6/site-packages/django/core/files/storage.py" in 
get_available_name 74. while self.exists(name): File 
"/usr/lib/python2.6/site-packages/django/core/files/storage.py" in exists 
218. return os.path.exists(self.path(name)) File 
"/usr/lib/python2.6/site-packages/django/core/files/storage.py" in path 
234. raise SuspiciousOperation("Attempted access to '%s' denied." % name) 
Exception Type: SuspiciousOperation at /uploader/process/ Exception Value: 
Attempted access to 'media/temp/09-08-2013/apache_test.txt' denied.

The permissions on the file system are 775.  User is my personal user, 
group is apache.

Directive in httpd.conf for the directory is:

Alias /media/ ""


">
Options Indexes
AllowOverride None
Order deny,allow
Allow from all
 

When I browse to http:///media  I am able to see the index of 
the directory.

I have attempted to set the sticky bit for the group.  I have attempted to 
set the user owner as apache.  I have attempted to set permissions on the 
whole Django site to 777.  I have temporarily disabled SELinux.  I even 
temporarily gave the apache user a login, Su'd to the apache user.  That 
user can read and write to the directory in question.

I have watched the video here:  
http://blip.tv/pycon-australia/getting-started-with-apache-mod_wsgi-3859481

Anywhere else I can check?

TIA

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Avoid unbounded memory consumption when running `manage.py test`

2013-07-26 Thread Matt McClure

On Thursday, July 18, 2013 7:30:55 PM UTC-7, Matt McClure wrote:

> 1.  TestSuite. Here's a minimal hack to release the reference that 
> TestSuite would otherwise hold after a TestCase runs until the remainder of 
> the suite had finished.
>
> +self._tests.remove(test)
>

That turns out to be a bad idea because it modifies a list inside a loop 
iterating over it, making the loop skip items it should visit. Here's an 
updated change that doesn't suffer from that problem:

diff --git a/django/utils/unittest/suite.py b/django/utils/unittest/suite.py
index f39569b..8530200 100644
--- a/django/utils/unittest/suite.py
+++ b/django/utils/unittest/suite.py
@@ -1,5 +1,6 @@
 """TestSuite"""
 
+import gc
 import sys
 import unittest
 from django.utils.unittest import case, util
@@ -96,7 +97,11 @@ class TestSuite(BaseTestSuite):
 
 # private methods
 def _wrapped_run(self, result, debug=False):
-for test in self:
+while True:
+try:
+test = self._tests.pop(0)
+except IndexError:
+break
 if result.shouldStop:
 break
 
@@ -116,6 +121,7 @@ class TestSuite(BaseTestSuite):
 test(result)
 else:
 test.debug()
+#gc.collect()
 
 def _handleClassSetUp(self, test, result):
 previousClass = getattr(result, '_previousTestClass', None)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Avoid unbounded memory consumption when running `manage.py test`

2013-07-18 Thread Matt McClure

On Thursday, July 18, 2013 6:12:16 PM UTC-4, Matt McClure wrote:

> AFAICT, TestResult holds references to each TestCase instance.


I can start to answer my own question. But I'm interested to know if others 
run into similar problems and how you solve them.

It looks like there are at least two participants that hold TestCase 
references longer than desirable.

1.  TestSuite. Here's a minimal hack to release the reference that 
TestSuite would otherwise hold after a TestCase runs until the remainder of 
the suite had finished.

diff --git a/django/utils/unittest/suite.py b/django/utils/unittest/suite.py
index f39569b..109a476 100644
--- a/django/utils/unittest/suite.py
+++ b/django/utils/unittest/suite.py
@@ -1,5 +1,6 @@
 """TestSuite"""
 
+import gc
 import sys
 import unittest
 from django.utils.unittest import case, util
@@ -116,6 +117,8 @@ class TestSuite(BaseTestSuite):
 test(result)
 else:
 test.debug()
+self._tests.remove(test)
+#gc.collect()
 
 def _handleClassSetUp(self, test, result):
 previousClass = getattr(result, '_previousTestClass', None)

I believe gc.collect() in the above is unnecessary. Calling it explicitly 
accelerates the collection, but the automatic collector will free the 
TestCase from memory soon enough as long as we can manage its lifetime by 
eliminating all the references to it.

2. TestResult and subclasses may hold references to TestCase instances via 
addSuccess() and friends. For example, 
django.utils.unittest.result.TestResult.addError does:

self.errors.append((test, self._exc_info_to_string(err, test)))

In my current scenario, I'm using unittest-xml-reporting's TestResult 
subclass, and the following patch eliminates the references to TestCase 
instances that it would otherwise save.

--- __init__.py~ 2013-05-02 16:45:19.0 -0400
+++ __init__.py 2013-07-18 21:39:13.0 -0400
@@ -8,6 +8,9 @@
 import os
 import sys
 import time
+
+import psutil
+
 from unittest import TestResult, _TextTestResult, TextTestRunner
 
 try:
@@ -27,13 +30,24 @@
 self.delegate = delegate
 
 def write(self, text):
-self._captured.write(text)
+#self._captured.write(text)
 self.delegate.write(text)
 
 def __getattr__(self, attr):
 return getattr(self._captured, attr)
 
 
+def testcase_name(test_method):
+testcase = type(test_method)
+
+# Ignore module name if it is '__main__'
+module = testcase.__module__ + '.'
+if module == '__main__.':
+module = ''
+result = module + testcase.__name__
+return result
+
+
 class _TestInfo(object):
 """This class keeps useful information about the execution of a
 test method.
@@ -44,11 +58,21 @@
 
 def __init__(self, test_result, test_method, outcome=SUCCESS, 
err=None):
 self.test_result = test_result
-self.test_method = test_method
 self.outcome = outcome
 self.elapsed_time = 0
 self.err = err
 
+#self.test_method = test_method
+self.test_description = 
self.test_result.getDescription(test_method)
+self.test_exception_info = (
+'' if not self.err
+else self.test_result._exc_info_to_string(
+self.err, test_method)
+)
+
+self.test_name = testcase_name(test_method)
+self.test_id = test_method.id()
+
 def test_finished(self):
 """Save info that can only be calculated once a test has run.
 """
@@ -58,16 +82,18 @@
 def get_description(self):
 """Return a text representation of the test method.
 """
-return self.test_result.getDescription(self.test_method)
+#return self.test_result.getDescription(self.test_method)
+return self.test_description
 
 def get_error_info(self):
 """Return a text representation of an exception thrown by a test
 method.
 """
-if not self.err:
-return ''
-return self.test_result._exc_info_to_string(self.err, \
-self.test_method)
+#if not self.err:
+#return ''
+#return self.test_result._exc_info_to_string(self.err, \
+#self.test_method)
+return self.test_exception_info
 
 
 class _XMLTestResult(_TextTestResult):
@@ -81,6 +107,8 @@
 self.successes = []
 self.callback = None
 self.elapsed_times = elapsed_times
+self.start_meminfo = psutil.Process(os.getpid()).get_memory_info()
+self.stop_meminfo = psutil.Process(os.getpid()).get_memory_info()
 
 def _prepare_callback(self, test_info, target_list, verbose_str,
 short_str):
@@ -103,6 +131,10 @@
 if self.showAll:
 self.stream.writeln('%s (%.3fs)' % \

Avoid unbounded memory consumption when running `manage.py test`

2013-07-18 Thread Matt McClure

What's a good way to avoid unbounded memory consumption when running 
manage.py test?

AFAICT, TestResult holds references to each TestCase instance. So 
attributes of my TestCase subclasses don't get freed by the garbage 
collector until the last reference to TestResult disappears, which isn't 
until the entire test run ends. The most obvious solution to me is to put 
tearDown methods in all my TestCase subclasses that delete the references 
to their own attributes. But it seems like there has to be a more general, 
automatic solution.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Having an issue with Minimal File Upload utility

2013-06-26 Thread Matt Lind

Just wanted to update everyone on the solve for this.

It turns out that I was overzealous in my pre-planning.

In my urls.py at the top of my project I had defined extra applications 
that I had not fleshed out yet.  So they didn't have any urls.py or 
views.py defined.  

Even though the error was pointing at my specific code, after commenting 
out the applications I hadn't fleshed out yet, everything is working great.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Having an issue with Minimal File Upload utility

2013-06-22 Thread Matt Lind

Updating with full error:

Request Method:GETRequest URL:http:///uploader/list/Django 
Version:1.3.7Exception Type:TemplateSyntaxErrorException Value:

Caught ImportError while rendering: No module named urls

Exception Location:/usr/lib/python2.6/site-packages/django/utils/importlib.py 
in import_module, line 35Python Executable:/usr/bin/pythonPython Version:
2.6.6Python Path:

['/home//myproject',
 '/usr/lib64/python26.zip',
 '/usr/lib64/python2.6',
 '/usr/lib64/python2.6/plat-linux2',
 '/usr/lib64/python2.6/lib-tk',
 '/usr/lib64/python2.6/lib-old',
 '/usr/lib64/python2.6/lib-dynload',
 '/usr/lib64/python2.6/site-packages',
 '/usr/lib/python2.6/site-packages']

Server time:Sat, 22 Jun 2013 09:06:18 -0500

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Having an issue with Minimal File Upload utility

2013-06-19 Thread Matt Lind

My Settings.py:

# Django settings for EqlAnalystToolSet project.


import os, ldap
from django_auth_ldap.config import LDAPSearch, ActiveDirectoryGroupType

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
# ('Your Name', 'your_em...@example.com'),
)

MANAGERS = ADMINS

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 
'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '',  # Or path to database 
file if using sqlite3.
'USER': '',  # Not used with sqlite3.
'PASSWORD': '',  # Not used with sqlite3.
'HOST': '',  # Set to empty string for 
localhost. Not used with sqlite3.
'PORT': '',  # Set to empty string for default. 
Not used with sqlite3.
}
}

# AUTH_LDAP settings added by MJL

AUTHENTICATION_BACKENDS = (
'django_auth_ldap.backend.LDAPBackend',
)

AUTH_LDAP_SERVER_URI = 'ldap://'

AUTH_LDAP_BIND_DN = 
"CN=,CN=Users,DC=,DC=,DC=com"
AUTH_LDAP_BIND_PASSWORD = ''
AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=,dc=,dc=com", 
ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)",)
AUTH_LDAP_GROUP_SEARCH = LDAPSearch("dc=,dc=,dc=com", 
ldap.SCOPE_SUBTREE, '(objectClass=group)',)
AUTH_LDAP_GROUP_TYPE = ActiveDirectoryGroupType()
AUTH_LDAP_REQUIRE_GROUP = 
'CN=,CN=Users,DC=,DC=,DC=com'

AUTH_LDAP_USER_ATTR_MAP = {
 "first name": "givenName",
 "last name":  "sn",
 "email":  "mail",
}

# SITE Variables adding by MJL

LOGIN_URL = '/login/'
SITE_ROOT = os.path.realpath(os.path.dirname(__file__))

# Hosts/domain names that are valid for this site; required if DEBUG is 
False
# See https://docs.djangoproject.com/en/1.3/ref/settings/#allowed-hosts
ALLOWED_HOSTS = []

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Chicago'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True

# Absolute filesystem path to the directory that will hold user-uploaded 
files.
# Example: "/home/media/media.lawrence.com/media/"
MEDIA_ROOT = os.path.join(SITE_ROOT, 'media')

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://media.lawrence.com/media/;, "http://example.com/media/;
MEDIA_URL = '/media/'

# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/home/media/media.lawrence.com/static/"
STATIC_ROOT = os.path.join(SITE_ROOT, 'static')

# URL prefix for static files.
# Example: "http://media.lawrence.com/static/;
STATIC_URL = '/static/'

# URL prefix for admin static files -- CSS, JavaScript and images.
# Make sure to use a trailing slash.
# Examples: "http://foo.com/static/admin/;, "/static/admin/".
ADMIN_MEDIA_PREFIX = '/static/admin/'

# Additional locations of static files
STATICFILES_DIRS = (
# Put strings here, like "/home/html/static" or "C:/www/django/static".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
#'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

# Make this unique, and don't share it with anybody.
SECRET_KEY = ')2)@8!m!6c3)a0u%%4($hj6srh^xio$e4dyr6+)'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)

ROOT_URLCONF = 'myproject.urls'

TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or 
"C:/www/django/templates".
# Always use

Having an issue with Minimal File Upload utility

2013-06-19 Thread Matt Lind

I am trying to get this utility to work within my own Django site:

https://github.com/doph/minimal-django-file-upload-example/tree/master/src/for_django_1-3/myproject

I have copied all the appropriate files to where they need to be.  I am on 
django 1.3.7 on Centos 6.4 64Bit.

When I run the list page I get the error:

Caught ImportError while rendering: No module named urls


When I copy the actual project and run it, it works.  When I attempt to add the 
functionality to my site, I get the above error.


Just wondering why a file copy would produce these different results.


Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Difficulties displaying correct value in select widget with ModelForm/formset_factory:

2013-05-10 Thread Matt Conrad

Answering my own question from a few days ago, this:

choice_vals = scene.begin_scene.all().values()
ChoiceFormFactory = formset_factory(ChoiceForm)
formset = ChoiceFormFactory(initial=choice_vals)

is all wrong. I needed to be using a modelformset_factory, not a
formset_factory, with a few extra params:

ChoiceFormSetFactory = modelformset_factory(Choice, form=ChoiceForm,
extra=0)
formset = ChoiceFormSetFactory(queryset=scene.begin_scene.all())

This assigns the value of the select widget correctly out of the box, and
also avoids some later issues I ran into with .values() and initial=.

Matt







On Mon, May 6, 2013 at 4:28 PM, Matt C <matt...@gmail.com> wrote:

> I'm resorting to code in __init__ to set my select dropdown properly when
> using a ModelForm with formset_factory, and wondering if there is a better
> way.
>
> I have a model that looks like this:
>
> class Choice(models.Model):
> desc = models.TextField()
> begin_scene = models.ForeignKey(Scene, related_name='begin_scene')
> end_scene = models.ForeignKey(Scene, related_name='end_scene')
>
> with a corresponding ModelForm.
>
> My view contains this:
>
> choice_vals = scene.begin_scene.all().values()
> ChoiceFormFactory = formset_factory(ChoiceForm)
> formset = ChoiceFormFactory(initial=choice_vals)
>
>
> When I render the form, I get a select dropdown which contains correct FK
> values from Scene. However, the correct option in the dropdown is not
> selected.
>
> I did a bunch of searching and playing around and ended up with this in my
> form class, which does work to set the selected option correctly.
>
> class ChoiceForm(ModelForm):
> def __init__(self, *args, **kwargs):
> if 'initial' in kwargs and 'end_scene_id' in kwargs['initial']:
> kwargs['initial']['end_scene'] =
> kwargs['initial']['end_scene_id']
> super(ChoiceForm, self).__init__(*args, **kwargs)
>
> But I'm wondering if I missed something simpler/cleaner/more idiomatic.
> Did I?
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Difficulties displaying correct value in select widget with ModelForm/formset_factory:

2013-05-06 Thread Matt C

I'm resorting to code in __init__ to set my select dropdown properly when 
using a ModelForm with formset_factory, and wondering if there is a better 
way.

I have a model that looks like this:

class Choice(models.Model):
desc = models.TextField()
begin_scene = models.ForeignKey(Scene, related_name='begin_scene')
end_scene = models.ForeignKey(Scene, related_name='end_scene')

with a corresponding ModelForm.

My view contains this:

choice_vals = scene.begin_scene.all().values()
ChoiceFormFactory = formset_factory(ChoiceForm)
formset = ChoiceFormFactory(initial=choice_vals)


When I render the form, I get a select dropdown which contains correct FK 
values from Scene. However, the correct option in the dropdown is not 
selected.

I did a bunch of searching and playing around and ended up with this in my 
form class, which does work to set the selected option correctly.

class ChoiceForm(ModelForm):
def __init__(self, *args, **kwargs):
if 'initial' in kwargs and 'end_scene_id' in kwargs['initial']:
kwargs['initial']['end_scene'] = 
kwargs['initial']['end_scene_id']
super(ChoiceForm, self).__init__(*args, **kwargs)

But I'm wondering if I missed something simpler/cleaner/more idiomatic. Did 
I?


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: performance of delete

2013-04-24 Thread Matt Long

I'm curious how long manually running a single delete query with the 
appropriate where clause would take. Surely there's a way to do that through 
the Django ORM without having to drop into raw SQL?

That being said, there are limits to how long a given query can be, so the ORM 
would have to do some intelligent batching of your IN clause, so maybe it does 
make sense to leave that hairy logic out of the ORM :-) 

-- 
Matt Long
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Wednesday, April 24, 2013 at 18:31, Larry Martell wrote:

> I changed it to delete one row at a time and it's taking 3 minutes.
> 
> On Wed, Apr 24, 2013 at 6:38 PM, Setiaman Lee <setiaman@gmail.com> wrote:
> > Larry,
> > 
> > 
> > Avoid to use IN clause. I'd say to insert the events you want to delete into
> > the temp table. You can put the session id or user id in temp table to
> > differentiate set of data from other users. Then do the delete by joining
> > the temp table to the Event_Message_ldx table. your sql might look like
> > this:
> > 
> > delete event_message_ldx
> > from temp
> > where event_message_ldx.event_id = temp.event_id
> > and temp.userid = %s
> > 
> > above SQL can be executed in Django using RAW SQL. See the link below.
> > https://docs.djangoproject.com/en/dev/topics/db/sql/
> > 
> > Test the SQL in the MySQL workspace when it works then move the code to the
> > Django.
> > 
> > Rgrds,
> > Setiaman Lee
> > 
> > 
> > 
> > 
> > 
> > On Thu, Apr 25, 2013 at 8:12 AM, Larry Martell <larry.mart...@gmail.com>
> > wrote:
> > > 
> > > I have a table that has 2.5 million rows and 9 columns that are all
> > > int except for 2 varchar(255) - i.e. not that big of a table. I am
> > > executing this statement:
> > > 
> > > Event_Message_Idx.objects.filter(event_id__in=event_ids).delete()
> > > 
> > > Where event_ids has around 1,500 items in it. There is an index on
> > > event_id. This statement is taking 1 hour and 5 minutes to run. There
> > > is nothing else hitting the database at that time, and the machine
> > > it's running on is 97% idle and has plenty of free memory. This seems
> > > extremely excessive to me. I would guess it's because of the in
> > > clause. Perhaps this is more of a MySQL question when a django one,
> > > but is there some better way to do a delete like this in django?
> > > 
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Django users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to django-users+unsubscr...@googlegroups.com.
> > > To post to this group, send email to django-users@googlegroups.com.
> > > Visit this group at http://groups.google.com/group/django-users?hl=en.
> > > For more options, visit https://groups.google.com/groups/opt_out.
> > > 
> > 
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to django-users+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-users@googlegroups.com.
> > Visit this group at http://groups.google.com/group/django-users?hl=en.
> > For more options, visit https://groups.google.com/groups/opt_out.
> > 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
> 
> 


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Help interpreting profiler results (or: why is my app so slow?)

2013-04-18 Thread Matt Andrews

I wouldn't say huge (see the page/view in question at 
http://www.scenepointblank.com) -- I grab a bunch of things and then 
display them, mostly truncated (with HTML stripped out, usually for the 
homepage). The custom template tags take a bunch of arguments (say for a 
news post, it takes arguments about where to show the photo, how large to 
make the headline, etc) so there's lots of loading of views inside loops 
for each news story etc.

I've tried experimenting before with the template fragment caching and saw 
little in the way of results.


On Thursday, 18 April 2013 13:40:10 UTC+1, Shawn Milochik wrote:
>
> Hit send too soon:
>
>
> https://docs.djangoproject.com/en/dev/topics/cache/#template-fragment-caching
>
>
> On Thu, Apr 18, 2013 at 8:39 AM, Shawn Milochik 
> <sh...@milochik.com
> > wrote:
>
>> Yes, it does look like template tags are taking some time. Is the page 
>> huge? Are you doing a ton of formatting? Is there something you could maybe 
>> move to server-side?
>>
>> Also, this might help with caching bits of your output: 
>>
>>
>> On Thu, Apr 18, 2013 at 6:17 AM, Matt Andrews 
>> <ma...@mattandrews.info
>> > wrote:
>>
>>>
>>> On Thursday, 18 April 2013 10:45:40 UTC+1, Tom Evans wrote:
>>>
>>> On Wed, Apr 17, 2013 at 11:18 PM, Matt Andrews <ma...@mattandrews.info> 
>>>> wrote: 
>>>> > Hi all. 
>>>> > 
>>>> > Having performance problems with my Django app. I've posted here 
>>>> before 
>>>> > talking about this: one theory for my slowness woes was that I'm 
>>>> using raw 
>>>> > SQL for everything after getting sick of Django doing things weirdly 
>>>> > (duplicating queries, adding bizarre things like "LIMIT 3453453" to 
>>>> queries, 
>>>> > not being able to JOIN things like I wanted etc). I'm not opposed to 
>>>> going 
>>>> > back to the ORM but need to know if this is where my bottleneck is. 
>>>> > 
>>>> > I've run a profiler against my code and the results are here: 
>>>> > http://pastebin.com/raw.php?i=**HQf9bqGp<http://pastebin.com/raw.php?i=HQf9bqGp>
>>>> >  
>>>> > 
>>>> > On my local machine (a not very powerful laptop) I see Django Debug 
>>>> Toolbar 
>>>> > load times of ~1900ms for my site homepage. This includes 168ms of db 
>>>> calls 
>>>> > (11 queries, which I think are fairly well-tuned, indexed, etc). I 
>>>> cache 
>>>> > pretty well on production but load times are still slow -- some of 
>>>> this may 
>>>> > be down to my cheap webhost, though. In my settings I enabled 
>>>> > django.template.loaders.**cached.Loader but this doesn't seem to 
>>>> make much 
>>>> > difference. 
>>>> > 
>>>> > I'm having trouble seeing what the profiler results above are telling 
>>>> me: 
>>>> > can anyone shed any light? 
>>>>
>>>> Most of your time is spent in pprint, which was called over 14,000 
>>>> times to generate your page. Over 2 seconds spent printing out debug. 
>>>> This should be telling you "don't use pprint when you want to see how 
>>>> fast your code is". 
>>>>
>>>> Cheers 
>>>>
>>>> Tom 
>>>>
>>>
>>> Good point Tom, apologies. Here's the profiler results with DebugToolbar 
>>> switched off (and ordered by cumulative time, thanks Shawn!): 
>>> http://pastebin.com/raw.php?i=y3iP0cLn
>>>
>>> The top one is obviously my "home" method inside my views, but I'm 
>>> struggling to get more from it than that. Lots of template rendering, but 
>>> caching not helping here...?
>>>  
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-users...@googlegroups.com .
>>> To post to this group, send email to django...@googlegroups.com
>>> .
>>> Visit this group at http://groups.google.com/group/django-users?hl=en.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>  
>>>  
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Help interpreting profiler results (or: why is my app so slow?)

2013-04-18 Thread Matt Andrews


On Thursday, 18 April 2013 10:45:40 UTC+1, Tom Evans wrote:

> On Wed, Apr 17, 2013 at 11:18 PM, Matt Andrews 
> <ma...@mattandrews.info> 
> wrote: 
> > Hi all. 
> > 
> > Having performance problems with my Django app. I've posted here before 
> > talking about this: one theory for my slowness woes was that I'm using 
> raw 
> > SQL for everything after getting sick of Django doing things weirdly 
> > (duplicating queries, adding bizarre things like "LIMIT 3453453" to 
> queries, 
> > not being able to JOIN things like I wanted etc). I'm not opposed to 
> going 
> > back to the ORM but need to know if this is where my bottleneck is. 
> > 
> > I've run a profiler against my code and the results are here: 
> > http://pastebin.com/raw.php?i=HQf9bqGp 
> > 
> > On my local machine (a not very powerful laptop) I see Django Debug 
> Toolbar 
> > load times of ~1900ms for my site homepage. This includes 168ms of db 
> calls 
> > (11 queries, which I think are fairly well-tuned, indexed, etc). I cache 
> > pretty well on production but load times are still slow -- some of this 
> may 
> > be down to my cheap webhost, though. In my settings I enabled 
> > django.template.loaders.cached.Loader but this doesn't seem to make much 
> > difference. 
> > 
> > I'm having trouble seeing what the profiler results above are telling 
> me: 
> > can anyone shed any light? 
>
> Most of your time is spent in pprint, which was called over 14,000 
> times to generate your page. Over 2 seconds spent printing out debug. 
> This should be telling you "don't use pprint when you want to see how 
> fast your code is". 
>
> Cheers 
>
> Tom 
>

Good point Tom, apologies. Here's the profiler results with DebugToolbar 
switched off (and ordered by cumulative time, thanks Shawn!): 
http://pastebin.com/raw.php?i=y3iP0cLn

The top one is obviously my "home" method inside my views, but I'm 
struggling to get more from it than that. Lots of template rendering, but 
caching not helping here...?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Help interpreting profiler results (or: why is my app so slow?)

2013-04-17 Thread Matt Andrews

Hi all.

Having performance problems with my Django app. I've posted here before 
talking about this: one theory for my slowness woes was that I'm using raw 
SQL for everything after getting sick of Django doing things weirdly 
(duplicating queries, adding bizarre things like "LIMIT 3453453" to 
queries, not being able to JOIN things like I wanted etc). I'm not opposed 
to going back to the ORM but need to know if this is where my bottleneck is.

I've run a profiler against my code and the results are here: 
http://pastebin.com/raw.php?i=HQf9bqGp

On my local machine (a not very powerful laptop) I see Django Debug Toolbar 
load times of ~1900ms for my site homepage. This includes 168ms of db calls 
(11 queries, which I think are fairly well-tuned, indexed, etc). I cache 
pretty well on production but load times are still slow -- some of this may 
be down to my cheap webhost, though. In my settings I 
enabled django.template.loaders.cached.Loader but this doesn't seem to make 
much difference.

I'm having trouble seeing what the profiler results above are telling me: 
can anyone shed any light?

If it helps at all, this is the homepage of my app: 
http://www.scenepointblank.com -- there's lot of HTML processing (stripping 
tags or truncating strings and parsing HTML etc). I use a couple of 
template tags quite frequently to re-use partial views (a news post, for 
example) so I can just pass it an object to work with. I don't know if this 
means a huge template overhead. I've explored replacing Django's templates 
with Jinja2 but fell at the first hurdle when none of the aforementioned 
template tags worked and I couldn't figure out how to port them to Jinja.

Any tips would be hugely appreciated.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: m2m symmetry confusion

2013-04-11 Thread Matt Schinckel

On Friday, April 12, 2013 9:52:10 AM UTC+9:30, Lachlan Musicman wrote:
>
> On 12 April 2013 08:43, Dennis Lee Bieber <wlf...@ix.netcom.com> 
> wrote: 
> > On Thu, 11 Apr 2013 16:41:37 +1000, Lachlan Musicman 
> > <dat...@gmail.com> 
>
> > declaimed the following in gmane.comp.python.django.user: 
> > 
> > An individual typically only has two parents (unless you are 
> > considering birth and adoption, which may be needed to handle some of 
> > the odd situations forming with the various "same sex marriage" models), 
> > so many-to-many isn't really appropriate. And an individual could have 
> > many children. 
> > 
> > So I'd have a pair of links for 
> > 
> > father 
> > mother 
> > 
> > and that is IT... Children and Siblings are all query results (children 
> > of X are those with the appropriate parent slot = X). 
>
>
> I like your reasoning, but I disagree on your result. First, it 
> doesn't relate partners, second, it presumes two parents (I am a step 
> father). 
>

But a partner relationship is not at all related to a parent-child 
relationship.

That is, you may have children with a person who is not your partner. And,
you may have a partner and not have children.

And, you may have more than one partner (depending upon where you are,
you may have more than one partner simultaneously).

If you will only store adoptive OR birth parents, then you can get away with
a single mother and father fields. But, even for adoptive relationships, you
probably need a start date (and possibly a finish date), so I think they 
should
be stored seperately.

Thus, you have:

Person
 - date-of-birth
 - date-of-death
 - mother (fk-> Person)
 - father (fk -> Person)
 - adoptive_parent (m2m -> Person, maybe use a through table
to get start and finish dates).

The issue you have then is that you have two fks to Parent,
which means you would have to query differently to get a father's
children to a mother's. Or you could wrap '.children()' to get the
value.

> So I think I'm going to go with a 
>
> parent m2m 
> partner m2m 
>
> and take your advise to trash sibling and children. 
>
> Back to the grindstone then. 
>
>
Or, you could use the parent m2m with a through table to add
the relevant data. birth/adoptive parent, etc.

And, the partner m2m could also use a through table to store
start and finish times of the relationship.

Family relationships are hard...

Matt.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django problem:AttributeError:Manage isn't accessible via Link instances

2013-03-21 Thread Matt Woodward

On Thursday, March 21, 2013 12:49:45 AM UTC-7, simon xue wrote:
>
> When I read to page 31,"To get an object by ID,type the following:
> >>>Link.object.get(id=1)


Unless this is just a typo that should be objects plural:
Link.objects.get(id=1)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Test database not auto-flushing

2013-03-01 Thread Matt

 

I have a bunch of unit test files, all of which consist of 
django.test.TestCase classes.

Wrote myself a little shell script to uncomment/comment test file imports 
in my __init__.py file, so I can run tests from certain test files, based 
off the command line arguments I give it. I am also able to run all the 
tests of all the test files in one go (for regression testing purposes).

I have this one test file that has some JSON fixtures and the first test 
checks that a certain model/table has 3 records in it (defined by the JSON 
fixture).

So here is the problem: when I run this test file on its own its tests pass 
with flying colours, but when I run the test with all other tests, that 
particular test case I mentioned, fails.

When I run all the tests, the database says there are 6 records in the 
table/model, but there should only be 3 (from the fixture), like when the 
test file is run by itself.

I also tried running the that test file with a few other test files (not 
all) and it passes. So the only time it doesn't, is when all the test files 
are run.

To me this seems like a bug in Django or PostgreSQL (DB I am using), 
because aren't Django TestCases supposed to auto-flush/reset the database 
between each test method, let alone test class?

Posted on stackoverflow (
http://stackoverflow.com/questions/15124828/django-test-database-not-auto-flushing),
 
but haven't got a response yet.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Using template fragment caching inside a sitewide cache: possible?

2013-02-15 Thread Matt Andrews

I tried using the approach above (template fragment cache everywhere 
instead of sitewide) and the same problem was there: the inner cache didn't 
reflect cachebusting changes.

I ended up rewriting the code to pull the view fragments via AJAX (which 
did respect the cachebusting). You can see the results 
here: http://www.scenepointblank.com/ (it's the "popular right now" tabbed 
widget near the bottom right.

On Thursday, 14 February 2013 17:19:28 UTC, Matt Andrews wrote:
>
> Hi Tom,
>
> Yep, you've got the problem right. Reading it back that way I see the 
> issue much more clearly...
>
> One approach I thought of: what if, instead of the sitewide caching, I 
> used template-fragment caching for *every* view (there aren't that many) 
> and invalidated the relevant ones along with the nested caches when needed 
> (there are probably only five or so views where this would need to happen)?
>
> I'm on shared hosting so I don't think Varnish is going to be a 
> possibility...
>
> thanks!
>
> On Thursday, 14 February 2013 16:02:53 UTC, Tom Evans wrote:
>>
>> On Mon, Feb 11, 2013 at 1:42 PM, Matt Andrews <ma...@mattandrews.info> 
>> wrote: 
>> > Hi all, 
>> > 
>> > I've been experimenting with an expensive query I need to call 
>> (essentially 
>> > grabbing data from some Google APIs). I tried this experiment: 
>> > 
>> > A sitewide cache with a long (days) expiry time 
>> > A template fragment with its own separate cache inside a view cached by 
>> the 
>> > sitewide cache -- this fragment simply displays the current time. 
>> > A view function which clears the named template fragment cache. 
>> > 
>> > The behaviour I expected was that on first pageload, the fragment would 
>> > display the time of page render, and all subsequent reloads would 
>> display 
>> > the same time. This was true. 
>> > 
>> > The second part, though, was that I expected to be able to call the 
>> view 
>> > function to clear the fragment cache and then reload the page to see 
>> the 
>> > time update. This didn't happen. 
>> > 
>> > Is it possible to achieve this behaviour? Essentially I want to run a 
>> > background cron task which just hits the Google API and updates my 
>> cached 
>> > fragments - the Google data changes every 15 minutes but the sitewide 
>> cache 
>> > has several hours timeout, normally. 
>> > 
>> > Hope this makes sense. 
>> > 
>> > Matt 
>> > 
>>
>> Hi Matt 
>>
>> Can I restate the problem to see if I am getting it right? 
>>
>> You have a page fragment, 'FRAG', stored in the cache. 
>> You have a page, 'PAGE', using that fragment stored in the cache. 
>> You update 'FRAG' in the cache. 
>> You reload 'PAGE', and it has the old contents of 'FRAG' instead of 
>> the updated one. 
>>
>> If so, this is to be expected. The site cache caches entire responses, 
>> with a key derived from that request. When a new request comes in, a 
>> key is generated from that request. If the key is found in the cache, 
>> the cached response is returned. 
>>
>> At no point does the cache know that one cache entry is built using 
>> another, or that invalidating the fragment should invalidate any page 
>> containing that fragment. 
>>
>> One potential solution is to use something like Varnish, an HTTP 
>> accelerator/cache that has support for ESI - Edge Side Includes - that 
>> allow you to build up responses from multiple components, and 
>> intelligently cache the components. Eg, an example of what you are 
>> trying to do in Varnish using ESI: 
>>
>> https://www.varnish-cache.org/trac/wiki/ESIfeatures#Anesi:includeexample 
>>
>> Alternatively, I may have completely misunderstood this :) 
>>
>> Cheers 
>>
>> Tom 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Using template fragment caching inside a sitewide cache: possible?

2013-02-14 Thread Matt Andrews

Hi Tom,

Yep, you've got the problem right. Reading it back that way I see the issue 
much more clearly...

One approach I thought of: what if, instead of the sitewide caching, I used 
template-fragment caching for *every* view (there aren't that many) and 
invalidated the relevant ones along with the nested caches when needed 
(there are probably only five or so views where this would need to happen)?

I'm on shared hosting so I don't think Varnish is going to be a 
possibility...

thanks!

On Thursday, 14 February 2013 16:02:53 UTC, Tom Evans wrote:
>
> On Mon, Feb 11, 2013 at 1:42 PM, Matt Andrews 
> <ma...@mattandrews.info> 
> wrote: 
> > Hi all, 
> > 
> > I've been experimenting with an expensive query I need to call 
> (essentially 
> > grabbing data from some Google APIs). I tried this experiment: 
> > 
> > A sitewide cache with a long (days) expiry time 
> > A template fragment with its own separate cache inside a view cached by 
> the 
> > sitewide cache -- this fragment simply displays the current time. 
> > A view function which clears the named template fragment cache. 
> > 
> > The behaviour I expected was that on first pageload, the fragment would 
> > display the time of page render, and all subsequent reloads would 
> display 
> > the same time. This was true. 
> > 
> > The second part, though, was that I expected to be able to call the view 
> > function to clear the fragment cache and then reload the page to see the 
> > time update. This didn't happen. 
> > 
> > Is it possible to achieve this behaviour? Essentially I want to run a 
> > background cron task which just hits the Google API and updates my 
> cached 
> > fragments - the Google data changes every 15 minutes but the sitewide 
> cache 
> > has several hours timeout, normally. 
> > 
> > Hope this makes sense. 
> > 
> > Matt 
> > 
>
> Hi Matt 
>
> Can I restate the problem to see if I am getting it right? 
>
> You have a page fragment, 'FRAG', stored in the cache. 
> You have a page, 'PAGE', using that fragment stored in the cache. 
> You update 'FRAG' in the cache. 
> You reload 'PAGE', and it has the old contents of 'FRAG' instead of 
> the updated one. 
>
> If so, this is to be expected. The site cache caches entire responses, 
> with a key derived from that request. When a new request comes in, a 
> key is generated from that request. If the key is found in the cache, 
> the cached response is returned. 
>
> At no point does the cache know that one cache entry is built using 
> another, or that invalidating the fragment should invalidate any page 
> containing that fragment. 
>
> One potential solution is to use something like Varnish, an HTTP 
> accelerator/cache that has support for ESI - Edge Side Includes - that 
> allow you to build up responses from multiple components, and 
> intelligently cache the components. Eg, an example of what you are 
> trying to do in Varnish using ESI: 
>
> https://www.varnish-cache.org/trac/wiki/ESIfeatures#Anesi:includeexample 
>
> Alternatively, I may have completely misunderstood this :) 
>
> Cheers 
>
> Tom 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Using template fragment caching inside a sitewide cache: possible?

2013-02-14 Thread Matt Andrews

Apologies for bumping this thread, but is there anybody with any insight on 
this? Really driving me crazy!

On Monday, 11 February 2013 13:42:05 UTC, Matt Andrews wrote:
>
> Hi all,
>
> I've been experimenting with an expensive query I need to call 
> (essentially grabbing data from some Google APIs). I tried this experiment:
>
>- A sitewide cache with a long (days) expiry time
>- A template fragment with its own separate cache *inside* a view 
>cached by the sitewide cache -- this fragment simply displays the current 
>time.
>- A view function which clears the named template fragment cache.
>
> The behaviour I expected was that on first pageload, the fragment would 
> display the time of page render, and all subsequent reloads would display 
> the same time. This was true.
>
> The second part, though, was that I expected to be able to call the view 
> function to clear the fragment cache and then reload the page to see the 
> time update. This didn't happen.
>
> Is it possible to achieve this behaviour? Essentially I want to run a 
> background cron task which just hits the Google API and updates my cached 
> fragments - the Google data changes every 15 minutes but the sitewide cache 
> has several hours timeout, normally.
>
> Hope this makes sense.
>
> Matt
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Using template fragment caching inside a sitewide cache: possible?

2013-02-11 Thread Matt Andrews

Hi all,

I've been experimenting with an expensive query I need to call (essentially 
grabbing data from some Google APIs). I tried this experiment:

   - A sitewide cache with a long (days) expiry time
   - A template fragment with its own separate cache *inside* a view cached 
   by the sitewide cache -- this fragment simply displays the current time.
   - A view function which clears the named template fragment cache.

The behaviour I expected was that on first pageload, the fragment would 
display the time of page render, and all subsequent reloads would display 
the same time. This was true.

The second part, though, was that I expected to be able to call the view 
function to clear the fragment cache and then reload the page to see the 
time update. This didn't happen.

Is it possible to achieve this behaviour? Essentially I want to run a 
background cron task which just hits the Google API and updates my cached 
fragments - the Google data changes every 15 minutes but the sitewide cache 
has several hours timeout, normally.

Hope this makes sense.

Matt

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Busting template fragment caches inside a sitewide cache: possible?

2013-02-10 Thread Matt Andrews

Hi all.

I have a site which makes use of Google's Analytics API which allows me to
grab the most-viewed content on my site and then show it in various ways
around the site ("most popular articles" etc). The query against this API
is slow (takes a couple of seconds) so I've been caching it as described in
my StackOverflow question about the technique
here<http://stackoverflow.com/questions/13323150/advice-needed-on-django-caching-strategy>
,

The basic process is this:

- A cron job hits a URL endpoint every 15 minutes
- This URL makes the API request and caches the results (for 1 hour, in
case the cron ever fails)
- Meanwhile, my app's views request this data and get served the cached
data.
- In my views, I wrap all usage of this data in named template fragment
caches
- The cron endpoint clears both the high-level cache (which the views
call) and the template fragment caches
- I use a site-wide cache for the entire app.

This kind of works, but occasionally results in blank spaces where the data
should be -- running the cron job manually has no effect on this.
Occasionally, experimenting with restarting the server, clearing the app's
cache and immediately calling the cron endpoints can fix things. After an
hour or so things tend to work themselves out but it seems that the 15
minute updates don't actually reflect in the templates.

The StackOverflow advice I got seemed to suggest that using template
fragment caching *inside* a sitewide cached view should respect the inner
cache setting, but it doesn't seem to.

Can anyone suggest a strategy for making this work reliably? Every time I
deploy a new build of the app (and restart the server) it wipes the cache
and pinging the update URL does nothing. I'm sure my logic is just a little
wrong.

thanks
Matt

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Struggling with slow startup / SQL in new app (using raw SQL)

Yes, local app, local DB (which is an exact copy of the production DB). 
Interestingly, I just tried pointing the local app at the remote database 
and total query time (for 10 queries) went from 115ms to 3477ms! Really odd.

On Tuesday, January 22, 2013 7:54:53 PM UTC, Nikolas Stevenson-Molnar wrote:
>
>  When running locally, are you also using a local database instance? IIRC 
> the time that phpMyAdmin reports is the time taken for the database to 
> execute the query, whereas the time reported by the debug toolbar (I *
> think*) is complete round-trip (send query, execute, return results). So 
> if there is latency between your application and your database, that could 
> account for the discrepancy.
>
> The debug toolbar *does* add overhead to your request (about doubles the 
> page load time), but the numbers it reports should be more or less accurate 
> (iow a page might take 1s to load with the toolbar, but the toolbar will 
> report 500ms, which is accurate for load-time *without* the toolbar).
>
> _Nik
>
> On 1/22/2013 9:26 AM, Matt Andrews wrote:
>  
> Hi Nik, 
>
>  I see the discrepancy both locally and when deployed, too -- that's 
> what's so puzzling. If it helps, I'm using MySQL and running it on Windows 
> locally and under Passenger on my remote Linux server (also on MySQL).
>
>  Only other thing I can think of is that this "overhead" might literally 
> be the extra work Django Debug Toolbar does to show queries etc. Obviously 
> when I turn the toolbar off then I can't see (directly) the SQL times 
> Django is getting, but could be a factor? Either way, with the toolbar off 
> on production, there's definitely a slight "lag" before page requests start 
> to kick in -- maybe this is unrelated to the SQL, though.
>
>
> On Tuesday, 22 January 2013 16:42:19 UTC, Nikolas Stevenson-Molnar wrote: 
>>
>> Hi Matt, 
>>
>>  It's unlikely the problem lies with dictfetchall. The "small 
>> performance hit" mentioned by the docs probably refers to the fact that the 
>> function has a time complexity of (number of colulmns) * (number of rows). 
>> Since you're only getting 10 rows back, I can't see that having an 
>> appreciable impact.
>>
>>  I haven't seen this kind of variation between performing a direct 
>> query, and executing the query as part of a Django request… especially with 
>> such a basic query. Do you see the same difference in timing locally or is 
>> it only when deployed?
>>
>>  _Nik
>>
>>  On Jan 22, 2013, at 4:47 AM, Matt Andrews <ma...@mattandrews.info> 
>> wrote:
>>
>> That looks really useful - for future reference, great! Sadly my webhost 
>> is still bundling Django 1.2.3 :( 
>>
>>  In general, though, these issues still come down to the fact that a SQL 
>> query as executed by Django takes several times longer than going directly 
>> to the DB. Is this down to my use of dictfetchall() or something more low 
>> level?
>>
>>
>> On Tuesday, January 22, 2013 12:09:50 PM UTC, Jani Tiainen wrote: 
>>>
>>> Hi, 
>>>
>>> I see that you had quite a bunch of m2m keys. 
>>>
>>> Have you tried recent version of Django (1.4+) with prefetch_related() 
>>> [1] ? 
>>>
>>>
>>> [1] 
>>> <
>>> https://docs.djangoproject.com/en/1.4/ref/models/querysets/#prefetch-related>
>>>  
>>>
>>>
>>>
>>> 22.1.2013 13:57, Matt Andrews kirjoitti: 
>>> > Hi Jani, 
>>> > 
>>> > I made a StackOverflow post last year with an example of the ORM stuff 
>>> I 
>>> > tried and the poor queries it produced: 
>>> > 
>>> http://stackoverflow.com/questions/5843457/django-objects-all-making-360-queries-how-can-i-optimise-this-manytomany
>>>  
>>> > 
>>> > There's also this discussion about how using the same queryset in two 
>>> > places in the template caused Django to request its data twice: 
>>> > 
>>> http://stackoverflow.com/questions/9447053/best-way-to-slice-a-django-queryset-without-hitting-the-database-more-than-once(this
>>>  
>>> > was easily fixed, but again, not very intuitive and frustrated me) 
>>> > 
>>> > I don't have the data to hand currently but I also remember seeing 
>>> weird 
>>> > things happen where queries would end with stuff like "... LIMIT 
>>> > 234423445" (or some crazy number which I'd never entered and was 
>>> orders 
>>> > of magnitude bigger than the number of rows in the table).

Re: Struggling with slow startup / SQL in new app (using raw SQL)

Hi Nik,

I see the discrepancy both locally and when deployed, too -- that's what's 
so puzzling. If it helps, I'm using MySQL and running it on Windows locally 
and under Passenger on my remote Linux server (also on MySQL).

Only other thing I can think of is that this "overhead" might literally be 
the extra work Django Debug Toolbar does to show queries etc. Obviously 
when I turn the toolbar off then I can't see (directly) the SQL times 
Django is getting, but could be a factor? Either way, with the toolbar off 
on production, there's definitely a slight "lag" before page requests start 
to kick in -- maybe this is unrelated to the SQL, though.


On Tuesday, 22 January 2013 16:42:19 UTC, Nikolas Stevenson-Molnar wrote:
>
> Hi Matt,
>
> It's unlikely the problem lies with dictfetchall. The "small performance 
> hit" mentioned by the docs probably refers to the fact that the function 
> has a time complexity of (number of colulmns) * (number of rows). Since 
> you're only getting 10 rows back, I can't see that having an appreciable 
> impact.
>
> I haven't seen this kind of variation between performing a direct query, 
> and executing the query as part of a Django request… especially with such a 
> basic query. Do you see the same difference in timing locally or is it only 
> when deployed?
>
> _Nik
>
> On Jan 22, 2013, at 4:47 AM, Matt Andrews 
> <ma...@mattandrews.info> 
> wrote:
>
> That looks really useful - for future reference, great! Sadly my webhost 
> is still bundling Django 1.2.3 :(
>
> In general, though, these issues still come down to the fact that a SQL 
> query as executed by Django takes several times longer than going directly 
> to the DB. Is this down to my use of dictfetchall() or something more low 
> level?
>
>
> On Tuesday, January 22, 2013 12:09:50 PM UTC, Jani Tiainen wrote:
>>
>> Hi, 
>>
>> I see that you had quite a bunch of m2m keys. 
>>
>> Have you tried recent version of Django (1.4+) with prefetch_related() 
>> [1] ? 
>>
>>
>> [1] 
>> <
>> https://docs.djangoproject.com/en/1.4/ref/models/querysets/#prefetch-related>
>>  
>>
>>
>>
>> 22.1.2013 13:57, Matt Andrews kirjoitti: 
>> > Hi Jani, 
>> > 
>> > I made a StackOverflow post last year with an example of the ORM stuff 
>> I 
>> > tried and the poor queries it produced: 
>> > 
>> http://stackoverflow.com/questions/5843457/django-objects-all-making-360-queries-how-can-i-optimise-this-manytomany
>>  
>> > 
>> > There's also this discussion about how using the same queryset in two 
>> > places in the template caused Django to request its data twice: 
>> > 
>> http://stackoverflow.com/questions/9447053/best-way-to-slice-a-django-queryset-without-hitting-the-database-more-than-once(this
>>  
>> > was easily fixed, but again, not very intuitive and frustrated me) 
>> > 
>> > I don't have the data to hand currently but I also remember seeing 
>> weird 
>> > things happen where queries would end with stuff like "... LIMIT 
>> > 234423445" (or some crazy number which I'd never entered and was orders 
>> > of magnitude bigger than the number of rows in the table). 
>> > 
>> > I'm aware these are probably edge cases that are down to my own novice 
>> > status, but even using things like select_related(), it wasn't doing 
>> > what I wanted. I just felt it easier to use my existing SQL (I'm 
>> > converting a PHP app over to Django) and I'm not concerned about 
>> > database portability (switching to postgres or whatever). 
>> > 
>> > Nik: just realised I missed your final question. For the SQL posted 
>> > above, the numbers are approximately: 12,000 rows in the `news` table, 
>> > maybe 10 `news_category` rows, about 100 `writers` and around 3000 
>> > `images`. All properly indexed and with sensible column types. 
>> > 
>> > On Tuesday, January 22, 2013 10:53:40 AM UTC, Jani Tiainen wrote: 
>> > 
>> > Hi, 
>> > 
>> >   From your raw SQL I saw you're doing few joins. So I suppose you 
>> do 
>> > quite a few foreign key fetches. 
>> > 
>> > You didn't mention anything how you originally tried to solve case 
>> with 
>> > ORM. Could you please publish what you had when things were slow? 
>> > 
>> > 22.1.2013 12:26, Matt Andrews kirjoitti: 
>> >  > Hi Nik, 
>> >  > 
>> >  > Thanks - I do feel like by circumventing the ORM I've just 
>>

Re: Struggling with slow startup / SQL in new app (using raw SQL)

That looks really useful - for future reference, great! Sadly my webhost is 
still bundling Django 1.2.3 :(

In general, though, these issues still come down to the fact that a SQL 
query as executed by Django takes several times longer than going directly 
to the DB. Is this down to my use of dictfetchall() or something more low 
level?


On Tuesday, January 22, 2013 12:09:50 PM UTC, Jani Tiainen wrote:
>
> Hi, 
>
> I see that you had quite a bunch of m2m keys. 
>
> Have you tried recent version of Django (1.4+) with prefetch_related() [1] 
> ? 
>
>
> [1] 
> <
> https://docs.djangoproject.com/en/1.4/ref/models/querysets/#prefetch-related> 
>
>
>
> 22.1.2013 13:57, Matt Andrews kirjoitti: 
> > Hi Jani, 
> > 
> > I made a StackOverflow post last year with an example of the ORM stuff I 
> > tried and the poor queries it produced: 
> > 
> http://stackoverflow.com/questions/5843457/django-objects-all-making-360-queries-how-can-i-optimise-this-manytomany
>  
> > 
> > There's also this discussion about how using the same queryset in two 
> > places in the template caused Django to request its data twice: 
> > 
> http://stackoverflow.com/questions/9447053/best-way-to-slice-a-django-queryset-without-hitting-the-database-more-than-once(this
>  
> > was easily fixed, but again, not very intuitive and frustrated me) 
> > 
> > I don't have the data to hand currently but I also remember seeing weird 
> > things happen where queries would end with stuff like "... LIMIT 
> > 234423445" (or some crazy number which I'd never entered and was orders 
> > of magnitude bigger than the number of rows in the table). 
> > 
> > I'm aware these are probably edge cases that are down to my own novice 
> > status, but even using things like select_related(), it wasn't doing 
> > what I wanted. I just felt it easier to use my existing SQL (I'm 
> > converting a PHP app over to Django) and I'm not concerned about 
> > database portability (switching to postgres or whatever). 
> > 
> > Nik: just realised I missed your final question. For the SQL posted 
> > above, the numbers are approximately: 12,000 rows in the `news` table, 
> > maybe 10 `news_category` rows, about 100 `writers` and around 3000 
> > `images`. All properly indexed and with sensible column types. 
> > 
> > On Tuesday, January 22, 2013 10:53:40 AM UTC, Jani Tiainen wrote: 
> > 
> > Hi, 
> > 
> >   From your raw SQL I saw you're doing few joins. So I suppose you 
> do 
> > quite a few foreign key fetches. 
> > 
> > You didn't mention anything how you originally tried to solve case 
> with 
> > ORM. Could you please publish what you had when things were slow? 
> > 
> > 22.1.2013 12:26, Matt Andrews kirjoitti: 
> >  > Hi Nik, 
> >  > 
> >  > Thanks - I do feel like by circumventing the ORM I've just "given 
> > up" 
> >  > and perhaps I'll reconsider -- none of my queries are 
> particularly 
> >  > "specialist" (as the sample above indicates) - I just found 
> Django 
> >  > generating odd things. 
> >  > 
> >  > To answer your questions: 
> >  > 
> >  > 1. Yes, reloading the page sees the same time for the queries (it 
> > just 
> >  > feels as though the entire process takes a second or two to 
> > start, which 
> >  > is perhaps not related to SQL itself). 
> >  > 
> >  > 2. I believe so, yes (it's shared hosting...). 
> >  > 
> >  > If you're curious, you can see a sample of the app at 
> >  > http://beta.scenepointblank.com <http://beta.scenepointblank.com> 
>
> > (obviously you won't see the SQL, but 
> >  > the "delay" between pages, even though these pages are all cached 
> > for 
> >  > 2hrs+, is partly my concern here). 
> >  > 
> >  > On Tuesday, January 22, 2013 4:24:09 AM UTC, Nikolas 
> > Stevenson-Molnar wrote: 
> >  > 
> >  > Hi Matt, 
> >  > 
> >  > Firstly, I encourage you to have another crack a the ORM. It 
> can 
> >  > certainly seem a bit aggravating at times if you're coming 
> > from a 
> >  > SQL mindset, but really pays off down the road in terms of 
> >  > maintainability and readability. Typically you should only 
> > need raw 
> >  > queries in Django in cases where you have super-specialized

Re: Struggling with slow startup / SQL in new app (using raw SQL)

Hi Jani,

I made a StackOverflow post last year with an example of the ORM stuff I 
tried and the poor queries it produced: 
http://stackoverflow.com/questions/5843457/django-objects-all-making-360-queries-how-can-i-optimise-this-manytomany

There's also this discussion about how using the same queryset in two 
places in the template caused Django to request its data twice: 
http://stackoverflow.com/questions/9447053/best-way-to-slice-a-django-queryset-without-hitting-the-database-more-than-once
 (this 
was easily fixed, but again, not very intuitive and frustrated me)

I don't have the data to hand currently but I also remember seeing weird 
things happen where queries would end with stuff like "... LIMIT 234423445" 
(or some crazy number which I'd never entered and was orders of magnitude 
bigger than the number of rows in the table).

I'm aware these are probably edge cases that are down to my own novice 
status, but even using things like select_related(), it wasn't doing what I 
wanted. I just felt it easier to use my existing SQL (I'm converting a PHP 
app over to Django) and I'm not concerned about database portability 
(switching to postgres or whatever). 

Nik: just realised I missed your final question. For the SQL posted above, 
the numbers are approximately: 12,000 rows in the `news` table, maybe 10 
`news_category` rows, about 100 `writers` and around 3000 `images`. All 
properly indexed and with sensible column types. 

On Tuesday, January 22, 2013 10:53:40 AM UTC, Jani Tiainen wrote:
>
> Hi, 
>
>  From your raw SQL I saw you're doing few joins. So I suppose you do 
> quite a few foreign key fetches. 
>
> You didn't mention anything how you originally tried to solve case with 
> ORM. Could you please publish what you had when things were slow? 
>
> 22.1.2013 12:26, Matt Andrews kirjoitti: 
> > Hi Nik, 
> > 
> > Thanks - I do feel like by circumventing the ORM I've just "given up" 
> > and perhaps I'll reconsider -- none of my queries are particularly 
> > "specialist" (as the sample above indicates) - I just found Django 
> > generating odd things. 
> > 
> > To answer your questions: 
> > 
> > 1. Yes, reloading the page sees the same time for the queries (it just 
> > feels as though the entire process takes a second or two to start, which 
> > is perhaps not related to SQL itself). 
> > 
> > 2. I believe so, yes (it's shared hosting...). 
> > 
> > If you're curious, you can see a sample of the app at 
> > http://beta.scenepointblank.com (obviously you won't see the SQL, but 
> > the "delay" between pages, even though these pages are all cached for 
> > 2hrs+, is partly my concern here). 
> > 
> > On Tuesday, January 22, 2013 4:24:09 AM UTC, Nikolas Stevenson-Molnar 
> wrote: 
> > 
> > Hi Matt, 
> > 
> > Firstly, I encourage you to have another crack a the ORM. It can 
> > certainly seem a bit aggravating at times if you're coming from a 
> > SQL mindset, but really pays off down the road in terms of 
> > maintainability and readability. Typically you should only need raw 
> > queries in Django in cases where you have super-specialized (that 
> > uses views or non-standard functions) queries or need some specific 
> > optimization. If there's really no way to perform many of your 
> > "day-to-day" queries with the ORM then that's an indication that a 
> > different database design may fit your data model better. I 
> > understand that you may have a unique situation, but I just wanted 
> > to throw that out there as I personally find the ORM to be a huge 
> > time saver. 
> > 
> > Now, with that out of the way... a couple of considerations: 1) you 
> > say it's a slow "startup"; if you refresh the page do the queries 
> > run just as slow the second time around? and 2) are your Django app 
> > and phpMyAdmin running on the same machine? If not, could transit 
> > time be an issue? 
> > 
> > Finally, can you give an idea about the size of the tables in 
> > question? How many rows in each? 
> > 
> > _Nik 
> > 
> > On 1/21/2013 3:25 PM, Matt Andrews wrote: 
> >> Hi all, 
> >> 
> >> Fairly new to Django. I ended up pulling out all of the 
> >> ORM-generated queries and writing my own SQL directly (I got fed 
> >> up trying to work out how to achieve the kind of things I needed 
> >> without Django adding in extra joins or unintended WHERE clauses 
> >> etc). All my app's SQL uses cursor.execute() and the 
> >> dictfetchall(

Re: Struggling with slow startup / SQL in new app (using raw SQL)