Re: Newbie: ImportError during syncdb

[Robin Lery]

What the name of the app folder? I think its just django_bookmarks.


On Thu, Mar 6, 2014 at 6:43 AM, David  wrote:

> Dear django users,
>
> I am new to django and have an ImportError: No module named bookmarks
> problem.
>
> It seems that as I tried to syncdb django does not recognize the
> bookmarks module I have created before. In the past I was able to
> syncdb, hence my confusion.
>
> I am trying to teach myself by following "Learning Website Development
> with Django" by Ayman Hourieh.
>
> I am not working in a virtualenv.
>
> Thanks for your guidance and hints!
>
> David
>
>
> settings.py:
>
> INSTALLED_APPS = (
> 'django.contrib.admin',
> 'django.contrib.auth',
> 'django.contrib.contenttypes',
> 'django.contrib.sessions',
> 'django.contrib.messages',
> 'django.contrib.staticfiles',
> 'django_bookmarks.bookmarks',
> )
>
> david@ubuntu:~/Tools/Python/Code/myCode/Django/django_bookmarks$ python
> manage.py syncdb
> Traceback (most recent call last):
>   File "manage.py", line 10, in 
> execute_from_command_line(sys.argv)
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py",
> line 399, in execute_from_command_line
> utility.execute()
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py",
> line 392, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py",
> line 242, in run_from_argv
> self.execute(*args, **options.__dict__)
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py",
> line 280, in execute
> translation.activate('en-us')
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/utils/translation/__init__.py",
> line 130, in activate
> return _trans.activate(language)
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
> line 188, in activate
> _active.value = translation(language)
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
> line 177, in translation
> default_translation = _fetch(settings.LANGUAGE_CODE)
>   File
>
> "/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
> line 159, in _fetch
> app = import_module(appname)
>   File
> "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line
> 40, in import_module
> __import__(name)
> ImportError: No module named bookmarks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/5317CBCC.4060908%40gmx.net.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2B4-nGorOGF10TOCuYAWohS%2B3nHjrL-DRpkj%2BDGTtmHDq4u00A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Improving django test startup performance.

[Andrew Farrell]

Hmm. That is a good point. If there is little difference, then I suppose
I'll blame buildout.

$ bin/django print_time
2014-03-05T22:07:33.051634
before test
2014-03-05T22:07:40.202247
command has been run

That is A difference of 7 seconds, significantly better.

$ strace bin/django print_time 2> cmdtrace
cat cmdtrace | wc -l
54361
$ cat cmdtrace | grep 'open(' | wc -l
34233
$ cat cmdtrace | grep 'stat64(' | wc -l
13625
$ cat cmdtrace | grep 'read(' | wc -l
2706
$ cat cmdtrace | grep 'fstat64(' | wc -l
2377
$ cat cmdtrace | grep 'close(' | wc -l
1563
$ cat cmdtrace | grep 'ENOENT (No such file or directory)' | wc -l
41561

There are still almost as many files being sought for and not opened
however. I suppose that means that failing to find a file on disc is
actually not that costly.
The difference between running a command and running a test also suggests
that either:
1) Linode disc performance varies throughout the day and is better outside
buisiness hours? Nope.
2014-03-05T22:20:33.604757
before test
nosetests --verbosity 1 match_mentors -anow -s
To reuse old database "test_02_06_18_29" for speed, set env var REUSE_DB=1.
Creating test database for alias 'default'...
2014-03-05T22:21:03.106680
test has begun

2) A significant portion of the time is spent in the other imports that I'm
doing in that test file. I should have broken it out and ran that test.
I'll do that now.
2014-03-05T22:23:05.650883
before test
nosetests --verbosity 1 match_mentors -anow -s
To reuse old database "test_02_06_18_29" for speed, set env var REUSE_DB=1.
Creating test database for alias 'default'...
2014-03-05T22:23:42.127545
test has begun
That is still 37 seconds, so it is almost certainly something in the test
discovery process.
I suppose this means that digging up and mapping out the piping that runs
through nose ,
django-nose,
the django test-discovery code, and unittest.py is in order. I suppose that
will build character.

-- 
*Andrew Farrell*
Software Engineer *|* MassChallenge, Inc.
1.888.782.7820x720 *|* amfarr...@masschallenge.org
ONE Marina Park Drive *|* Boston, MA 02210


* Free m entorship, resources, coworking space. No equity taken.
 Applications now open: 2014 MassChallenge accelerator
 Apply or refer great startups today!*

On Wed, Mar 5, 2014 at 8:45 PM, Tom Evans  wrote:

> On Wed, Mar 5, 2014 at 8:52 PM, Andrew Farrell 
> wrote:
> > Привет Djanonaughts!
> >
> > tl;dr: django with buildout spends a lot of time looking for files that
> > aren't there when trying to do imports. How can I reduce this?
>
> (I did read :)
>
> The first thing that springs to mind is test discovery. I may be way
> off track, an easy way to test (other than read all the code/know
> already) would be a simple django management command that spits out
> the time. Comparing the time difference for that operation compared to
> when running a test should size the costs of starting django
> environment versus starting django environment and finding a specific
> test.
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAFHbX1%2B6qoCp%3DprVoCZ5X-CGX%2BxBZ%3DZembJZ937Qjydd9JA92g%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2By5TLZNe15tBpX92%3D%3D-mPix%3Dg3rU_ddsYjf1r5RvCpat8DB8w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Newbie: ImportError during syncdb

[David]

Dear django users,

I am new to django and have an ImportError: No module named bookmarks
problem.

It seems that as I tried to syncdb django does not recognize the
bookmarks module I have created before. In the past I was able to
syncdb, hence my confusion.

I am trying to teach myself by following "Learning Website Development
with Django" by Ayman Hourieh.

I am not working in a virtualenv.

Thanks for your guidance and hints!

David


settings.py:

INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_bookmarks.bookmarks',
)

david@ubuntu:~/Tools/Python/Code/myCode/Django/django_bookmarks$ python
manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 10, in 
execute_from_command_line(sys.argv)
  File
"/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py",
line 399, in execute_from_command_line
utility.execute()
  File
"/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py",
line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File
"/usr/local/lib/python2.7/dist-packages/django/core/management/base.py",
line 242, in run_from_argv
self.execute(*args, **options.__dict__)
  File
"/usr/local/lib/python2.7/dist-packages/django/core/management/base.py",
line 280, in execute
translation.activate('en-us')
  File
"/usr/local/lib/python2.7/dist-packages/django/utils/translation/__init__.py",
line 130, in activate
return _trans.activate(language)
  File
"/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
line 188, in activate
_active.value = translation(language)
  File
"/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
line 177, in translation
default_translation = _fetch(settings.LANGUAGE_CODE)
  File
"/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py",
line 159, in _fetch
app = import_module(appname)
  File
"/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line
40, in import_module
__import__(name)
ImportError: No module named bookmarks

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5317CBCC.4060908%40gmx.net.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Improving django test startup performance.

[Russell Keith-Magee]

On Thu, Mar 6, 2014 at 4:52 AM, Andrew Farrell wrote:

> Привет Djanonaughts!
>
> tl;dr: django with buildout spends a lot of time looking for files that
> aren't there when trying to do imports. How can I reduce this?
>
>
>
> I've long been frustrated with the time it takes my django installation to
> run tests. Now, I'm aware that some ways to think about to how to write
> fast and efficient tests in 
> django
>  include:
> - putting things in setUpClass rather than setUp
> - treating the database as hot lava (at least, when you don't have
> ManyToMany relations)
> - using sqlite3 in-memory
> - using mock to emulate instances
> - using nose to only run the tests you need
> But I want to talk about another component of test performance: Disk I/O
> syscalls
>
> My company (the startup accelerator MassChallenge) uses 
> buildout to
> manage dependencies and the several apps modules within our appication. So
> I run tests from the root of the buildout by calling `bin/django test`, a
> script that looks like the following: (note the datetime printing)
> ```
> #!/home/afarrell/projects/masschallenge/venv/bin/python
> from datetime import datetime
> print datetime.now().isoformat()
> print "before test"
>
> import sys
> sys.path[0:0] = [
>
>   
> '/home/afarrell/projects/masschallenge/mc2013/eggs/MySQL_python-1.2.3-py2.7-linux-i686.egg',
>   ... 38 other paths to various eggs...
>   '/home/afarrell/projects/masschallenge/venv/lib/python2.7/site-packages',
>   '/home/afarrell/projects/masschallenge/mc2013', #the root of our app
>   ]
>
> import djangorecipe.manage
>
> if __name__ == '__main__':
> sys.exit(djangorecipe.manage.main('mcproject.develop_settings'))
> ```
>
> I am in the habit of writing a single test as I am writing a method. To do
> this, I use nose's attrib 
> tag in
> a test file that looks like this:
> ```
> from nose.plugins.attrib import attr
> import nose.tools as nt
> from django.test import TestCase
> from datetime import datetime
>
> class TestExample(Testcase):
> @attr('now')
> def test_trivial_case(self):
> print datetime.now().isoformat()
> print "test has begun"
> nt.assert_equal(1 + 1, 2)
> ```
>
> I then run tests with `bin/django test mentor_match -a'now' -s` and this
> prints out the following:
> ```
> 2014-03-05T13:46:02.774142
> before test
> nosetests --verbosity 1 mentor_match -anow -s
> To reuse old database "test_02_06_18_29" for speed, set env var REUSE_DB=1.
> Creating test database for alias 'default'...
> 2014-03-05T13:46:28.612895
> test has begun
> ```
> As you can see, there are about 26 seconds from the test command is issued
> and when the first test actually runs. This delay seems small but it is
> noticeable and kinda annoying when that is the only test. You could load,
> present, and fire a musket in that time. There are two possible causes that
> I can think of:
> 1) The test database takes a long time to set up
> 2) The test command takes a long time to import all required modules
>
> However the if I `export REUSE_DB=1` and then run `bin/django test
> mentor_match -a'now' -s`, I see
> 2014-03-05T15:22:51.743338
> before test
> nosetests --verbosity 1 mc_allocate -s -anow
> 2014-03-05T15:23:15.813821
> test has begun
> which is still a difference of 24 seconds. Working on a linode, I often
> get notifications about the high disk i/o rate, so it makes sense that the
> cause is the latter. So, I turn to the sysadmin's microsope: strace.
>
> With REUSE_DB not set, I run `strace -f bin/django test mentor_match -s
> -a'now' 2> tracelog`. I can now see all of the system calls made by the
> command, 67626. `cat tracelog | grep "open(" | wc -l` shows that 37635 of
> those are open commands.
>
> total:   67626
> open:37635  56.7%
> stat64:  15401  22.8%
> read: 5362   7.9%
> fstat64:  2619   3.9%
> write:2316   3.4%
> poll: 2268   3.4%
> close:1754   2.6%
> mmap2: 949   1.4%
> munmap:847   1.3%
>
> Since there are far more open calls than there are read calls, and
> there are 45847 lines containing "ENOENT (No such file or directory)"
> (86.4% of the open or stat64 calls), I think that django spends a lot of
> time looking for files in the wrong location.
>
>
> I plan to write a blog post about this, but before I dig in, does anyone
> have any comments on the following approaches I'm considering:
> - reordering the paths assigned to sys.path
> - removing items from sys.path when I am running tests
> - caching the locations where modules were previously found and looking
> there first
>
> Has anyone looked at doing anything like this?
>

Not to my knowledge, but based on your data, it certainly looks like it
would be worthwhile doing so.

I have to say that the data you've presented 

Re: Terrible performance when dropping into raw SQL Oracle

[junctionapps]

You understood it exactly. I've been cleaning up some old SQL statements 
(outside of use in Django) and have seen some real benefits of 
restructuring to reduce execution times. Unnecessary overuse of 'distinct' 
was a favourite of someone in my organization a few years back. Good luck!

On Wednesday, 5 March 2014 14:21:12 UTC-4, Shawn H wrote:
>
> I can't do that, as your assumption is incorrect. I don't have a table 
> that holds recordnumb as a unique value.  Each recordnumb occurs many 
> times, and occurs many times per case number, by design. The number is used 
> to identify unique ownership among many properties for a particular case - 
> each unique owner is assigned a recordnumb and that number is assigned to 
> each property that the unique owner owns. That's why we're doing a 
> distinct, so we can identify the number of unique recordnumb values for a 
> particular case.  BTW, I tried using max() instead of distinct, with no 
> change, which I expected since the database solution would be nearly 
> identical in that case.  Hopefully I didn't misunderstand what you were 
> requesting I try.
>
> On Monday, March 3, 2014 1:43:35 PM UTC-6, junctionapps wrote:
>>
>> Hey Shawn, would you do me a favour and try something a query with a 
>> implicit distinct like the following substituting YOURRECORDTABLE for 
>> whatever table holds your recordnumb as a unique value (I'm making a large 
>> assumption you have a table where each recordnum occurs once). Could you 
>> let me know if you see improvements in speed from the SQLPlus, and 
>> subsequently Django?
>>
>> SELECT count(1) from (
>>
>> SELECT RECORDNUMB FROM YOURRECORDTABLE
>> where RECORDNUMB IN 
>> (
>> SELECT RECORDNUM
>> FROM NOTICED_PARCELS 
>> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
>>  UNION count(1)
>> SELECT RECORDNUMB 
>> FROM CONDONOTICE 
>> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
>> )', [case_number, case_number]
>>
>> On Thursday, 27 February 2014 11:48:40 UTC-4, Shawn H wrote:
>>>
>>> The cursor.execute specifically.  I'm printing a timestamp immediately 
>>> before and immediately after that line.  I'm positive it is ONLY the query 
>>> that takes 16 seconds, whether using django.cursor or a cx_Oracle cursor.
>>>
>>> On Thursday, February 27, 2014 9:27:48 AM UTC-6, Scott Anderson wrote:

 Are you timing the query in the view specifically, or the entire view? 
 That is, do you know for sure that it is *only* the query that is taking 
 16 
 seconds and not the rest of the view?

 -scott

 On Wednesday, February 26, 2014 5:53:15 PM UTC-5, Shawn H wrote:
>
> I said that before testing it.  The exact same code using cx_Oracle 
> takes ~4 seconds outside of the django environment, but still takes ~16 
> seconds when running in the django view.  What the heck is going on? 
>  Updated portion of code below
>
> cnxn = cx_Oracle.connect('notification/notifydev@landmgm')
> cursor = cx_Oracle.Cursor(cnxn) #connections['landtest_11'].cursor()
> print datetime.datetime.now()
> cursor.execute('SELECT count(1) from (SELECT DISTINCT RECORDNUMB FROM 
> DEVGIS.NOTICED_PARCELS WHERE CASE_NUMBER =  AND RECORDNUMB > 0 UNION \
> SELECT DISTINCT RECORDNUMB FROM DEVGIS.CONDONOTICE WHERE CASE_NUMBER = 
>  AND RECORDNUMB > 0)', [case_number, case_number])
>  print datetime.datetime.now()
> number_count = cursor.fetchone()
>
>
> On Wednesday, February 26, 2014 4:41:06 PM UTC-6, Shawn H wrote:
>>
>> Because this worked so well, I've gone directly to cx_Oracle in my 
>> django view and used that to get the result in the 4 seconds.  There is 
>> definitely a problem with the Django implementation - I wonder if 
>> perhaps 
>> the indexes on the tables aren't being used properly.
>>
>> On Wednesday, February 26, 2014 3:49:47 PM UTC-6, Shawn H wrote:
>>>
>>> 3.8 seconds.  It seems to be django, not cx_Oracle.
>>>
>>> On Wednesday, February 26, 2014 2:50:58 PM UTC-6, Shawn H wrote:

 Good idea.  I'll try that and report back

 On Wednesday, February 26, 2014 1:22:52 PM UTC-6, Tom Evans wrote:
>
> On Wed, Feb 26, 2014 at 6:16 PM, Shawn H  
> wrote: 
> > Yes.  I've tested with several case numbers, and I'm using a 
> similar 
> > parameterized approach in my gui Oracle client as well, with the 
> same 
> > results.  It's always about 3 to 4 times slower running via 
> django.  I've 
> > tried it both on my local development web server as well as my 
> production 
> > apache linux box, and it always takes much longer running via 
> django. 
> > 
> > 
>
> If you write a standard python program, ie not using django, but 
> still 
> using whatever oracle DB adapter Django uses, that connects to 
> your 

Re: Terrible performance when dropping into raw SQL Oracle

[Kurtis Mullins]

Sort of off-topic as I don't have anything constructive to add to help fix
your problem but I recommend not running complex (long-running) queries
directly in Django views. Perhaps use Celery.

I know this doesn't solve your problem (although it might help a bit if
Django's environment is causing issues).

Good luck!


On Wed, Mar 5, 2014 at 1:21 PM, Shawn H  wrote:

> I can't do that, as your assumption is incorrect. I don't have a table
> that holds recordnumb as a unique value.  Each recordnumb occurs many
> times, and occurs many times per case number, by design. The number is used
> to identify unique ownership among many properties for a particular case -
> each unique owner is assigned a recordnumb and that number is assigned to
> each property that the unique owner owns. That's why we're doing a
> distinct, so we can identify the number of unique recordnumb values for a
> particular case.  BTW, I tried using max() instead of distinct, with no
> change, which I expected since the database solution would be nearly
> identical in that case.  Hopefully I didn't misunderstand what you were
> requesting I try.
>
>
> On Monday, March 3, 2014 1:43:35 PM UTC-6, junctionapps wrote:
>
>> Hey Shawn, would you do me a favour and try something a query with a
>> implicit distinct like the following substituting YOURRECORDTABLE for
>> whatever table holds your recordnumb as a unique value (I'm making a large
>> assumption you have a table where each recordnum occurs once). Could you
>> let me know if you see improvements in speed from the SQLPlus, and
>> subsequently Django?
>>
>> SELECT count(1) from (
>>
>> SELECT RECORDNUMB FROM YOURRECORDTABLE
>> where RECORDNUMB IN
>> (
>> SELECT RECORDNUM
>> FROM NOTICED_PARCELS
>> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
>>  UNION count(1)
>> SELECT RECORDNUMB
>> FROM CONDONOTICE
>> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
>> )', [case_number, case_number]
>>
>> On Thursday, 27 February 2014 11:48:40 UTC-4, Shawn H wrote:
>>>
>>> The cursor.execute specifically.  I'm printing a timestamp immediately
>>> before and immediately after that line.  I'm positive it is ONLY the query
>>> that takes 16 seconds, whether using django.cursor or a cx_Oracle cursor.
>>>
>>> On Thursday, February 27, 2014 9:27:48 AM UTC-6, Scott Anderson wrote:

 Are you timing the query in the view specifically, or the entire view?
 That is, do you know for sure that it is *only* the query that is taking 16
 seconds and not the rest of the view?

 -scott

 On Wednesday, February 26, 2014 5:53:15 PM UTC-5, Shawn H wrote:
>
> I said that before testing it.  The exact same code using cx_Oracle
> takes ~4 seconds outside of the django environment, but still takes ~16
> seconds when running in the django view.  What the heck is going on?
>  Updated portion of code below
>
> cnxn = cx_Oracle.connect('notification/notifydev@landmgm')
> cursor = cx_Oracle.Cursor(cnxn) #connections['landtest_11'].cursor()
> print datetime.datetime.now()
> cursor.execute('SELECT count(1) from (SELECT DISTINCT RECORDNUMB FROM
> DEVGIS.NOTICED_PARCELS WHERE CASE_NUMBER =  AND RECORDNUMB > 0 UNION \
> SELECT DISTINCT RECORDNUMB FROM DEVGIS.CONDONOTICE WHERE CASE_NUMBER =
>  AND RECORDNUMB > 0)', [case_number, case_number])
>  print datetime.datetime.now()
> number_count = cursor.fetchone()
>
>
> On Wednesday, February 26, 2014 4:41:06 PM UTC-6, Shawn H wrote:
>>
>> Because this worked so well, I've gone directly to cx_Oracle in my
>> django view and used that to get the result in the 4 seconds.  There is
>> definitely a problem with the Django implementation - I wonder if perhaps
>> the indexes on the tables aren't being used properly.
>>
>> On Wednesday, February 26, 2014 3:49:47 PM UTC-6, Shawn H wrote:
>>>
>>> 3.8 seconds.  It seems to be django, not cx_Oracle.
>>>
>>> On Wednesday, February 26, 2014 2:50:58 PM UTC-6, Shawn H wrote:

 Good idea.  I'll try that and report back

 On Wednesday, February 26, 2014 1:22:52 PM UTC-6, Tom Evans wrote:
>
> On Wed, Feb 26, 2014 at 6:16 PM, Shawn H 
> wrote:
> > Yes.  I've tested with several case numbers, and I'm using a
> similar
> > parameterized approach in my gui Oracle client as well, with the
> same
> > results.  It's always about 3 to 4 times slower running via
> django.  I've
> > tried it both on my local development web server as well as my
> production
> > apache linux box, and it always takes much longer running via
> django.
> >
> >
>
> If you write a standard python program, ie not using django, but
> still
> using whatever oracle DB adapter Django uses, that connects to
> your
> oracle 

Improving django test startup performance.

[Andrew Farrell]

Привет Djanonaughts!

tl;dr: django with buildout spends a lot of time looking for files that
aren't there when trying to do imports. How can I reduce this?



I've long been frustrated with the time it takes my django installation to
run tests. Now, I'm aware that some ways to think about to how to write
fast and efficient tests in
django
 include:
- putting things in setUpClass rather than setUp
- treating the database as hot lava (at least, when you don't have
ManyToMany relations)
- using sqlite3 in-memory
- using mock to emulate instances
- using nose to only run the tests you need
But I want to talk about another component of test performance: Disk I/O
syscalls

My company (the startup accelerator MassChallenge) uses
buildout to
manage dependencies and the several apps modules within our appication. So
I run tests from the root of the buildout by calling `bin/django test`, a
script that looks like the following: (note the datetime printing)
```
#!/home/afarrell/projects/masschallenge/venv/bin/python
from datetime import datetime
print datetime.now().isoformat()
print "before test"

import sys
sys.path[0:0] = [
  
'/home/afarrell/projects/masschallenge/mc2013/eggs/MySQL_python-1.2.3-py2.7-linux-i686.egg',
  ... 38 other paths to various eggs...
  '/home/afarrell/projects/masschallenge/venv/lib/python2.7/site-packages',
  '/home/afarrell/projects/masschallenge/mc2013', #the root of our app
  ]

import djangorecipe.manage

if __name__ == '__main__':
sys.exit(djangorecipe.manage.main('mcproject.develop_settings'))
```

I am in the habit of writing a single test as I am writing a method. To do
this, I use nose's attrib
tag in
a test file that looks like this:
```
from nose.plugins.attrib import attr
import nose.tools as nt
from django.test import TestCase
from datetime import datetime

class TestExample(Testcase):
@attr('now')
def test_trivial_case(self):
print datetime.now().isoformat()
print "test has begun"
nt.assert_equal(1 + 1, 2)
```

I then run tests with `bin/django test mentor_match -a'now' -s` and this
prints out the following:
```
2014-03-05T13:46:02.774142
before test
nosetests --verbosity 1 mentor_match -anow -s
To reuse old database "test_02_06_18_29" for speed, set env var REUSE_DB=1.
Creating test database for alias 'default'...
2014-03-05T13:46:28.612895
test has begun
```
As you can see, there are about 26 seconds from the test command is issued
and when the first test actually runs. This delay seems small but it is
noticeable and kinda annoying when that is the only test. You could load,
present, and fire a musket in that time. There are two possible causes that
I can think of:
1) The test database takes a long time to set up
2) The test command takes a long time to import all required modules

However the if I `export REUSE_DB=1` and then run `bin/django test
mentor_match -a'now' -s`, I see
2014-03-05T15:22:51.743338
before test
nosetests --verbosity 1 mc_allocate -s -anow
2014-03-05T15:23:15.813821
test has begun
which is still a difference of 24 seconds. Working on a linode, I often get
notifications about the high disk i/o rate, so it makes sense that the
cause is the latter. So, I turn to the sysadmin's microsope: strace.

With REUSE_DB not set, I run `strace -f bin/django test mentor_match -s
-a'now' 2> tracelog`. I can now see all of the system calls made by the
command, 67626. `cat tracelog | grep "open(" | wc -l` shows that 37635 of
those are open commands.

total:   67626
open:37635  56.7%
stat64:  15401  22.8%
read: 5362   7.9%
fstat64:  2619   3.9%
write:2316   3.4%
poll: 2268   3.4%
close:1754   2.6%
mmap2: 949   1.4%
munmap:847   1.3%

Since there are far more open calls than there are read calls, and
there are 45847 lines containing "ENOENT (No such file or directory)"
(86.4% of the open or stat64 calls), I think that django spends a lot of
time looking for files in the wrong location.


I plan to write a blog post about this, but before I dig in, does anyone
have any comments on the following approaches I'm considering:
- reordering the paths assigned to sys.path
- removing items from sys.path when I am running tests
- caching the locations where modules were previously found and looking
there first

Has anyone looked at doing anything like this?
-- 
*Andrew Farrell*
Software Engineer *|* MassChallenge, Inc.
1.888.782.7820x720 *|* amfarr...@masschallenge.org
ONE Marina Park Drive *|* Boston, MA 02210


* Free m entorship, resources, coworking space. No equity taken.
Applications now open: 2014 MassChallenge accelerator
 Apply or refer great startups today! *

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe 

Re: Terrible performance when dropping into raw SQL Oracle

[Shawn H]

I can't do that, as your assumption is incorrect. I don't have a table that 
holds recordnumb as a unique value.  Each recordnumb occurs many times, and 
occurs many times per case number, by design. The number is used to 
identify unique ownership among many properties for a particular case - 
each unique owner is assigned a recordnumb and that number is assigned to 
each property that the unique owner owns. That's why we're doing a 
distinct, so we can identify the number of unique recordnumb values for a 
particular case.  BTW, I tried using max() instead of distinct, with no 
change, which I expected since the database solution would be nearly 
identical in that case.  Hopefully I didn't misunderstand what you were 
requesting I try.

On Monday, March 3, 2014 1:43:35 PM UTC-6, junctionapps wrote:
>
> Hey Shawn, would you do me a favour and try something a query with a 
> implicit distinct like the following substituting YOURRECORDTABLE for 
> whatever table holds your recordnumb as a unique value (I'm making a large 
> assumption you have a table where each recordnum occurs once). Could you 
> let me know if you see improvements in speed from the SQLPlus, and 
> subsequently Django?
>
> SELECT count(1) from (
>
> SELECT RECORDNUMB FROM YOURRECORDTABLE
> where RECORDNUMB IN 
> (
> SELECT RECORDNUM
> FROM NOTICED_PARCELS 
> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
>  UNION count(1)
> SELECT RECORDNUMB 
> FROM CONDONOTICE 
> WHERE CASE_NUMBER = %s AND RECORDNUMB > 0
> )', [case_number, case_number]
>
> On Thursday, 27 February 2014 11:48:40 UTC-4, Shawn H wrote:
>>
>> The cursor.execute specifically.  I'm printing a timestamp immediately 
>> before and immediately after that line.  I'm positive it is ONLY the query 
>> that takes 16 seconds, whether using django.cursor or a cx_Oracle cursor.
>>
>> On Thursday, February 27, 2014 9:27:48 AM UTC-6, Scott Anderson wrote:
>>>
>>> Are you timing the query in the view specifically, or the entire view? 
>>> That is, do you know for sure that it is *only* the query that is taking 16 
>>> seconds and not the rest of the view?
>>>
>>> -scott
>>>
>>> On Wednesday, February 26, 2014 5:53:15 PM UTC-5, Shawn H wrote:

 I said that before testing it.  The exact same code using cx_Oracle 
 takes ~4 seconds outside of the django environment, but still takes ~16 
 seconds when running in the django view.  What the heck is going on? 
  Updated portion of code below

 cnxn = cx_Oracle.connect('notification/notifydev@landmgm')
 cursor = cx_Oracle.Cursor(cnxn) #connections['landtest_11'].cursor()
 print datetime.datetime.now()
 cursor.execute('SELECT count(1) from (SELECT DISTINCT RECORDNUMB FROM 
 DEVGIS.NOTICED_PARCELS WHERE CASE_NUMBER =  AND RECORDNUMB > 0 UNION \
 SELECT DISTINCT RECORDNUMB FROM DEVGIS.CONDONOTICE WHERE CASE_NUMBER = 
  AND RECORDNUMB > 0)', [case_number, case_number])
  print datetime.datetime.now()
 number_count = cursor.fetchone()


 On Wednesday, February 26, 2014 4:41:06 PM UTC-6, Shawn H wrote:
>
> Because this worked so well, I've gone directly to cx_Oracle in my 
> django view and used that to get the result in the 4 seconds.  There is 
> definitely a problem with the Django implementation - I wonder if perhaps 
> the indexes on the tables aren't being used properly.
>
> On Wednesday, February 26, 2014 3:49:47 PM UTC-6, Shawn H wrote:
>>
>> 3.8 seconds.  It seems to be django, not cx_Oracle.
>>
>> On Wednesday, February 26, 2014 2:50:58 PM UTC-6, Shawn H wrote:
>>>
>>> Good idea.  I'll try that and report back
>>>
>>> On Wednesday, February 26, 2014 1:22:52 PM UTC-6, Tom Evans wrote:

 On Wed, Feb 26, 2014 at 6:16 PM, Shawn H  
 wrote: 
 > Yes.  I've tested with several case numbers, and I'm using a 
 similar 
 > parameterized approach in my gui Oracle client as well, with the 
 same 
 > results.  It's always about 3 to 4 times slower running via 
 django.  I've 
 > tried it both on my local development web server as well as my 
 production 
 > apache linux box, and it always takes much longer running via 
 django. 
 > 
 > 

 If you write a standard python program, ie not using django, but 
 still 
 using whatever oracle DB adapter Django uses, that connects to your 
 oracle server and executes the query, is it still slow? 

 IE is the problem something django does, or how the adapter works. 

 Cheers 

 Tom 

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 

Re: django 1.6.1 raw_id_fields open a 'change' popup instead of 'select' popup

[Vittorio]

Il giorno 05/mar/2014, alle ore 03:54, Camilo Torres ha scritto:

> On Tuesday, March 4, 2014 4:47:58 AM UTC-4:30, Vittorio wrote:
> In an admin form I input all data for a specific invoice (client, date of 
> issue, number of invoice) and, by means of inlines, the ordered items. In 
> these inlines I defined a raw_id_fields referring to the 'items' model which 
> is showing correctly in the form, BUT ... when I click on it  a "change"   
> page is  often (not always) popping up instead of the expected "select" one. 
> Tis happens both in development and in production. 
> 
> Surfing the net I saw a message with the same subject as mine 
> http://stackoverflow.com/questions/21009319/django-1-6-1-raw-id-fields-open-a-change-popup-instead-of-select-popup
>  with a somewhat cryptic only answer... 
> Hello,
> 
> Try to delete your browser cache.
> 

No, it isn't a matter of browser cache. 
Let's narrow the problem focusing on the linux machine only where there's myapp 
and the apache2 server.
If I run my app via "python manage.py runserver 192.168.100.100:8000" it all 
works correctly:

 "GET /warehouse/warehouse/invoice/ HTTP/1.1" 200 2145
 "GET /warehouse/jsi18n/ HTTP/1.1" 200 2005
 "GET /warehouse/warehouse/invoice/5/ HTTP/1.1" 200 22551
 "GET /warehouse/warehouse/items/?t=code&_popup=1 HTTP/1.1" 200 21793 

and the correct select popup window's shown.

Whilst, *** using the same settings.py file ***, and Apache2 (see the 
enabled-site conf file below) in production I get a change popup window only 
and a nasty
/?e=1 appears at the and of the http called page.

Please help, I'm really stranded here.
Thanks a lot
Vittorio

=
#settings.py
import os
ABSOLUTE_PATH='%s/' % 
os.path.abspath(os.path.dirname(locals.__name__)).replace('\\','/')

DEBUG = True
ALLOWED_HOSTS=['*']
ADMINS = (
 ('Amministratore', 'wa...@gmail.com'),
)
EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'wa...@gmail.com'
EMAIL_HOST_PASSWORD = 'AbsolutelyFake99'

MANAGERS = ADMINS

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',#, 'mysql', 
'sqlite3' or 'oracle'.
'NAME': 'warehouse',  # Or path to database file if 
using sqlite3.
'USER': 'mycompany',  # Not used with sqlite3.
'PASSWORD': 'mycompany',  # Not used with sqlite3.
'HOST': '127.0.0.1',  # Set to empty string for 
localhost. Not used with sqlite3.
'PORT': '',  # Set to empty string for default. Not 
used with sqlite3.
},
'noematica': {
'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 
'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'noematica',  # Or path to database file if 
using sqlite3.
'USER': 'mycompany',  # Not used with sqlite3.
'PASSWORD': 'mycompany',  # Not used with sqlite3.
'HOST': '127.0.0.1',  # Set to empty string for 
localhost. Not used with sqlite3.
'PORT': '',  # Set to empty string for default. Not 
used with sqlite3.
},
}

TIME_ZONE = 'Europe/Rome'

LANGUAGE_CODE = 'it-it'
DECIMAL_SEPARATOR=','
THOUSAND_SEPARATOR='.'

SITE_ID = 1
USE_I18N = True
USE_L10N = True

USE_TZ = True
MEDIA_ROOT=''
MEDIA_URL = '/media/'
STATIC_ROOT = ''
STATIC_URL = '/static/'

STATICFILES_DIRS = (
 )

STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
SECRET_KEY = 'b(%_qs2$sk3_ht+@9+6=%t1ff$ey@f9$rrydsqwr$-z)xbk%*'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
MIDDLEWARE_CLASSES = (
'django.middleware.cache.UpdateCacheMiddleware',
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.gzip.GZipMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
'django.middleware.cache.FetchFromCacheMiddleware',
)
ROOT_URLCONF = 'mywh.urls'
WSGI_APPLICATION = 'mywh.wsgi.application'

TEMPLATE_DIRS = (
'/home/wh/mywh/mywh/templates',
)

SESSION_EXPIRE_AT_BROWSER_CLOSE = True

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',

Re: Class that extends a model that is not a model

[Javier Guerra Giraldez]

On Wed, Mar 5, 2014 at 10:52 AM, Tom Evans  wrote:
> Your derived classes are models, and they should be models. What they
> shouldn't be is have different tables for each model type, there
> should be one table that all instances are stored in to, as the data
> for each Calculation instance, regardless of type, is the same.
>
> This is a common idiom in ORMs, and is called Single Table Inheritance:
>
> http://www.martinfowler.com/eaaCatalog/singleTableInheritance.html
>
> Django doesn't support STI, but that doesn't really matter, you can
> get the same behaviour by using composition rather than inheritance,
> something like this:

that sounds very similar to django's proxy models:

https://docs.djangoproject.com/en/1.6/topics/db/models/#proxy-models

but I agree that this kind of problems are typically better solved byt
composition rather than inheritance.

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFkDaoSi0ETcKpTfRMNe1ziAXWhV%3DTgvdoeVzgW6jDaXxxN9EA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Class that extends a model that is not a model

[Domagoj Kovač]

Thanks Tom, this looks like something that can help me. Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/12ccd761-54d1-443c-86c2-f40f65838f44%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Class that extends a model that is not a model

[Tom Evans]

On Wed, Mar 5, 2014 at 1:03 PM, Domagoj Kovač  wrote:
> Hi Guys,
>
> I have a model called Calculation, this model has fields that are calculated
> and saved to the database, this model also hold some common calculation
> functions.
>
> I also have two other classes TrailerCalculation and TruckCalculation they
> both calculate values of the fields that will be saved to Calculation, they
> calculate same properties but in a different way. So i extended Calcution
> with TrailerCalculation and TruckCalculation. Is there some way that child
> classes don't need to be models even though they inherit a class that
> inherits a model.Models? I tried with class Meta: abstract = True but this
> gives me some kind of strange error.

Your derived classes are models, and they should be models. What they
shouldn't be is have different tables for each model type, there
should be one table that all instances are stored in to, as the data
for each Calculation instance, regardless of type, is the same.

This is a common idiom in ORMs, and is called Single Table Inheritance:

http://www.martinfowler.com/eaaCatalog/singleTableInheritance.html

Django doesn't support STI, but that doesn't really matter, you can
get the same behaviour by using composition rather than inheritance,
something like this:

CALC_TYPE_MAP = {
  'trailer': TrailerCalculation,
  'truck': TruckCalculation,
}

class Calculation(models.Model):
.
#
def calculate(self):
   calculator = CALC_TYPE_MAP[self.calculation_type]()
   calculator.calculate(self)

TrailerCalculation and TruckCalculation do not derive from
Calculation, they are simply objects that provide functionality to
Calculation. This approach is called "Composition over inheritance".

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFHbX1%2BcUr-ub91_jBuhA7bkoco6e-bT358e5bGJBJfyLg6kGA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL Namespaces

[Jonathan Baker]

Much like module namespacing, they allow you to group subitems in more
logical and descriptive ways while preventing naming collisions. So, if a
Django project has a blog, with a list of Posts at /blog/posts/, a sensible
url name for that pattern might be "post_list" (and using url names in your
application is a good thing). Now, say the same project also exposes a
public API, which includes includes Posts at the endpoint
/api/v1/blog/posts/. Since using url names is a good thing, what would you
name the API endpoint while not clobbering the already-defined 'post_list'
url name? Sure, something like 'api_post_list' might be acceptable for the
endpoint I just described, but I prefer the namespaced version
'api:post_list' for a few reasons:

1) You can define a namespace one time when you include a collection of URL
patterns into the top-level urls.py
2) Because you only have to do this once, changing namespaces is a matter
of changing one thing in one place
3) each url pattern in the namespace url pattern collection has no idea
it's even been namespaced, which means you can go on defining the url names
without having to remember some convention for preventing naming collisions
(since django takes care of the for you by following the namespace:url_name
pattern across the board).

Admittedly, this isn't very useful on small projects, but if you encounter
a situation like the one above (or your project grows very large), then the
potential for naming collisions very quickly becomes a reality.

Hope this helps,
Jonathan


On Wed, Mar 5, 2014 at 4:25 AM, anubhav joshi  wrote:

> Can anyone explain how URL Namespaces work in django.
> Some examples would be great.
> (Apart from those in the docs.)
>
> Regards,
> Anubhav Joshi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/1d59cdb2-d375-493d-88ae-da2ece757eee%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAPMFOb68DHThkFMW2mPtNPLW2cs4pA4XOVcYi-PygPpWCvgkAw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Class that extends a model that is not a model

[Domagoj Kovač]

I did not try that, and i don't think this would help. Separation to more 
files doesn't have anything to do with the way works, theoretically your 
whole code can be just one file.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b6eb2a4a-c92e-4cc1-8c15-7919e619f473%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Class that extends a model that is not a model

[Kelly Nicholes]

Just a guess, but have you tried putting it in a file not called models.py?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/831057be-5a5b-4fb9-a74a-5da810833b5f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Class that extends a model that is not a model

[Domagoj Kovač]

Hi Guys,

I have a model called Calculation, this model has fields that are 
calculated and saved to the database, this model also hold some common 
calculation functions.

I also have two other classes TrailerCalculation and TruckCalculation they 
both calculate values of the fields that will be saved to Calculation, they 
calculate same properties but in a different way. So i extended Calcution 
with TrailerCalculation and TruckCalculation. Is there some way that child 
classes don't need to be models even though they inherit a class that 
inherits a model.Models? I tried with class Meta: abstract = True but this 
gives me some kind of strange error.

Best,
Domagoj

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8009d592-ea63-44a8-943b-d3c83e576a54%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Intermittent mysql connection errors

[Nazeem S]

Hi,
 
Could you please share what was the solution, I'm experiencing the same.

On Saturday, October 27, 2012 2:49:50 AM UTC+5:30, Guy wrote:

> After more testing it appears that this is an O/S layer issue and not 
> specifically a django issue.
>
> On Thursday, October 25, 2012 3:07:16 PM UTC-7, Guy wrote: 
>>
>> I'm experiencing an annoying problem with a couple of django apps. Both 
>> apps are games built on Django that make regular calls to the db using 
>> python-db. We have error emails turned on, and every so often (the interval 
>> doesn't seem to follow a pattern but it's usually no more than once in a 
>> half-hour) we get an email with the following info (I've obscured/changed 
>> some identifying info, for the record):
>>
>>  OperationalError at /analytics/
>>
>> (2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")
>>
>>   Request Method: POST Request URL: 
>> http://hostname.domain/analytics
>>  Django 
>> Version: 1.3 Exception Type: OperationalError Exception Value: 
>>
>> (2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")
>>
>>  Exception Location: /usr/lib/pymodules/python2.6/MySQLdb/connections.py 
>> in __init__, line 170 Python Executable: /usr/sbin/uwsgi-1.2.3 Python 
>> Version: 2.6.5 Python Path: 
>>
>> ['app_path',
>>  '.',
>>  '',
>>  '/usr/lib/python2.6',
>>  '/usr/lib/python2.6/plat-linux2',
>>  '/usr/lib/python2.6/lib-tk',
>>  '/usr/lib/python2.6/lib-old',
>>  '/usr/lib/python2.6/lib-dynload',
>>  '/usr/lib/python2.6/dist-packages',
>>  '/usr/lib/python2.6/dist-packages/PIL',
>>  '/usr/lib/pymodules/python2.6',
>>  '/usr/local/lib/python2.6/dist-packages',
>>  '/usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103',
>>  'app_path/src',
>>  'app_path/src/expedition',
>>  'app_path/src/expedition/settings']
>>
>>  Server time: Thu, 25 Oct 2012 14:14:08 -0700
>> Traceback 
>>
>>- /usr/local/lib/python2.6/dist-packages/django/core/handlers/base.pyin 
>>get_response 
>> 1. 
>>   
>>   response = callback(request, *callback_args, 
>> **callback_kwargs)
>>   
>>Local Vars
>>  Variable Value  exceptions 
>>
>>> '/usr/local/lib/python2.6/dist-packages/django/core/exceptions.pyc'>
>>
>> e 
>>
>>OperationalError(2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")
>>
>> callback_args 
>>
>>()
>>
>> receivers 
>>
>>[(, None)]
>>
>> middleware_method 
>>
>>
>>
>> self 
>>
>>
>>
>> settings 
>>
>>
>>
>> request 
>>
>>>GET:,
>>POST:,
>>COOKIES:{},
>>META:{'CONTENT_LENGTH': '275',
>> 'CONTENT_TYPE': 'application/x-www-form-urlencoded',
>> 'DOCUMENT_ROOT': '/opt/nginx-1.2.1/html',
>> 'HTTP_ACCEPT': '*/*',
>> 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
>> 'HTTP_ACCEPT_LANGUAGE': 'en-us',
>> 'HTTP_CONTENT_LENGTH': '275',
>> 'HTTP_CONTENT_TYPE': 'application/x-www-form-urlencoded',
>> 'HTTP_HOST': 'hostname.domain',
>> 'HTTP_USER_AGENT': 'HTTPagent/1.2 CFNetwork/609 Darwin/13.0.0',
>> 'HTTP_VIA': 'HTTP/1.1 akrmspsrvz4ts212.wnsnet.attws.com',
>> 'HTTP_X_FORWARDED_FOR': 'x.x.x.x, x.x.x.x',
>> 'HTTP_X_REAL_IP': 'x.x.x.x',
>> 'PATH_INFO': u'/analytics/',
>> 'QUERY_STRING': '',
>> 'REMOTE_ADDR': 'x.x.x.x',
>> 'REMOTE_PORT': '49328',
>> 'REQUEST_METHOD': 'POST',
>> 'REQUEST_URI': '/analytics/',
>> 'SCRIPT_NAME': u'',
>> 'SERVER_NAME': 'hostname.domain',
>> 'SERVER_PORT': '80',
>> 'SERVER_PROTOCOL': 'HTTP/1.0',
>> 'uwsgi.core': 15,
>> 'uwsgi.node': 'hostname',
>> 'uwsgi.version': '1.2.3',
>> 'wsgi.errors': ,
>> 'wsgi.file_wrapper': ,
>> 'wsgi.input': > 0x309b5d0>,
>> 'wsgi.multiprocess': True,
>> 'wsgi.multithread': True,
>> 'wsgi.run_once': False,
>> 'wsgi.url_scheme': 'http',
>> 'wsgi.version': (1, 0)}>
>>
>> callback 
>>
>>
>>
>> resolver 
>>
>>
>>
>> urlresolvers 
>>
>>> '/usr/local/lib/python2.6/dist-packages/django/core/urlresolvers.pyc'>
>>
>> callback_kwargs 
>>
>>{}
>>
>> response 
>>
>>None
>>
>> urlconf 
>>
>>'expedition.urls'
>>
>>- 
>>
>> /usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103/newrelic/api/object_wrapper.pyin
>>  
>>__call__ 
>> 1. 
>>   
>>   self._nr_instance, args, kwargs)
>>   
>>   
>> The db connection works fine most of the time, it just seems like 
>> intermittently it can't connect. I've done a couple of things, like enabled 
>> tcp_tw_reuse (it seems I usually have about 150-200 connections to mysql in 
>> TIME_WAIT on any one web server), checked the mysql server and web servers 
>> message logs for too many connections errors, and looked at the mysql log 
>> to see if there's any info there. I also wrote a script that 

URL Namespaces

[anubhav joshi]

Can anyone explain how URL Namespaces work in django.
Some examples would be great.
(Apart from those in the docs.)

Regards,
Anubhav Joshi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1d59cdb2-d375-493d-88ae-da2ece757eee%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Django 1.7 dev: migration fails when adding a m2m (Related model cannot be resolved)

[baptiste . goupil]

Hello,

I'm struggling with migrations when trying to add a m2m between two apps in 
Django dev. I don't know if this is a bug or if I'm doing it wrong but 
basically, if main.models is:

from django.conf import settings
> from django.db import models
>
> class Restaurant(models.Model):
> name = models.CharField(max_length = 80)
> 
> class UserRestaurant(models.Model):
> restaurant = models.ForeignKey(Restaurant)
> user = models.ForeignKey('custom_auth.User')
>

And custom_auth.models is: 

from django.db import models
>
> class User(models.Model):
> address = models.CharField(max_length=100, blank=True)
>

Then adding the following field to custom_auth.models.User (that cannot be 
done in one migration because of a CircularDependency exception):

restaurants = models.ManyToManyField('main.Restaurant')
>

 Will raise the following exception when migrating (makemigrations is fine):

$ python3 manage.py migrate
> Operations to perform:
>   Synchronize unmigrated apps: admin, sessions, auth, contenttypes
>   Apply all migrations: custom_auth, main
> Synchronizing apps without migrations:
>   Creating tables...
>   Installing custom SQL...
>   Installing indexes...
> Running migrations:
>   Applying custom_auth.0002_user_restaurants...Traceback (most recent call 
> last):
>   File "manage.py", line 10, in 
> execute_from_command_line(sys.argv)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/core/management/__init__.py",
>  
> line 427, in execute_from_command_line
> utility.execute()
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/core/management/__init__.py",
>  
> line 419, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/core/management/base.py",
>  
> line 287, in run_from_argv
> self.execute(*args, **options.__dict__)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/core/management/base.py",
>  
> line 336, in execute
> output = self.handle(*args, **options)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/core/management/commands/migrate.py",
>  
> line 145, in handle
> executor.migrate(targets, plan, fake=options.get("fake", False))
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/migrations/executor.py",
>  
> line 60, in migrate
> self.apply_migration(migration, fake=fake)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/migrations/executor.py",
>  
> line 94, in apply_migration
> migration.apply(project_state, schema_editor)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/migrations/migration.py",
>  
> line 97, in apply
> operation.database_forwards(self.app_label, schema_editor, 
> project_state, new_state)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/migrations/operations/fields.py",
>  
> line 35, in database_forwards
> field,
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/backends/sqlite3/schema.py",
>  
> line 110, in add_field
> return self.create_model(field.rel.through)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/backends/schema.py",
>  
> line 191, in create_model
> definition, extra_params = self.column_sql(model, field)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/backends/schema.py",
>  
> line 108, in column_sql
> db_params = field.db_parameters(connection=self.connection)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1754, in db_parameters
> return {"type": self.db_type(connection), "check": []}
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1745, in db_type
> rel_field = self.related_field
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1651, in related_field
> return self.foreign_related_fields[0]
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1410, in foreign_related_fields
> return tuple(rhs_field for lhs_field, rhs_field in self.related_fields)
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1397, in related_fields
> self._related_fields = self.resolve_related_fields()
>   File 
> "/usr/local/lib/python3.3/dist-packages/Django-1.7a2-py3.3.egg/django/db/models/fields/related.py",
>  
> line 1382, in resolve_related_fields
> raise ValueError('Related model %r cannot be resolved' % self.rel.to)
> ValueError: Related 

Ajax Request on Admin Edit Form

[Jonathan Querubina]

Hi guys.

How can i edit the Admin Default Edit Form on django to insert my custom JS on 
the page?

I mean, when i am editing a model, i need to insert some JS code on it

Is this possible?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/26F9D3F2-917C-44A9-B926-C4185C74EC4F%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Showing which users are viewing a given page

[Some Developer]

On 04/03/2014 13:06, Paul J Stevens wrote:

On 04-03-14 13:28, Some Developer wrote:

Hopefully someone here will be able to point me in the right direction.

Basically I want to be able to view which users are viewing a page at
any given time and I want to be able to update it in real time using
JavaScript but I'm at a loss as to how to track which users are viewing
any given page in Django.

So enter some long-polling javascript that notifies the server every X
seconds that some user is still at the page you're tracking.


So for instance if I had mydomain.com/page/ I'd like a little box on the
top of the page saying "Page being viewed by users: xxx and yyy" and
then when one of those users navigates away from the page the box
updates to remove that user from the list.

This means that apart from the "I'm still here", you also need a "Who's
here at" as well.


I'm not sure if I have described what I want clearly enough so if you
need any more information let me know. Any help is appreciated :).

So for each page you want to track:

use a settimeout loop in javascript to:
- do a POST to some mini-view where you register who's at some page. You
can use the REMOTEUSER (auth-user) to find out who's who, and the
Referer header to see where they are in your site.
- this POST returns some data (json?) that will tell you who else is
there. You can now update your DOM to reflect this information.

If you don't want or care about authenticating users, you can't know who
is actually at the page, just the number of people, and their IP
addresses, browser version, etc.

Of course some people have done this before. Maybe check out TogetherJS
for more real-time collaboration goodies.

Thank you for the information; it is helpful. Along with the other tips 
here (I hadn't thought of using socket.io which looks really useful) I 
should be able to hack something together to get this working.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5316FDA8.3010105%40googlemail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Confused about ContentType

[Robin Lery]

Hello,

Suppose this is a model for a user's photo:

class Photo(models.Model):
user = models.ForeignKey(User)
title = models.CharField()
image = models.ImageField()
pub_date = models.DateTimeFied(auto_now=True, auto_now_add=pub_date)
update = models.DateTimeFied(auto_now=False, auto_now_add=True)

This is another model for user's status:

class Status(models.Model):
user = models.ForeignKey(User)
tweet = models.TextFied()
pub_date = models.DateTimeFied(auto_now=True, auto_now_add=pub_date)

Now, this is a model for all the feeds of a user:

class StreamItem(models.Model):
user = models.ForeignKey(User)
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = GenericForeignKey('content_type','object_id')

1. In the above class '*StreamItem*', is the *content_type *the data_type
of the instance?

2. What is *object_id*? Is it the pk of the instance or the pk for that
StramItem instance?

3. Lastly, is how do I get the pub_date of the above two class instance
(Photo and Status) and initialize it to the StreamItem instance?

I hope I made myself clear. I really need to know about content types. I
will really appreciate if you could help me understand.

Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2B4-nGqh4phgzjbiGpZ4SQpn356L6CwebGrd%3DPSGaHMnP1imkg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: confused about sql in django

[Robin Lery]

Thank you Alex and Tom! Its great to know something new. I think I will
stick with django orm for now! Thanks again!


On Wed, Mar 5, 2014 at 2:48 AM, Tom Evans  wrote:

> On Tue, Mar 4, 2014 at 8:08 PM, Robin Lery  wrote:
> > Hello,
> >
> > I am really being confused. How to use sql in django? I found two:
> > 1. Performing raw queries.
> > 2.Executing custom SQL directly.
> >
> > Does it have better performance than the django's orm? And how is the
> > performance the same between those two?
>
> 1 and 2 are the same thing. Running the same query using the ORM and
> running the same query directly will take the same time - the clue is
> in the name, it is the *same* query.
>
> Running queries that return model instances is slower using the ORM
> than simply fetching the data row at a time, simply because django
> will construct model instances from each row of data, which costs
> time.
>
> There are queries that django's ORM cannot express, for those
> obviously you must use SQL directly.
>
> The ORM will not write "bad" or slow queries, there is always a
> possibility that a query could be written better by hand. The ORM is
> powerful, but this power does allow you to write queries that look
> simple in code, but actually do complex and unexpected things when
> run. Of course, the same is true of writing SQL by hand.
>
> I think it is important to understand that the two do different
> things. ORM stands for Object Relational Mapper, which means it is
> perfect for extracting objects that are related to each other. SQL
> stands for Structured Query Language, which means it can be used to
> extract almost any data in whatever format you require. If you want
> objects, use the ORM, if you want arbitrary data, use SQL.
>
> Using an ORM does not mean that you no longer have to design and
> optimise your database and the queries you run on it. You still need
> to check your queries to ensure that your DB engine is optimising them
> correctly, that you have the correct indices on the columns you are
> querying on - probably also that you aren't repeating the same read
> queries over and over again within one request.
>
> >
> > And lastly, what is sqlalchemy? Are sqlalchemy and django's orm the
> same? In
> > the website of sqlalchemy it says "SQLAlchemy is the Python SQL toolkit
> and
> > Object Relational Mapper that gives application developers the full power
> > and flexibility of SQL".
>
> sqlalchemy is a different ORM to django. All ORMs do basically the
> same thing under the hood, so there are obvious similarities between
> the two. sqlalchemy has a few more features than django's ORM, at the
> cost of being more complex.
>
> You can use sqlalchemy in django instead of django's ORM/models, but
> you lose the ability to use anything based on django models - the
> admin site, authentication, basically anything that touches the
> database. You can still use templates, i18n, some session backends,
> forms (but not model forms), views and url handling.
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAFHbX1%2B7nkk-4Mh1MD03aHZyJnoNwAFOi%3DHiie7ahpg6iCBHUQ%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2B4-nGpGPLpTP7uRv7PrNRp1CwEiCLy-p1O9BUuU0Yc%3DDCLfOA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.