Re: Office365/SharePoint Online Integration

[Erik Cederstrand]

> Den 15. dec. 2017 kl. 22.18 skrev Alexander Joseph 
> :
> 
> I'm still new to django but I'm building an app that allows users to 
> authenticate through Office365. I'm using this -  
> https://github.com/Lamelos/django-allauth-office365
> and I got users to be able to successfully sign in with office365 but I now 
> need users to be able to upload documents to the sharepoint online sites 
> within office365. I'm not entirely sure how to go about this. If someone else 
> has any idea or has done something like this before any advice is much 
> appreciated. Since the users are already authenticated with Office365 and, 
> assuming I put the correct permissions/scope in my settings file and 
> dev.microsoft app, I thought maybe I could just try to upload a file from the 
> model like a normal upload path 
> ('https://company.sharepoint.com//SiteName/Shared%20Documents/') but this 
> didn't work. It seemed to upload the file without throwing any exceptions but 
> the file wasn't actually in the directory on SharePoint

Your question doesn't really have anything to do with Django, but I've used 
https://pypi.python.org/pypi/sharepoint previously with some success to work 
with SharePoint sites. I don't think it supports file uploads directly, but you 
could use it as a starting point to explore the Sharepoint SOAP API. But maybe 
Office365 Sharepoint sites also offer a more modern REST API you could use 
instead.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/EB0F3A4F-E224-4984-B054-6F40968BB7C6%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: transferring data between Postgres databases

[Erik Cederstrand]

> Den 24. okt. 2016 kl. 06.04 skrev Mike Dewhirst :
> 
> I have a user who has done a lot of work (instead of playing) in the staging 
> server and now wants to get that work into the production database.
> 
> I'm about to research selective dump and load capabilities of Postgres but in 
> the meantime can anyone suggest any shortcuts?

How much data are we talking about? Is it too much for django-admin 
dumpdata/loaddata to handle?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/476D393A-E05E-4D4D-8C42-F8EEA9704A5D%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Sharding DB: one database per user dynamically

[Erik Cederstrand]

> Den 6. okt. 2016 kl. 15.34 skrev Stefano Tranquillini 
> :
> 
> Well, not really.
> 
> I would like to shard the db, but probably i can just build a function to 
> write on DB1 or DB2 depending on the user id, this should be feasible, isn't 
> it? 

Yes, that would be feasible, of course. You could use standard Django routing 
if it fits your requirements, or implement your own.

You just haven't come up with a convincing argument for why you need the 
sharding. Sharding is normally something you use when dealing with enormous 
amounts of data, for performance reasons, not business logic. It goes beyond 
just splitting data into multiple databases with identical schemas, BTW. 
https://en.wikipedia.org/wiki/Shard_(database_architecture)

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/68D4DAFA-9627-4682-BC8E-E2FFC179892A%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Sharding DB: one database per user dynamically

[Erik Cederstrand]

> Den 6. okt. 2016 kl. 09.54 skrev Stefano Tranquillini 
> :
> 
> Hi all,
> 
> I quickly skimmed into the group but i did not find an updated answer. 
> 
> what i want to do is to have a database for each user that register to my 
> service, the structure of the databse is the same for all, the data are just 
> separated.
> I want to do this for two reasons:
> - be able to move database of users from place to place
> - be able to dump and restore single user databases

While your proposal is probably feasible, I would expect heavier arguments than 
the above to accept the added complexity and possible fighting with Django the 
get it to work.

Unless you have clear technical or legal requirements that dictate separate 
databases, just write export and import scripts that know to do "WHERE user_id 
= XXX" on the relevant tables.

Erik


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5FDDA79F-D343-4B8E-A0A4-514B6E75115D%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: covert julian date to calander date

[Erik Cederstrand]

> Den 21. sep. 2016 kl. 02.41 skrev sum abiut :
> 
> Thanks Erik, 
> 
> i think that should do the trick.is there a way to covert the date column 
> straight from the sql query or from the template?

If you want to do this from the template, create a custom template tag: 
https://docs.djangoproject.com/en/1.10/howto/custom-template-tags/

I don't have experience with MSSQL, but if you want to convert the date 
server-side, this should work: 
http://blogs.msmvps.com/robfarley/2009/03/25/converting-to-and-from-julian-format-in-t-sql/
 You could add the custom SQL as a RawSQL: 
https://docs.djangoproject.com/el/1.10/ref/models/expressions/#django.db.models.expressions.RawSQL

Depending on how your data is represented, you could also solve this with a 
custom model field: 
https://docs.djangoproject.com/en/1.10/howto/custom-model-fields/ The 
conversion code should go in the from_db_value() method.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5FA21E07-B29C-4036-BD7C-9F2DC30993AA%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: covert julian date to calander date

[Erik Cederstrand]

> Den 20. sep. 2016 kl. 01.40 skrev sum abiut :
> 
> Hi,
> how to you convert from Julian date to a Calender date. I am pulling data 
> from an MSSQL and i what to convert the Julian date to calender date before 
> displaying data on my template.

If I understand your question correctly, that should be easy. Just convert the 
Julian date to a Python date in your view:

   my_date = (datetime.datetime(my_year, 1, 1) + 
datetime.timedelta(days=julian_day)).date()


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/236EFE25-2F80-4247-9972-56DF60C662AA%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Django exit function

[Erik Cederstrand]

> Den 14. sep. 2016 kl. 23.00 skrev Krešimir :
> 
> Thank you, dear Sir!
> 
> I have tried as-user-atexit and it does not work with Ctrl+c event. Don't 
> know about other exit events since my machine powers the light in question. 
> So the light goes off when machine goes off anyway.

You *might* be able to wrap the UWSGI event loop to catch KeyboardInterrupt, 
and set up a signal handler for SIGTERM, SIGABRT etc, but it seems a wee bit 
overkill for your needs.

> Maybe if I constantly check fo existance of some django system process?

That would be the simpler and more fool-proof solution. Use cron if you can 
live with 1-minute resolution, otherwise use one of the many available service 
monitoring tools to check if your WSGI server is running and call 
turn_the_light_off() if not.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/E4E88B42-1817-4B11-BAFC-C5D44A08061E%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Django with digital certificates

[Erik Cederstrand]

> Den 12. sep. 2016 kl. 15.24 skrev Dimitris Kougioumtzis :
> 
> I have an application with django framework. The users have in their usb 
> sticks their certificates. How to access their certificates from the django 
> application to the client computer 

Do you have a Django app on a remote server that users access through a 
browser? In that case, it's probably not possible except for a small subset of 
problems (see e.g. https://developer.chrome.com/apps/app_usb or trusted Java 
applets) since normally, the browser can't access USB devices (or the 
filesystem in general) for security reasons.

I believe some of the Yubico products can emulate a keyboard, in which case the 
YubiKey simply types the secure token for you. That should work in a browser.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CDC04C7A-C10D-4BD2-8A2D-45F4A4BE9929%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Recipe to upgrade to Python 3.x on Ubuntu 12.04 and 14.04

[Erik Cederstrand]

> Den 7. sep. 2016 kl. 10.49 skrev James Schneider :
> 
> You may also want to consider building in a configuration manager such as 
> Ansible or Salt stack. Once set up, you can deploy multiple staging and prod 
> servers with a couple commands. With the right planning, that could 
> potentially work across major Ubuntu versions as an upgrade path. 
> 
> Probably won't save you time up front, but it definitely will next time this 
> comes around, with the added bonus of a near instantaneous recovery of a 
> dev/prod system build with little need for backups beyond your custom app 
> code and the play books for your config management.

I'll chime in and say that this is really a must nowadays if you're hosting 
your own services. Not having a server configuration that you can deploy to a 
new server within minutes is like building a Django project and not having a 
requirements.txt. Sure, you can pip install everything manually and memorize 
which packages you're using and which versions are compatible with your 
project, but a list of requirements is just so much better.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/39079C9D-A124-4BD1-9553-41121AC0FC35%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Maintaining old django code

[Erik Cederstrand]

> Den 6. sep. 2016 kl. 10.20 skrev Lekan Wahab :
> 
> Good morning guys.
> I was handed a project at work which was written as far back as 2012.
> Quiet a lot of the packages used in the project are either no longer being 
> maintained.
> Rebuilding the project from scratch is not option.
> There are way too many moving parts and way too many apps to rebuild it.
> Here are my questions:
> 
> 1. In the scenario i am in, what would be the best approach to update this 
> project or at least get it set up locally?

First, get the project to run on your local machine with the exact same package 
versions as was specified originally. You may have to look in 
lib/pythonx.x/site-packages on wherever it was running before, for the exact 
versions used, if there is no requirements file. Run the test-suite for your 
project and make sure everything is working.

Second, write more tests! You need lots of tests to verify any future changes 
you make to your own and external packages.

Third, either upgrade external packages to the newest versions, find 
alternatives if they are no longer maintained, or maintain the external package 
yourself from now on (assuming the license permits, of course). If the packages 
depend Django, you may have to do this gradually, as the newest package 
versions may no longer support Django 1.4 (see below).

> 2.  Quite a lot of changes have been made between django 1.4 (which was 
> originally used) and the current version(1.10), what would be the best 
> version for me to update the project to.
> I mostly use 1.8 anyway.

Take one minor Django version at a time, reading the upgrade notes along the 
way. Update your code to the new way of doing things, and move on to the next 
version. That's the supported and recommended way. Even though that's six 
versions to upgrade to, it will probably save you time in the long run instead 
of upgrading from 1.4 -> 1.10 in one go.

Happy coding!

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/EC2B71EE-D181-41FE-A953-C1D0A829F7C4%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Signal handling when deleting lots of objects?

[Erik Cederstrand]

> Den 1. sep. 2016 kl. 20.14 skrev Stodge :
> 
> I have two models, Volume and Group. The Volume model has a foreign key to 
> Group.
> 
> When a user deletes a Volume the post_delete signal handler sends an HTTP 
> DELETE request (/volume) to another server process. This works great. 
> However, when the user deletes a Group, the cascading delete also deletes all 
> volumes in that group. That means I get lots (I'm talking <100) of 
> post_delete signals for the Volume model and therefore lots of HTTP requests.
> 
> Is there anyway to avoid this? Ideally, I'd like to send the HTTP DELETE  
> request (/volume) when a volume is deleted, but send a different HTTP DELETE 
> request (/group) when the group is deleted and avoid sending any volume HTTP 
> DELETE requests.

You could disconnect the post_delete signal for Volume temporarily, but that's 
a hack.

You probably have to abandon signals on the Volume model. I would attach the 
post_delete signals logic directly to the Volume.delete() method and add an 
option to disable signaling:

class Volume(models.Model):
[]
def my_signal_logic(self):
do_whatever()

def delete(self, *args, **kwargs):
   with_signal = kwargs.pop('signal', True)
   if with_signal:
   self.my_signal_logic()
   super().delete(*args, **kwargs)
   

Then in the post_delete signal for Group, you delete the Volumes explicitly, 
telling delete() not to signal:

for v in instance.volumes.all():
v.delete(signal=False)

requests.delete('/group/%s' % instance.pk)


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/AD4D08EC-8C3E-49CC-BE1A-8358DC14F738%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: A tricky query in one to many relationship - atleast for me:)

[Erik Cederstrand]

> Den 30. aug. 2016 kl. 11.20 skrev Erik Cederstrand 
> <erik+li...@cederstrand.dk>:
> 
> I'm not even sure that's possible to express in SQL, but it would probably be 
> quite convoluted if it is. Here's an easier-to-understand solution:
> 
> res = set()
> for b in B.objects.all().select_related('a').annotate(Max('date_created')):
>if b.date_created != b.date_created__max:
>continue
>if b.text != 'ABCD':
>continue
>res.add(a)

I did some more experimenting. I think this actually does what you want:

res = [
b.a for b in B.objects
.filter(date_created=Max('a__b__date_created'))
.annotate(Max('a__b__date_created'))
.filter(text='ABCD')
.select_related('a')
]

which you can rewrite as:

A.objects.filter(
b__in=B.objects
.filter(date_created=Max('a__b__date_created'))
.annotate(Max('a__b__date_created'))
.filter(text='ABCD')
)

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/BE1AB708-D9DF-499D-92A4-950C11697778%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: A tricky query in one to many relationship - atleast for me:)

[Erik Cederstrand]

> for b in B.objects.all().select_related('a').annotate(Max('date_created')):

That should probably be: Max('a__b__date_created') instead.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/E911BFD1-0A2A-4943-AC80-E09D68C1A464%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: A tricky query in one to many relationship - atleast for me:)

[Erik Cederstrand]

I'm not even sure that's possible to express in SQL, but it would probably be 
quite convoluted if it is. Here's an easier-to-understand solution:

res = set()
for b in B.objects.all().select_related('a').annotate(Max('date_created')):
if b.date_created != b.date_created__max:
continue
if b.text != 'ABCD':
continue
res.add(a)


You'll get too many B objects, but it's only one query. If you want a more 
optimal query, try to write it in SQL first. Maybe it's possible to express in 
the ORM.

I used set() because 'date_created' is not unique so there could be two B 
objects with the same 'a' and 'date_created' values.

Erik


> Den 30. aug. 2016 kl. 08.33 skrev Web Architect :
> 
> Hi,
> 
> I am looking for an elegant and efficient mechanism to have a query filter or 
> a solution for the following one to many relationship model. Please note the 
> following is just an illustration of the models - hope it should provide what 
> I am looking for:
> 
> class A(models.Model):
> name = models.CharField(_("Name"), max_length=255, unique=True)
> 
> 
> class B(models.Model):
> text = models.CharField(_("Text"), max_length=255, unique=True)
> date_created = models.DateTimeField(_("Date Created"), auto_now_add=True) 
>  a = models.ForeignKey(A, related_name='b')
> 
> To get all the instances of B associated with a specific instance of A (say 
> 'a'), I could do the following : a.b.all()
> 
> The latest instance of B associated with 'a' would be : 
> a.b.latest('date_created')
> 
> Now I would like to have a list of all instances of A where the latest 
> instance of B associated with each instance of A will have the 'text' field 
> as 'ABCD'. 
> 
> A.objects.filter(b__text='ABCD') will give all instances of A where each 
> instance of A will have atleast one instance of B with 'text' = 'ABCD'.
> 
> There could be a brute force way of getting a solution for the above where in 
> I can do A.objects.filter(b__text='ABCD')  and then go through each instance  
> of A in a for loop over the queryset and check for the latest instance of B 
> for text='ABCD'.
> 
> As mentioned earlier, I am looking for an elegant and optimal way (if any in 
> Django) for the above query.
> 
> Thanks.
> 
>  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/a9ea312a-578e-4a2f-a739-a38e87a4b15b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/713734EC-F9E9-4CFC-B024-99DC693421D1%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: What is this? CASCADE clausule on DROP COLUMN???

[Erik Cederstrand]

> Den 25. aug. 2016 kl. 17.09 skrev RompePC :
> 
> I did a migration of Django, and then applied a sqlmigrate, giving me this 
> output...
> 
> BEGIN;
> --
> -- Remove field my_column from my_table
> --
> ALTER TABLE `my_table` DROP COLUMN `my_column` CASCADE;
> 
> I don't find info about it, and I've never seen something like this in my 
> MySQL experience. And the best part is that works (although it is marked as a 
> syntax error). Can anyone throw me some light about it?

Are you using MySQL as your backend? I believe that syntax is only for 
postgresql. CASCADE defines what to do if anything outside the table references 
the column. CASCADE will drop these references.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/809BE9C3-9AAD-43B5-818F-D1E6AA3E1ADB%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How o run shell command in Django ?

[Erik Cederstrand]

> Den 8. aug. 2016 kl. 13.14 skrev Asad ur Rehman :
> 
> Internal server error is coming..

If you want help, please be much more specific in your replies; post the 
relevant parts of the code, stacktraces, what have you tried already etc.

I assume you already added the code to your Django project and "Internal Server 
Error" is what you see in the browser. You need to set DEBUG=True in 
settings.py so you get better error reports than just an HTTP 500.

But start by verifying that the code snippet I sent previously actually works. 
Start a Python interpreter from the commandline (make sure you're in the same 
virtualenv as the server running your Django code) and do:

>>> from sh import fs_cli

If that works, we need a full stacktrace from Django to see what's going on.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0AB65718-BF99-4364-9086-EEC435CBE1D0%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How o run shell command in Django ?

[Erik Cederstrand]

> Den 8. aug. 2016 kl. 12.54 skrev Asad ur Rehman :
> 
>  from sh import fs_cli 
> this is also not working ...

What's the error message? Did you install the 'sh' module (pip install sh)?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/BB27E6C3-B9F8-4E9D-BF9B-DA9552387050%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How o run shell command in Django ?

[Erik Cederstrand]

I've had a lot of success interacting with the command-line by using the 'sh' 
module: http://amoffat.github.io/sh/ It's extremely simple but allows a lot of 
advanced features when you need them.

I'm not on a system where fs_cli is avaliable, but you should be able to just 
do:

from sh import fs_cli
result = fs_cli('-x')


And put that somewhere useful in your Django code, e.g. in a view or a 
management command.


Erik

> Den 8. aug. 2016 kl. 11.33 skrev Asad ur Rehman :
> 
> I want to run fs_cli -x command in django . Can anybody help me ?
> How can i create function in django to execute this command ?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/0f9ea28d-37c9-40f2-b67a-b566c3691998%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/114B7906-583D-4E5A-9348-05A34A961DF6%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Inconsistent dedent error

[Erik Cederstrand]

> Den 23. jun. 2016 kl. 21.21 skrev Gary Roach :
> 
> class Choice(models.Model):
>question = models.ForeignKey(Question, on_delete=models.CASCADE)
>choice_text = models.CharField(max_length=200)
>votes = models.IntegerField(default=0)
> 
>def _str__(self):
>return self.choice_text
> 
> The last entry (def __str__(self): throws an error "inconsistent dedent at 
> line  I can find nothing wrong.

The error is actually in the line above. Your field definitions are 7-space 
indented when they should only be 3- or 4-space indented.

Python requires you to have consistent indenting, preferably 4 spaces per every 
indentation level. Your editor should help you with that. See 
https://www.python.org/dev/peps/pep-0008/#indentation

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/D7F096DD-DC1E-4A51-9723-66C9EABFEFD8%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to aggregate on insert?

[Erik Cederstrand]

> Den 24. maj 2016 kl. 01.11 skrev James Schneider <jrschneide...@gmail.com>:
> On Mon, May 23, 2016 at 12:58 PM, Erik Cederstrand 
> <erik+li...@cederstrand.dk> wrote:
> 
> I have inherited a legacy Item model that has a composite unique key 
> consisting of a Customer ID and a per-customer, incrementing Item ID. Assume 
> I can't change the model.
> 
> On inserts, the legacy code would let the database increment the Item ID to 
> avoid race conditions. Something like this:
> 
>INSERT INTO item_table (customer_id, item_id, name, ...) VALUES (123, 
> (SELECT MAX(item_id) FROM item_table WHERE customer_id =123) + 1, 'MyItem', 
> ...); 
> 
> Is there any way I can do the same using the Django ORM without opening up 
> for race conditions? I.e. something better than:
> 
>i = Item(customer_id=123, name='MyItem', ...)
>i.item_id = 
> Item.objects.filter(customer_id=123).aggregate(Max('item_id'))['item_id__max']
>  + 1
>i.save()
> 
> 
> I feel like an explicit transaction wrapping this set of queries would be the 
> way to go to help avoid (or at least detect) a race condition: 

Thank you for your suggestions! The default isolation level for PostgreSQL is 
READ COMMITTED, so another thread could insert an identical (customer_id, 
item_id) pair between the select and the insert. I don't think a transaction 
would help here. I have a unique index on (customer_id, item_id) so save() 
would fail, and I only have one write statement, so there's nothing else to 
rollback if save() fails.

After digging around a bit more in the documentation and Django codebase, this 
solution seems to work for me:


from django.db.models import Value
from .models import Customer, Item


class IncrementingValue(Value):
def __init__(self, field, *args, **kwargs):
super().__init__(*args, **kwargs)
self.f = field

def as_sql(self, compiler, connection):
return '(SELECT MAX(%s) FROM %s WHERE customer_id=%%s) + 1' % 
(self.f.get_attname_column()[1], self.f.model._meta.db_table), [self.value.id]


c = Customer.objects.get(id=123)
i = Item.objects.create(customer=c, name='MyItem', 
item_id=IncrementingValue(value=c, field=Item._meta.get_field('item_id')))
i.refresh_from_db()


I didn't audit thoroughly for SQL injections. I would have used the ORM to 
generate the SQL for the SELECT MAX, but I can't seem to get the SQL for an 
aggregate() query.


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/F41E71C4-A91F-4BF4-B3A5-6E70966FF2CC%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to aggregate on insert?

[Erik Cederstrand]

> Den 23. maj 2016 kl. 22.49 skrev Ketan Bhatt :
> 
> Hey Erik,
> 
> What Django version are you on?

I'm on Django 1.9.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0175F099-8727-43B9-B419-4D2E48A63856%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

How to aggregate on insert?

[Erik Cederstrand]

Hi,

I have inherited a legacy Item model that has a composite unique key consisting 
of a Customer ID and a per-customer, incrementing Item ID. Assume I can't 
change the model.

On inserts, the legacy code would let the database increment the Item ID to 
avoid race conditions. Something like this:

   INSERT INTO item_table (customer_id, item_id, name, ...) VALUES (123, 
(SELECT MAX(item_id) FROM item_table WHERE customer_id =123) + 1, 'MyItem', 
...);


Is there any way I can do the same using the Django ORM without opening up for 
race conditions? I.e. something better than:

   i = Item(customer_id=123, name='MyItem', ...)
   i.item_id = 
Item.objects.filter(customer_id=123).aggregate(Max('item_id'))['item_id__max'] 
+ 1
   i.save()

Or do I just wrap that in a loop and catch IntegrityError if I don't want to 
use raw SQL?


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5361025E-2D7E-49D0-A704-47A165ECC217%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: get all columns as a list

[Erik Cederstrand]

> Den 20. maj 2016 kl. 00.12 skrev Larry Martell :
> 
> This is probably very simple, but I just can't figure out how to do it.
> 
> I want to get all the columns in some rows as a list. I know I could
> use values_list and flat=True and list all the columns, but is that
> the only way?
> 
> I want to do something like this:
> 
> rows = FOO.objects.filter(bar='baz')
> 
> and get a list of lists instead a list of FOO objects.

"MyModel.objects.filter().values_list()" returns a list of tuples with all 
column values in MyModel._meta.fields order. Does that not suffice?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DCCE3570-B1B7-4E44-A0B6-1CB0FE8A24BF%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Double free or Corruption error when using GeoDjango

[Erik Cederstrand]

> Den 19. maj 2016 kl. 22.17 skrev Tapan Pandita :
> 
> I am running django 1.9.4 on gunicorn19.4.5 with the gevent worker 
> (gevent==1.0.2, greenlet==0.4.9). My app is deployed on heroku. For some 
> requests, I have noticed this error: "*** Error in 
> `/app/.heroku/python/bin/python': double free or corruption (out): 
> 0x0403bc90 ***". I can't reproduce it deterministically, but it 
> happen in about 5% of all requests. I was able to get the stack trace for it 
> and everytime it's the same:
> 
> May 19 13:50:35  app/web.2:File 
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/gis/geos/geometry.py",
>  line 125, in __del__.
> May 19 13:50:35  app/web.2:  capi.destroy_geom(self._ptr).
> May 19 13:50:35  app/web.2:File 
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/gis/geos/libgeos.py",
>  line 157, in __call__.
> May 19 13:50:35  app/web.2:  return self.func(*args, **kwargs).
> May 19 13:50:35  app/web.2:File 
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/gis/geos/prototypes/threadsafe.py",
>  line 52, in __call__.
> May 19 13:50:35  app/web.2:  if not self.thread_context.handle:.
> May 19 13:50:35  app/web.2:File 
> "/app/.heroku/python/lib/python2.7/site-packages/gevent/local.py", line 173, 
> in __getattribute__.
> May 19 13:50:35  app/web.2:  d = object.__getattribute__(self, 
> '_local__dicts').get(getcurrent()).
> May 19 13:50:35  app/web.2:  *** Error in 
> `/app/.heroku/python/bin/python': double free or corruption (out): 
> 0x03dd6470 ***

This is bad. The python binary itself is crashing with a memory violation in 
C/C++ code in either Python or an external module (e.g. gevent). Unfortunately, 
your stack trace leaves no further hints about the source of the problem.

Normally, you would compile Python and any external C modules with debugging 
symbols and attach a debugger. I don't know Heroku very well, but your options 
are probably limited. Your best bet is probably to find a reproducible test 
case and contact their support team.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/92EC4AAF-B11F-4215-80E3-0221875A654F%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Proxy in front of django app

[Erik Cederstrand]

> Den 16. maj 2016 kl. 20.48 skrev Maks Materkov :
> 
> Hi everyone!
> 
> i' ve got an unusual question, that is probably not very specific to django. 
> We have a large multi-tenant django app, and we want to split all traffic 
> between two tenants (tenant 1 and tenant 2, for now). This tenant will be 
> connected to different databases. Our django app is 100% API, with token 
> authentication. Every request has auth_token parameter (as GET or POST 
> parameter). The token looks like: tenant1., tenant-2., tenant-3.xxx.
> 
> So, we want to route all traffic with tokens tenant1.xxx to app1 and tokens 
> tenant2. to app2. to do this, we need to setup something like "proxy 
> router" in front of out existing django app. We already using nginx, and we 
> can route to different backends only if token is GET parameter. There is no 
> way to route based on POST params, as far as I know. 
> 
> We are discussing several alternatives:
> 
> 1) Add LUA module to nginx (this module can read POST body data) and perform 
> routing here
> 2) Write another app (for example, based on Tornado or python3 asyncio) that 
> will be reading POST and GET params and route traffic to appropriate django 
> app. (or another languages? NodeJS (oh, no :) )? Go lang?)
> 
> What is the best option here for us? Or some completely different approach?

The answer to this depends in part on your performance requirements. Do you 
have *any* control of the client side? If so, you could simply send all API 
requests to tenant1.api.example.com, tenant2.api.example.com etc.

If you have no control over clients, the most simple solution would be to write 
a simple Django view that handles every request and passes it on to the correct 
backend. This keeps your code within Django. But this also requires two trips 
all the way down your stack and may not yield performance. I've had great 
success in the past with Nginx scripting, so that would be my next suggestion 
if the above are not viable. It may cost you a bit more in maintainability, 
though.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/F3BE4330-C320-4C0F-B04D-799747F12F48%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to flush/clear database for single app, i.e. NOT the Project?

[Erik Cederstrand]

> Den 16. maj 2016 kl. 19.16 skrev JoeCodeswell :
> 
> How can I flush/clear database for single app, i.e. NOT the Project?
> 
> I have a multi-app project. 
> In a particular app I created my model with single class. 
> I populated the db for that model with the Admin Interface.
> Now i have changed the model to have 2 classes related by a ForeignKey.
> I ran makemigrations on that app and it said, "you are trying to add a 
> non-nullable field 'song' to part without a default; we can't do that (the 
> database needs something to populate existing rows)."
> I don't want to keep the existing populated tables for that app.

I can do it manually before the migration, i.e.:

   YourModel.objects.all().delete()

If you want to do it as part of the migration, then create a data migration 
(https://docs.djangoproject.com/en/1.9/topics/migrations/#data-migrations) that 
deletes all relevant model instances, and place the migration before the 
migration that adds the ForeignKey.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/136259B2-2BE4-4AD0-A49E-5F4D7C4CC9A0%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Displaying single-line progress while a management command runs

[Erik Cederstrand]

> Den 9. maj 2016 kl. 14.23 skrev Phil Gyford :
> 
> I have a custom management command which calls a method in another class, 
> which fetches lots of data from a third-party API. Fetching the data could 
> take a few seconds or it could take over an hour, depending on the quantity.
> 
> [...]
> Things I've tried so far:
> 
> 1) Using print(), e.g.:
> 
> print('Fetched %d of %d' % (n, total), end='\r')
> 
> In a loop, this nicely shows a single line that constantly updates with 
> progress. But print() is nasty and when I run my unit tests, this output is 
> displayed among the testing output. I assume it'll also be a pain to have 
> that output when running the commands scheduled with cron (or whatever).

I do this kind of progress reporting a lot. Usually, I get around the test/cron 
output pollution by adding a 'silent' argument to the management command which 
determines if the commend should print progress reports or not. See below.

> 2) Using Django logging. This is "better" than print(), and doesn't mess up 
> test output, but as far as I can tell there's no way to display a single, 
> constantly updated, line showing progress. It's only going to show one line 
> after another:
> 
> Fetched 1 of 3000
> Fetched 2 of 3000
> Fetched 3 of 3000

It's actually quite simple. You need to create a custom handler like so:

  import logging
  import time
  from django.core.management.base import BaseCommand

  class OverwriteHandler(logging.StreamHandler):
  # The extra spaces wipe previous output in case your messages are 
wariable-width
  terminator = ' '*80 + '\r'

  log = logging.getLogger('')
  h = OverwriteHandler()
  log.addHandler(h)

  class Command(BaseCommand):
def handle(self, silent=False, **options):
  log.setLevel(logging.DEBUG if silent else logging.INFO)
  log.info('1 of 2')
  time.sleep(1)
  log.info('2 of 2')
  time.sleep(1)

If you want to mix normal and progress logging in your management command, you 
need to use two loggers with different handlers.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/463A7786-888C-4CB0-9C68-43F855401924%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: manipulate n insert field values

[Erik Cederstrand]

> Den 4. maj 2016 kl. 10.07 skrev Shameema Mohsin :
> 
> 
> Z value calculation
> 
> step1: converting lat and long to int
> 
> latitude int = (latitude + 90) × 10^6
> longitude int = (latitude + 180) × 10^6
> 
> note 10^6 = 100 
> 
> We compute the Morton value for a spatial point P (x, y)
> by interleaving the bits of x and y. For example, when x =
> 3(011 ) and y = 4(100), the Morton value for P is
> 100101 = 37 
> 
> step2: convert int (base 10) to binary digits (base 2)
> ..?
> step3: interleave both values bits and compute z value
> take first bit from y cordinate  (longitude) : 1
> take first bit from x cordinate  (lat) : 0
> take 2nd bit from y cordinate  (longitude) :0
> take 2nd bit from x cordinate  (lat) : 1
> #take 3rd bit from y cordinate  (longitude) : 0
> #take 3rd bit from x cordinate  (lat) : 1
> 
> therefore z binary value is 100101 
> and z value is 37
> 
> #this is the technique used to convert spatial 2D indexing to 1D to improve 
> efficiency.
> 
> I do not know the python syntax. Kindly help.

I'm sorry, but asking for a full Python implementation of the above is too much 
to ask for free help for. Implement it in PHP if that's what you're familiar 
with, and grab a Python handbook to look up the equivalent Python syntax.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/EDD2FC06-6471-44E4-8BFD-F02C59AC42FD%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: manipulate n insert field values

[Erik Cederstrand]

> Den 3. maj 2016 kl. 02.45 skrev Shameema Mohsin :
> 
> 
> Stil stuck with the z order calculation and usage of the property field. 
> Kindly help. I am new to Django, only knows php well.

Where are you stuck? Show your error messages, current non-working code and 
expected result, or describe your problem in more detail.

I assume you have code to calculate the z order (I have no idea what it is). 
Otherwise, other forums would be better for help on this. Assuming that, you 
would do:

def get_z_order(lat, long):
   # Jiggle those numbers
   return z_order


class Employees(models.Model):
   [...]
   @property
   def zval(self):
   return get_z_order(self.empLat, self.empLong)


You can then access zval as if it was a model field.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0E25E884-F38C-4314-B2B3-0ADAF813253E%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: manipulate n insert field values

[Erik Cederstrand]

> Den 2. maj 2016 kl. 15.38 skrev Shameema Mohsin :
> 
> Could anybody help me out as I am new to Django.
> In my model:
> 
> from django.db import models
> 
> class Employees(models.Model):
>empId = models.AutoField(primary_key=True)
>empName = models.CharField(max_length=200)
>empEmail = models.EmailField()
>empMob = models.CharField(max_length=200)
>empAddr = models.TextField()
>empLat = models.DecimalField(max_digits=9,decimal_places=6)
>empLong = models.DecimalField(max_digits=9,decimal_places=6)
>empDOB = models.DateField()
>zval = models.IntegerField(default=0)
> 
>def __str__(self):
>return self.empName
> 
> In the z value field i need to insert the z curve value calculated from 
> latitude and longitude. where should I write the code? In models.py or 
> somewhere else?

There are two strategies here. If zval is always generated from empLat and 
empLong, then you could change zval from a field to a property on the Employees 
class. Otherwise, you risk zval falling out of sync if you are not careful to 
always re-calculate zval when empLat/empLong changes:

  class Employees(models.Model):
 [...]
 @property
 def zval(self):
return # your calculation here


The other strategy would be to place the calculation in the save() method:

  class Employees(models.Model):
 [...]
 def save(self, *args, **kwargs):
self.zval = # your calculation here
super().save(*args, **kwargs)

Beware that save() is not always called. For example, bulk_create() doesn't 
call save(). You need to handle these situations yourself.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/97FEF314-509F-48AF-8A41-860CDBC6F514%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Strange Bug

[Erik Cederstrand]

Hi Neto,

> Den 22. apr. 2016 kl. 06.59 skrev Neto :
> 
> [...]
> class ActionLog(Log):
> action = models.ForeignKey('Action', on_delete=models.SET_NULL)
> 
> [...]
> 
> ActionLog.objects.create(account=account, action=action)

Something's wrong with your example. The ActionLog class only has an 'action' 
field, but you're creating an ActionLog instance as if it has both 'action' and 
'account'.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B2B07CF6-A91A-4162-900B-0DF4E0F9151F%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Issue with Django Password Normalization

[Erik Cederstrand]

> Den 20. apr. 2016 kl. 14.41 skrev Arun S :
> 
> basically I would like to know if the latest version of django already 
> supports any kind of normalization for the login passwords.

What exactly do you mean by "password normalization"? Do you want passwords to 
be case-insensitive? If so, you can subclass AuthenticationForm and override 
clean_password(), and set_password() on your user model, and put any 
transformations of the raw password there.

If you want to enforce certain password rules (length, must contain numbers and 
special chars, etc) then override AuthenticationForm.clean_password() and raise 
ValidationError() for your rules.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/AA50597A-699E-4390-B442-97D76BDF5D78%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Design hints for a sanity checker

[Erik Cederstrand]

Hi list,

I need to implement a sanity checker for my (power) users. Basically they 
should press a button on my website, and my backend runs a checklist of 20-30 
items to check that their setup is OK and report progress while the checks are 
running. Some checks are fast, and others could take a long time - testing 
connections to external systems and things like that.

I'm still a static HTML kinda guy, so I'm looking for hints on how to design 
this with Django. I'm looking for something simple, but also best practice so I 
learn the right habits from the start :-)

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B051E034-7D58-472E-9D64-2EC8B1FC4712%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Q() chaining with & returns empty set

[Erik Cederstrand]

> Den 31. mar. 2016 kl. 04.14 skrev Bai Shun :
> 
> I've got a problem in using Q()

To see what's going on, try to print the SQL that Django generates for working 
and non-working queryset:

> print 
> Basket.objects.filter(Q(fruit__ofkind__name__in=['apple'])(fruit__ofkind__name__in=['banana'])).distinct()

> Basket.objects.filter(fruit__ofkind__name__in=['apple']).filter(fruit__ofkind__name__in=['banana']).distinct()

Print SQL like this:

   print(my_queryset.query)


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2C4D0275-020B-4236-9F11-154979510967%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: reading CSV file with non ASCII characters

[Erik Cederstrand]

> Den 6. feb. 2016 kl. 09.14 skrev elcaiaimar :
> 
> Hello,
> 
> I have a CSV File and I want read it. The problem is that it has non ASCII 
> characters such as 'Ñ' and accents and I need that they are recognised to 
> save the CSV content in a DB.
> 
> To simplify, I've summed up my code in django to the next:
> 
> ​import csv
> 
> reader = csv.DictReader(open("file.csv", "rb"))

You should always convert bytestrings to unicode as soon as possible in Python. 
You need to specify the encoding of your file, e.g.:

reader = csv.DictReader(open("file.csv", "rb", encoding='utf-8'))

See https://docs.python.org/3/library/functions.html#open

For Python 2, have a look at the notes about encodings in 
https://docs.python.org/2/library/csv.html

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7E3805BB-72BF-4865-8C82-3BCBAD689F16%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Scaling Django

[Erik Cederstrand]

> Den 3. feb. 2016 kl. 22.30 skrev Joshua Pokotilow :
> 
> At the startup where I work, we've written a lot of our server code in 
> Django. So far, we've adopted a "build it fast" mentality, so we invested 
> very little time in optimizing our code. A small amount of load testing has 
> revealed our codebase / infrastructure as it stands today needs to run faster 
> and support more users.
> 
> We recently hired some new engineers who are extremely skeptical that we 
> should optimize our existing code.

I was in a startup like that. We *had* a working solution, and we *had* 
customers. Not enough to pay our salaries, but enough to keep us and our 
investors hopeful.

Someone decided we needed a rewrite, because Django, because blog posts, 
because WebScale(TM), because in one year we might have 1000-fold users if our 
wildest startup dreams came true. So we started to rewrite. It was supposed to 
take one month. Our working, legacy solution started to deteriorate because we 
were busy rewriting. Two months. Bugs reports piled up in the tracker, but we 
didn't care because the rewrite was just around the corner and would solve 
everything, and we couldn't possibly work on two systems at the same time. Some 
customers left, but it was okay because our WebScale solution would make us 
filthy rich. Three months. Everyone was overworked, tired and the WebScale 
solution was still just around the corner... Four months, and our investors 
decided we were not part of the 1%.

In short, don't rewrite. Refactor. And know *exactly* why you are refactoring. 
As in, "We have profiled, discussed architecture, hardware, algorithms, etc, 
etc. The GIL is killing us and Guido doesn't care", not "Django doesn't scale". 
The year "monolithic" is an argument in itself is the year of the HURD desktop.

Except if you're programming in VBScript. Then by all means, rewrite.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DC23E544-1108-40A5-821D-681E498DBEAC%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to define custom labels for admin.ModelAdmin readonly fields.

[Erik Cederstrand]

> Den 11. jan. 2016 kl. 23.21 skrev Jonty Needham :
> 
> I have a ModelAdmin class where I am using callables to define accessors to a 
> foreign key's field.
> 
> I.e.
> 
> class MyAdmin(admin.ModelAdmin):
> 
> def field_name(self, obj):
> return obj.fk.field_name
> 
> readonly_fields=(field_name, other_field)
> 
> 
> Ultimately I need to translate the labels, but I need access to them first 
> and I can't see from the ModelAdmin code where that is.

It sounds like you're looking for the 'short_description' option described in 
https://docs.djangoproject.com/en/1.9/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_display

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9E3DD769-26A9-4DD5-A1D7-08327ECB5358%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Disable autocomplete in admin field

[Erik Cederstrand]

> Den 12. nov. 2015 kl. 14.41 skrev Erik Cederstrand 
> <erik+li...@cederstrand.dk>:
> 
> Hi Ezequeil,
> 
> Thanks for the explanation! This worked, but defining a new widget from 
> scratch meant that I lost the other helpful attributes (length, class etc.) 
> that the admin adds for me. Instead, in the DRY spirit, I opted to just add 
> this one extra attribute. Here's the relevant code:
> 
> class MyAdminForm(forms.ModelForm):
>def __init__(self, *args, **kwargs):
>super().__init__(*args, **kwargs)
>self.fields['username'].widget.attrs['autocomplete'] = 'off'

Just to follow up, this didn't actually work for me, I just thought so. 
Apparently, all major browsers have go to great lengths to ignore any attempts 
to disable autocomplete, and any suggestions to actually get it working 
involves hidden fields, custom javascript and all sorts of hoops and tricks 
(see e.g. 
http://stackoverflow.com/questions/22661977/disabling-safari-autofill-on-usernames-and-passwords).

So it seems I'm stuck with either disabling autocomplete entirely in my 
browser, or having the username for my LDAP server overwritten with 'erik' 
every time I open that form. Infuriating.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2A4E02D8-0813-4FA0-9804-BA1015C76472%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Disable autocomplete in admin field

[Erik Cederstrand]

Hi Ezequeil,

Thanks for the explanation! This worked, but defining a new widget from scratch 
meant that I lost the other helpful attributes (length, class etc.) that the 
admin adds for me. Instead, in the DRY spirit, I opted to just add this one 
extra attribute. Here's the relevant code:

class MyAdminForm(forms.ModelForm):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields['username'].widget.attrs['autocomplete'] = 'off'

Erik


> Den 12. nov. 2015 kl. 13.33 skrev Ezequiel Bertti <eber...@gmail.com>:
> 
> 
> In html:
> 
> 
> 
> Um django you need to create a form for your model and set a field to 
> username. In this field, set a widget like this:
> 
> username = forms.CharField(widget=forms.TextInput(attrs={'autocomplete': 
> 'off'}))
> 
> Now you set a form on your model admin, like this example
> 
> https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#adding-custom-validation-to-the-admin
> 
> 
> On Thu, Nov 12, 2015 at 8:46 AM, Erik Cederstrand <erik+li...@cederstrand.dk> 
> wrote:
> Hello,
> 
> I have a model with a CharField named "username". When I edit a model 
> instance in the admin, my browser likes to autocomplete the contents of the 
> "username" field with my username for the Django site, regardless of what was 
> entered previously.
> 
> Is there anything I can do to disable this behaviour, except for renaming the 
> field to "username_PLEASE_DONT_AUTOCOMPLETE_SAFARI_IM_LOOKING_AT_YOU"?
> 
> Thanks,
> Erik
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/DFF8D42F-D4F1-4686-942E-A906F919488B%40cederstrand.dk.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> -- 
> Ezequiel Bertti
> E-Mail: eber...@gmail.com
> Cel: (21) 99188-4860
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CACrQMYrb9aDDD1L%2BFvOtn90M6YDEuaVODoO-JWPqF0E5YiS9UA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B74769F9-9905-4998-AF98-2461854321C1%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Disable autocomplete in admin field

[Erik Cederstrand]

Hello,

I have a model with a CharField named "username". When I edit a model instance 
in the admin, my browser likes to autocomplete the contents of the "username" 
field with my username for the Django site, regardless of what was entered 
previously.

Is there anything I can do to disable this behaviour, except for renaming the 
field to "username_PLEASE_DONT_AUTOCOMPLETE_SAFARI_IM_LOOKING_AT_YOU"?

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DFF8D42F-D4F1-4686-942E-A906F919488B%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: bug: mymodel.objects.first().get('id',None) AttributeError: 'NoneType' object has no attribute 'get'

[Erik Cederstrand]

> Den 27. okt. 2015 kl. 04.01 skrev gugeshi...@gmail.com:
> 
>  hi ,how are you 
> 
> i run 
> 
> myid=mymodel.objects.first().get('id',None) 
> 
> If there is a record， it's ok
> 
> If there is no record， error info：
> 
> AttributeError: 'NoneType' object has no attribute 'get'
> 
> 
> If there is a record，Can you let it return an empty dictionary,


I don't see how .get() would not throw an AttributeError if first() returns a 
mymodel instance, but here's one solution to your problem:

myid = mymodel.objects.values('id').first() or {}


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B464C21A-55DC-475C-B7F4-CCA679219AD8%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: What is the recommended schema design for dynamic dates in Django for PostgreSQL?

[Erik Cederstrand]

> Den 20. okt. 2015 kl. 18.09 skrev Radek Svarz :
> 
> Hi Erik,
> 
> thanks for m2m suggestion.
> 
> What do you think about using suggested view in DB? (@SO) (i.e. the DB would 
> do the conversion on the fly)

In general, I think of DB views as either a performance optimization or a way 
to store business logic. I like to keep my business logic in one place (Python 
code) and hold off on performance optimizations until they are identified as 
necessary, so I would stay away from views until you have no other reasonable 
options.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1CF8A3E2-4751-424C-BCD3-B7612F2421A7%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: What is the recommended schema design for dynamic dates in Django for PostgreSQL?

[Erik Cederstrand]

> Den 20. okt. 2015 kl. 14.09 skrev Radek Svarz :
> 
> [...]
> Django code and further long details are @ SO: 
> http://stackoverflow.com/questions/33167079/what-is-the-recommended-schema-design-for-dynamic-dates-in-django-for-postgresql
>  

Regarding your need for a way to denote "natural language" relative dates, take 
a look at https://pypi.python.org/pypi/dateparser. You could either use that 
directly or build on top of it.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/486AC74C-3E93-42D6-A628-A16511A368ED%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: What is the recommended schema design for dynamic dates in Django for PostgreSQL?

[Erik Cederstrand]

> Den 20. okt. 2015 kl. 14.09 skrev Radek Svarz :
> 
> [...]
> 1 Milestone is a date stored as a string in form -MM-DD or a special tag 
> "today", which means daily changing date (dynamic - the date was not stated, 
> but until today is some state valid - if today is smaller then the next 
> milestone).
> [...]
> Overall current implementation works fine, however we have issues with 
> reporting questions such as:
> 
> Which items will hit milestones of certain characteristics in December 2015?

If you want the dates to be searchable, then don't store them in a Charfield. 
Store them in a DateField instead. In fact, always store dates in a DateField.

In any case I would create a separate "Milestone" model with an m2m relation to 
Lifecycle. Put your "today" logic for milestones in this model, instead of 
using N duplicate date fields. Either use null=True to denote your concept of 
"today", or add a separate Boolean field "is_today" on the "Milestone" model.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DDD53C31-A5F2-4DE9-9848-6BB788E1A08F%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Prefetch() with through models

[Erik Cederstrand]

> Den 16/09/2015 kl. 19.08 skrev Simon Charette :
> 
> If you want to use this approach I'm afraid you'll have to do
> what prefetch_related does under the hood by yourself.
> 
> e.g.
> 
> from collections import defaultdict
> 
> prefetched = defaultdict(list)
> subjects = Subject.objects.filter(...)
> lessons = Lesson.objects.filter(...)
> for lesson_subject in 
> Lesson.subjects.through.objects.filter(lesson__in=lessons, 
> subject__in=subject).select_related('subject').iterator():
> prefetched[lesson_subject.lesson_id].append(lesson_subject.subject)

I hacked up a quick solution that works for my limited requirements. No chained 
querysets, limited prefetch recursion, no order_by, extras etc. But maybe 
someone finds it useful: 
https://gist.github.com/ecederstrand/6748a3496acdc95e40ef

At least it speeds up my queries exponentially; 10-50x improvement in my tests 
so far.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/6E808964-026D-40B4-BD59-52E13EC4CFF2%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Prefetch() with through models

[Erik Cederstrand]

> Den 17/09/2015 kl. 09.22 skrev Javier Guerra Giraldez <jav...@guerrag.com>:
> 
> On Thu, Sep 17, 2015 at 2:07 AM, Erik Cederstrand
> <erik+li...@cederstrand.dk> wrote:
>>> Den 16/09/2015 kl. 16.45 skrev Mike Dewhirst <mi...@dewhirst.com.au>:
>>> 
>>> On 16/09/2015 9:53 AM, Erik Cederstrand wrote:
>>>> issues because the prefetch query does something along the lines of
>>>> "SELECT ... FROM lesson_subjects WHERE lesson_id IN
>>>> [insane_list_of_lesson_ids]".
>>> 
>>> I'm no expert so I'm wondering if len([insane_list_of_lesson_ids]) == 
>>> "hundreds of thousands"?
>> 
>> In my, case, yes. I think the backend might process the query in chunks if 
>> the length of the SQL exceeds the max SQL query size.
> 
> 
> Just curious, which RDBMS are you using?  I remember that on MySQL
> there used to be an advice to populate a temporary table and do a JOIN
> instead of very big `xxx IN ()` statements.  I'm not sure if
> there's a hard limit, but anything over a few hundreds would be better
> with JOIN than with IN(...)

I'm using PostgreSQL and yes, I'm sure there are techniques to work around huge 
IN clauses. It just seems plain wrong to send 1-2MB of textual SQL to an SQL 
server when my original filters should work just fine, and hopefully much 
faster.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/6FC97256-8DC7-4E68-ACDA-FCD459F5D519%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Prefetch() with through models

[Erik Cederstrand]

> Den 16/09/2015 kl. 16.45 skrev Mike Dewhirst <mi...@dewhirst.com.au>:
> 
> On 16/09/2015 9:53 AM, Erik Cederstrand wrote:
>> Hi folks,
>> 
>> I'm working on a school timetable app. I want to fetch hundreds of
>> thousands of Lesson instances with prefetched m2m relations (e.g.
>> subjects). My m2m relations use through models (I'm not sure this
>> actually makes a difference here), and I'm running into performance
>> issues because the prefetch query does something along the lines of
>> "SELECT ... FROM lesson_subjects WHERE lesson_id IN
>> [insane_list_of_lesson_ids]".
> 
> I'm no expert so I'm wondering if len([insane_list_of_lesson_ids]) == 
> "hundreds of thousands"?

In my, case, yes. I think the backend might process the query in chunks if the 
length of the SQL exceeds the max SQL query size.

I've had a look at the prefetch code in Django 1.8. The prefetcher is designed 
to kick in *after* the QuerySet has constructed the list of objects. It only 
has access to the resulting list of items, not the original SQL filters, so the 
only sane way to fetch related objects is to use an IN clause. I can't see any 
way to replace this with the filters without rewriting the whole prefetch code.

I'll try the route Simon suggested and run the prefetch queries myself. I can 
probably populate the _prefetched_objects_cache on each object so the 
optimization is reasonably transparent to consumers. Prefetching 'foo__bar' 
this way gets more complicated, so I think I'll go with a simple solution first.

> I'm not much help but I do remember many years ago a school timetable 
> programming guru who told me they are definitely not trivial. I think he was 
> referring to timetable creation given enrolments, student preferences, 
> curriculum/course requirements, availability of teachers and location of 
> campuses. One of the main issues (ISTR) was memory - or lack of it.

Constraints solvers to calculate the optimal timetable given a set of (soft or 
hard) requirements and costs is mathematically well understood but insanely 
difficult to get right in practice. Lack of memory and the limited life span of 
human beings are just some of the issues. Luckily, I'm just working on the 
output of that calculation :-)

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/69D8C444-936A-46CA-904A-FC9B4C0CA39D%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Prefetch() with through models

[Erik Cederstrand]

Hi folks,

I'm working on a school timetable app. I want to fetch hundreds of thousands of 
Lesson instances with prefetched m2m relations (e.g. subjects). My m2m 
relations use through models (I'm not sure this actually makes a difference 
here), and I'm running into performance issues because the prefetch query does 
something along the lines of "SELECT ... FROM lesson_subjects WHERE lesson_id 
IN [insane_list_of_lesson_ids]".

The initial query on Lesson uses a properly indexed filter on e.g. dates, so I 
thought I'd try to use the same filter to get the related Subjects via the 
relation to Lessons:

qs = Subject.objects.filter(lessons__school_id=8, 
lessons__start__lt=datetime(2015, 8, 1))
Lesson.objects.filter(school_id=8, start__lt=datetime(2015, 8, 1))\
  .prefetch_related(Prefetch('subjects', queryset=qs))

I can see that the extra filters are added to the prefetch query, but the huge 
IN clause is still there.

Am I using Prefetch() wrong? Are there any other techniques to avoid the huge 
IN clause in prefetch queries?

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3A748402-9FAC-4670-BDD0-140CD4F080A4%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Strip ArrayField input from admin

[Erik Cederstrand]

> Den 18/08/2015 kl. 10.54 skrev Erik Cederstrand <erik+li...@cederstrand.dk>:
> 
> Hi list
> 
> I have a model with a field defined like this:
> 
>   categories = ArrayField(models.CharField(max_length=32))
> 
> 
> This field is accessible via the admin interface, and when I enter 
> comma-separated values in the charfield, they are stored as an array in the 
> DB.
> 
> However, I just discovered that if I enter comma+space separated values, e.g. 
> "teachers, locations, students", it gets converted to ['teachers', ' 
> locations', ' students'] which is not what I want. How go I get the 
> ArrayField to strip the values after splitting by comma?

Well, that was easy. Just subclass ArrayField:

   class StrippedArrayField(ArrayField):
   def pre_save(self, model_instance, add):
   stripped_array = [s.strip() for s in getattr(model_instance, 
self.attname)]
   setattr(model_instance, self.attname, stripped_array)
   return stripped_array


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/775AFBB7-B637-4412-973D-A9EC95323D87%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Strip ArrayField input from admin

[Erik Cederstrand]

Hi list

I have a model with a field defined like this:

   categories = ArrayField(models.CharField(max_length=32))


This field is accessible via the admin interface, and when I enter 
comma-separated values in the charfield, they are stored as an array in the DB.

However, I just discovered that if I enter comma+space separated values, e.g. 
"teachers, locations, students", it gets converted to ['teachers', ' 
locations', ' students'] which is not what I want. How go I get the ArrayField 
to strip the values after splitting by comma?

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5CFD8C84-1159-411E-B375-77FB835B9048%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: bulk add m2m relationship for multiple instances

[Erik Cederstrand]

> Den 13/08/2015 kl. 04.09 skrev yakkades...@gmail.com:
> 
> I'll run a test with the dict vs list+position counter. I know I saw a speed 
> improvement but I can't remember if that was the only thing I changed. 
> 
> I'd have to change a lot of code if I change the DB scheme so I'm not wanting 
> to create an intermediate table. I'm going to go down the SQL path.

The intermediate model doesn't change the DB schema in your case. A 
models.ManyToManyField already implicitly creates a table in the DB to hold the 
m2m relation. The intermediate model just makes this explicit.

The only thing this changes in your code is that you can't do 
"my_datapoint.sensors.add(my_sensor)" anymore. You need to always create (and 
delete) a DatapointSensorRel explicitly.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ABFA88FA-3D1F-42F8-A634-9E04C0C6DCD3%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: bulk add m2m relationship for multiple instances

[Erik Cederstrand]

> Den 12/08/2015 kl. 20.00 skrev yakkades...@gmail.com:
> 
> In the actually code I create and preload all the DataPoints and Sensors 
> outside the loop.  I found a dict was too slow for DataPoints.

That's suspicious. Compared to loading data from the database, Python dicts are 
not slow, for any reasonable value of slow.

> Can you explain what you mean by “If you need better bulk-insert performance 
> than this, you can convert the m2m relation between Sensor and DataPoint to 
> an explicit m2m model. You can then bulk-insert all m2m relations in one go 
> instead of per-object”?   I’m not sure how to implement this.

Create an intermediate model for the m2m relation as described in 
https://docs.djangoproject.com/en/1.8/topics/db/models/#extra-fields-on-many-to-many-relationships

You can then bulk_create() on this model instead of using add(). Something like:


  class Datapoint(models.Model):
  sensors = models.ManyToManyField(Sensor, through='DatapointSensorRel')


  class DatapointSensorRel(models.Model):
  datapoint = models.ForeignKey(Datapoint)
  sensor = models.ForeignKey(Sensor)


Used like this:

  relations = [DatapointSensorRel(datapoint=d, sensor=s) for d, s in 
my_collected_relations]
  DatapointSensorRel.objects.bulk_create(relations)


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/E9071BDE-2AD3-4D77-8D9F-0ABD1791CDDB%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: bulk add m2m relationship for multiple instances

[Erik Cederstrand]

> Den 12/08/2015 kl. 04.47 skrev yakkades...@gmail.com:
> 
> for row in rows:
> dp = DataPoint.objects.get(Taken_datetime=row['date']) 
> 
> sensorToAdd = []
> for sensor in sensors:
> s = Sensor.objects.get(Name=sensor.name, Value=sensor.value  )
> sensorToAdd.append( s )
> 
> dp.Sensors.add( sensorToAdd )
> 
> In the actually app I bulk create all the DataPoint and Sensor instances.  
> The problem is that |dp.Sensors.add( sensorToAdd )| does a lot of hits on the 
> db.  I want a way bulk add all the sensors.  

Try fetching all the data you need from the database up-front and place objects 
in a dict. Depending on the volume of your data, you may want to split this 
into reasonable batches:

  from django.db.models import Q

  datapoints = DataPoint.objects.filter(Taken_datetime__in={r['date'] for r in 
rows})
  datapoints_map = {(d.Taken_datetime, d) for d in datapoints}

  # To generate efficient SQL, make sure (name, value) pairs are unique
  unique_sensor_values = {(s.name, s.value) for s in my_list_of_sensors}
  sensors_q = Q()
  for name, value in my_unique_sensor_values:
  sensors |= Q(Name=name, Value=value) 
  sensors = Sensor.objects.filter(sensors_q)
  sensors_map = {((s.name, s.value), s) for s in sensors}

This reduces your queries to only two. You can then bulk-insert m2m relations 
per-object like this:

   for some_date, some_sensors in my_data:
   dp = datapoints_map[some_date]
   dp.Sensors.add(*[sensors_map[(s.name, s.value)] for s in some_sensors])


If you need better bulk-insert performance than this, you can convert the m2m 
relation between Sensor and DataPoint to an explicit m2m model. You can then 
bulk-insert all m2m relations in one go instead of per-object.

You should probably add indexes on DataPoint.Taken_datetime and Sensors.[Name, 
Value] to increase query performance.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7FCE3968-7C15-4D92-8DB3-E4A6EE56442B%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Django M2M with Oracle database FieldDoesNotExist error

[Erik Cederstrand]

> Den 03/08/2015 kl. 08.50 skrev mohsenba...@gmail.com:
> 
> Hello Django community.
> I have two Django models Foo and Bar that have a  ManyToMany relation:
> 
> Class Foo(models.model):
>   
> ...
> 
>   bars = models.ManyToManyField('Bar', related_name='foos')
> somewhere in the code, i do direct assignment to replace the relation with a 
> new list:
> 
> foo_object.bars = [bar_object_list]

You can't do that, AFAIK. You need to do:

 foo_object.bars.clear()
 foo_object.bars.add(*bar_object_list) 


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/C137F257-7E3B-4C68-98B7-BC540272315C%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: post_delete and determining if a related object has been deleted

[Erik Cederstrand]

> Den 28/07/2015 kl. 14.36 skrev Stefan Schindler :
> 
>> This makes no sense to me. You want to delete an Item or Order but 
>> then immediately create a new one?
> 
> My actual goal is this: Whenever an Item object itself is deleted, I
> want to create a LogEntry object attached to the item's order. If an
> order is deleted however, I don't want to do anything.
> 
> In code, I expected it to look something like this:
> 
>  @receiver(post_delete, sender=Item)
>  def on_item_post_delete(instance, **kwargs):
>if instance.order is not None:
>  LogEntry(order=instance.order).save()
> 
> It's technically impossible (AFAIK) however to determine the case
> between "Item alone is deleted" and "Item is cascaded by Order
> deletion", at the moment.

Signals are nice, but sometimes they just make code more complicated. You could 
go for this (naive) approach:


class Order(models.Model):
def delete(self, *args, **kwargs):
# Detatch this order from its' items
for item in self.items.all():
item.order = None
item.save()
# Or is this what you really wanted?
# item.delete()  
super().delete(*args, **kwargs)


class Item(models.Model):
def delete(self, *args, **kwargs):
if self.order:
LogEntry(order=instance.order).save()
super().delete(*args, **kwargs)


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7983ADA9-024A-4DEC-94B1-EC2D7FEDD7DA%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: post_delete and determining if a related object has been deleted

[Erik Cederstrand]

> Den 28/07/2015 kl. 09.37 skrev Stefan Schindler :
> 
> If the Order object itself is deleted, all post_delete handlers of all
> related objects are fired, including Item. Item however stores a new
> object with a relation to the *deleted Order* object, which will
> result in a constraint error (at least when using transactions in
> PostgreSQL, which I do).

This makes no sense to me. You want to delete an Item or Order but then 
immediately create a new one?

If you want to delete an Order but make sure the Item stays, you should use the 
"on_delete=SET_NULL" argument of ForeignKey fields: 
https://docs.djangoproject.com/en/1.8/ref/models/fields/#django.db.models.ForeignKey.on_delete

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/D25DE259-4267-4010-A683-1F3A745D6FD0%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: System requirements

[Erik Cederstrand]

> Den 27/07/2015 kl. 20.48 skrev Ingo Hohmann :
> 
> Thank you.
> 
> It's the age old problem between management and development. I'd say develop 
> on a small server, and then test how far it can go. But someone wants to know 
> _now_.

Well, then tell them that the waterfall development model died in the 90's.

Or, estimate the impossibly hand-wavy "Expect unexpected changes in several 
ways" at somewhere between 0 and €140 mio yearly.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5DA2ED71-D10A-4643-BAA5-652729A598C4%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: System requirements

[Erik Cederstrand]

> Den 27/07/2015 kl. 07.32 skrev Ingo Hohmann :
> 
> Hi,
> 
> I'm new to django, and I would like to get a hint about system requirements. 
> If you know about any helpful links, these are welcome, too. So far what I 
> dog up by googlng wasn't too helpful.
> 
> Currently we have a
> 
> - Database with mostly static entries in the thousands
> - it should be possible to search the database on different fields / over 
> several fields
> - the data contains images, as well
> 
> And the spec says: Expect unexpected changes in several ways.
> 
> I guess things like voting and commenting will come.
> 
> Maybe later some sort of api access ...
> 
> Do you have a rough idea, what kind of setup will be needed?

There is absolutely no sensible way to answer this question in terms of 
hardware. For all we know, you could be describing the specs for google.com, or 
the intranet for your county church. Everything depends on what your service 
needs to do, how many visitors you have, and how fast they need a response.

My suggestion would be to start with some cheap or free hosting and hack away. 
While writing your software, you should always keep in mind the load you 
expect/dream for, and make sure you are prepared to add whatever solutions are 
necessary to handle increasing load. Prepare yourself by collecting performance 
metrics for your software and your hardware, and define what response times 
your visitors should expect.

Do performance tests before launch with whatever artificial traffic you can 
generate. Hardware requirements are roughly proportional to the number of 
requests. Know where your hot spots are and what you can do to solve them. As 
real-world traffic starts coming in, look at your metrics. To increase 
performance, you can either improve your software or add hardware. The former 
is expensive in salary, the latter is expensive in invoices.


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/F69A1D5E-35EE-427C-9EA2-E6FB0A277523%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: preserve data when migrating ForeignKey to ManyToManyField

[Erik Cederstrand]

> Den 14/07/2015 kl. 23.32 skrev A Lee :
> 
> I'd like to change a ForeignKey field into a ManyToManyField, preserving 
> existing data by creating entries in the many-to-many through table. Running 
> makemigrations after changing the model class creates a migration that 
> executes a RemoveField on the existing ForeignKey field and then an AddField 
> on the new ManyToManyField, which destroys any existing ForeignKey data.
> 
> I ended up implementing this by creating three migrations:
> 
> 1. schema migration: create the new ManyToManyField
> 2. data migration: copy existing ForeignKey data into the ManyToManyField
> 3. schema migration: remove the ForeignKey field and rename the 
> ManyToManyField
> 
> Is there a simpler way to do this type of schema change?

No. AFAIK, this is the recommended and most robust and flexible way of making 
non-trivial changes to a model field definition.

If you have lots of data to migrate, you may need to optimize the data 
migration with e.g. raw SQL instead of the naive:

   for obj in MyModel.objects.all():
  obj.my_m2m.add(obj.my_fk)

Erik


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/52FB9F20-D170-4787-AB16-979008008A1A%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Custom UserChangeForm shows one list correctly and the other doesn't show values

[Erik Cederstrand]

> Den 20/06/2015 kl. 14.20 skrev Néstor Boscán :
> 
> I've created a custom UserChangeForm in DJango 1.7.8 where I have 2 custom 
> ComboBox associated with a Model. Both are defined exactly the same. One 
> shows the values and the + button to add a new entity. The second only shows 
> an empty list.
> 
> Any ideas?

Post your code?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/73F6BAE5-6C05-4C54-936F-644893DE9CBF%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Django 1.8 Transaction Begin

[Erik Cederstrand]

> Den 18/06/2015 kl. 17.49 skrev Carl Meyer :
> 
> I wrote a blog post [2] which discusses this in more detail. It's about
> PostgreSQL instead of MySQL, but the part near the beginning about PEP
> 249 and Django applies equally well to MySQL (I think; I'm not very
> familiar with transactions on MySQL).

Just remember that the only MySQL storage engine that actually supports 
transactions is InnoDB: 
https://dev.mysql.com/doc/refman/5.5/en/storage-engines.html

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8B74519F-19E0-4292-A0C9-4190B1A55CA3%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Squashed migration

[Erik Cederstrand]

> Den 12/06/2015 kl. 11.38 skrev Cherie Pun  >:
> 
> Hi,
> 
> I have trying to experiment with squashmigration to see if it will make it 
> faster to build the database when running tests. So I have squashed the 
> migrations following the instructions on the Django website. However when I 
> run the tests, it still uses the original migrations. I thought Django 
> automatically uses the squashed one over the separated ones. Is there some 
> settings that I have to configure? Also, it says that no optimisation was 
> available even though there are a few AddField which should in theory be 
> combined into AddModel.

squashmigrations has limits to what it can do. You are free to edit the 
migration by hand if you can spot any more optimizations, and file a bug report 
if you spot something that the squasher should reasonably detect.

Have you seen the new "manage.py test --keepdb" in Django 1.8? I have a 
complicated project with lots of apps and use it daily to speed up testing. 
https://docs.djangoproject.com/en/1.8/ref/django-admin/#test-app-or-test-identifier
 


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5968C3C9-1515-4B61-9E9C-964EC60B030E%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Bug in Django? It throws server error when context processor raises Http404

[Erik Cederstrand]

> Den 09/06/2015 kl. 18.54 skrev Alexey Grigoriev :
> 
> I found this issue in my big project. And then I started new Django project 
> from scratch, and reproduced the problem. 
> You can reproduce this bug: 
> 1) start new Django project
> 2) create context processor that raises Http404 
> 3) set Debug=False 
> 4) create custom '404.html' template
> And then you get Server Error Page, instead of 404 page

Please create a bug report and attach the code for your test project. That way, 
it's easier for everyone to reproduce.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/77D00C42-920C-4EBD-95FD-6721F2214FA2%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: heavy refactoring

[Erik Cederstrand]

> Den 09/06/2015 kl. 10.09 skrev Mike Dewhirst :
> 
> Does anyone have any experience with app-renames and maybe some refactoring 
> advice for me?
> 
> Maybe I don't even need to replace and destroy ...

If you wanted, you could set Meta.db_table for all models to the current table 
name. Then you shouldn't need to migrate any data. In your Python code, simply 
rename the directory and everything referencing the app, and you should be good 
to go, unless your code contains too much clever magic.

I can't remember if Django migrations detect table renames. If not, and you 
want to rename the tables too, just issue raw "ALTER TABLE name RENAME TO 
new_name" SQL in an empty migration.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2FED37F9-2951-4CA2-84AC-BA62D3C20010%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Graph theory ??

[Erik Cederstrand]

Hi Rafael,

Assuming you have a static and limited set of actions, and they can only be 
defined programmatically, I would do something like this:


class Vertex(models.Model):
 some_field = models.CharField()


class Edge(models.Model):
from = models.ForeignKey(Vertex, related_name='outgoing')
to = models.ForeignKey(Vertex, related_name='incoming')
action = models.CharField(choices=(('action1', 'action1'), ('action2', 
'action2'), ...))

def action1(self):
# Do something here
pass

def action2(self):
# Do something else here
pass

   def run_action(self):
  return getattr(self, self.action)()


When and where to call run_action() would depend on your requirements - either 
in the save() method on Edge, in a signal, or within a view.

If you were looking to define custom actions on the fly via a GUI, you would 
need to define a DSL containing the constructs you allow, create a view where 
the user can compose an AST using the constructs of the DSL, and store the 
resulting AST in the database. When you need to run the action, you would 
translate the AST to Python and run that.

You can also have a look at 
https://www.djangopackages.com/grids/g/trees-and-graphs/ 
 and see if anything 
suits your needs.

Erik


> Den 02/06/2015 kl. 13.47 skrev Rafael E. Ferrero  >:
> 
> Hello everyone,
> 
> I think to do an app where i would save a directed graph. 
> The vertex and edges there must be dinamics.  
> The vertex can represent some Tag or Category... or something. 
> The edge who links the vertex can be N between two vertex and represent some 
> event or action... and let me explain here what i need to do:
> 
> Suppose you have Vertex1 and Vertex2 and are linked by edge1 from vertex1 to 
> vertex2, edge1 have the action "Add if not exist" and mean that if the user 
> select the vertex1, the system, automatically add vertex2. Even further, 
> suppose you have a third vertex related to vertex1 by edge2 and the action of 
> that edge its "delete if exist" so if the user have, previously, selected 
> vertex3, when select vertex1, vertex3 would be deleted by edge2.
> 
> Make a model to save the graph it's not a problem (save vertex and edges and 
> specify what action have every edge)... my problem its develop every type of 
> event or action for the edges, save that event on database and then execute 
> that event when needed.
> 
> Someone have a recommendation for me?
> THANK YOU ALL!!!
> 
> --
> Rafael E. Ferrero
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-users@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/django-users 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAJJc_8V7RiW%2Bsv4-1x9e9a7Evz4_AetSNZoyAqfoFJApwa6JJQ%40mail.gmail.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/A560D552-8EF4-4A6C-A763-E3613CC3B647%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Custom RelatedManager?

[Erik Cederstrand]

Hi folks,


I have some models with a custom Manager that prefetches certain fields:

class PrefetchManager(models.Manager):
def get_queryset(self):
return super().get_queryset()\
.select_related(*self.model.select_fields)\
.prefetch_related(*self.model.prefetch_fields)


class MyModel(models.Model):
foo = models.ForeignKey(Foo)
bars = models.ManyToManyField(Bar, related_name='mymodels')

select_fields = ('foo',)
prefetch_fields = ('bars')
prefetched = PrefetchManager()


This allows the nice and succinct "MyModel.prefetched.filter(...)" to get model 
instances with prefetched fields. However, I would also like this to work with 
RelatedManager, i.e.:

m = MyModel.objects.get(pk=1)
m.bars.prefetched.filter(...)

to get all Bar connected to m with the prefetched fields I have defined on Bar. 
Is there a way to do this? Can I create a custom RelatedManager for 'bars'? I 
can rewrite it as:

   Bar.prefetched.filter(mymodels=m, ...)

but it feels more awkward.

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9E6B2519-C3BC-4820-B8A2-043E8EC450C2%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Erik Cederstrand]

> Den 15/05/2015 kl. 20.54 skrev Timothy W. Cook :
> 
>  def formfield_for_many_to_many(self, db_field, *args, **kwargs):
> formfield = super(ClusterAdmin, 
> self).formfield_for_many_to_many(db_field, *args, **kwargs)
> if db_field.name in 
> ['cluster','dvboolean','dvuri','dvstring','dvcodedstring','dvidentifier','dvparsable','dvmedia',
>  
> 'dvordinal','dvcount','dvquantity','dvratio','dvtemporal']:
> formfield.queryset = formfield.queryset.select_related('project')
> return formfield

I just noticed you have a typo:

if db_field.name in ['cluster', ...

should be:

if db_field.name in ['clusters', ...

according to your model definition.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ED7D33CB-EE4B-451D-B00F-7B24BFAD7A90%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Erik Cederstrand]

> Den 15/05/2015 kl. 20.54 skrev Timothy W. Cook :
> 
> def formfield_for_many_to_many(self, db_field, *args, **kwargs):
> formfield = super(ClusterAdmin, 
> self).formfield_for_many_to_many(db_field, *args, **kwargs)
> if db_field.name in 
> ['cluster','dvboolean','dvuri','dvstring','dvcodedstring','dvidentifier','dvparsable','dvmedia',
>  
> 'dvordinal','dvcount','dvquantity','dvratio','dvtemporal']:
> formfield.queryset = formfield.queryset.select_related('project')
> return formfield
> 
> 
> 
> ​Each of the ManyToMany references have this in their model:
> 
> 
> ​def __str__(self):
> return self.prj_name.prj_name + ":" + self.data_name

Are you sure you don't mean

   formfield.queryset.select_related('prj_name')

If 'prj_name' is the FK on your m2m models, then that's what should be passed 
to select_related()

Django 1.8 should catch this for you, if 'project' isn't also a FK on your 
model.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/D28F7970-28EB-46DE-9FD2-3BD3F63A085E%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Erik Cederstrand]

> Den 14/05/2015 kl. 22.19 skrev Timothy W. Cook :
> 
> That is exactly the problem Simon.  Everyone of those models reference a 
> model called Project.  I did this so that when the items are displayed in the 
> selects, the user knows which project it is from.  In the interim I guess 
> I'll remove the call to Project from the __str__ 
> 
> I wonder if there is another approach that I can use to solve this?  

Does the suggestion to append select_related() / prefetch_related() to the 
queryset in your admin view not work?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/A395D2BC-D3B1-40C5-84B7-678A070DFCB6%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Ensure an object is saved even when the atomic bock rollsback

[Erik Cederstrand]

> Den 08/05/2015 kl. 15.15 skrev Marc Aymerich :
> 
> HI, 
> 
> I'm using atomic requests, but one of my views needs to save a model 
> regardless of wheter the current transaction rolls back or not.
> 
> I'm confused about how to proceed with the current autocommit behaviour. Do I 
> need to use a different db connection? or perhaps there is some way of 
> telling django to ignore autocommit for some particular block ? 

You need to control your transactions explicitly. Have a look at the docs: 
https://docs.djangoproject.com/en/1.8/topics/db/transactions/#controlling-transactions-explicitly
 Especially the third example code in that paragraph shows how to commit some 
queries while rolling back others.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/93D79E32-B433-4AE8-96B7-90E839982321%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to represent a calendar month as a field in django models

[Erik Cederstrand]

> Den 04/05/2015 kl. 14.21 skrev Erik Cederstrand <erik+li...@cederstrand.dk>:
> 
> class Month(models.Model):
>year = models.IntegerField()
>month = models.PositiveSmallIntegerField()
> 
>def add_months(self, n):
>assert n >= 0
>dy, dm = divmod(self.month + n, 12)
>self.year += dy
>self.month += dm

Sorry, that's supposed to be:

def add_months(self, n):
assert n >= 0
dy, self.month = divmod(self.month + n, 12)
self.year += dy


Disclaimer: This possibly only applies to the Gregorian calendar. Read up on 
the others as needed: http://en.wikipedia.org/wiki/List_of_calendars :-)

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B025BC73-DE58-4E4F-9F55-B94CC990381E%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: How to represent a calendar month as a field in django models

[Erik Cederstrand]

> Den 04/05/2015 kl. 13.26 skrev Matthys Kroon :
> 
> I'm specifically looking at only situations where the year and month alone 
> are significant.
> 
> The downside I see with using a DateField and forcing the day to the first of 
> the month, with custom widget etc. is that somebody looking at the database 
> may not realize that something unusual is going on and I'm not sure how I 
> would go about finding the month x months before or after after a given month 
> ... solutions like
> where 12 * year(t1.month_field) + month(t1.month_field) = 12 * 
> year(t2.month_field) + month(t2.month_field) + x
> come to mind for sql but not sure how this would translate into django. Not 
> that I'm saying it is not the correct solution, all of the options I 
> considered have some downside.

Python stdlib doesn't support operating on months the way I think you expect. 
If you want to truly only operate on years and month values, then I think the 
most Pythonic/Djangoic way is to create your own Month model/class (unless you 
want to marry ourself to PostgreSQL and its' tuple type). The math to 
add/subtract months is very simple:


class Month(models.Model):
year = models.IntegerField()
month = models.PositiveSmallIntegerField()

def add_months(self, n):
assert n >= 0
dy, dm = divmod(self.month + n, 12)
self.year += dy
self.month += dm

# Expand as needed with __add__, __sub__, remove_months() etc.


> The logic for adding and subtracting these "months" requires something like 
> timedelta(month=2), which doesn't exist in the standard library as far as I 
> know - though I'm not an expert.
> 
> There are recipes on stackoverflow for incrementing months though not very 
> elegant.

That's because incrementing months is ambiguous when operating on actual 
datetimes. timedelta(months=2) is nonsense because months have different 
lengths. Adding 1 month to March 2 would mean April 2 to most people, but what 
about January 31 (let's ignore for a moment that both PHP and MySQL accept 
February 31 as a valid date)? Do you want February 28 (or possibly 29), or 
March 2 (or 3), or possibly 4 weeks ahead?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7399E8A3-DC57-4709-B280-65BA11465439%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Erik Cederstrand]

> Den 21/04/2015 kl. 09.20 skrev SHINTO PETER :
> 
> Hi 
> François Schiettecatte , limit memory and CPU usage for python socket client 
> service

Really, if you want qualified help, you need to be more verbose.

Do you want to kill a process violating your limits? Or can the process be 
expected to cooperate, so you can communicate to a process that it needs to cut 
down on memory usage / CPU? Are these hard limits, or can the process use 
available resources if they are available? Do you want to build some sort of 
accounting, like AWS pay-per-use?

Anyway, this is not really a Django or Python problem. It's a hard problem to 
solve in most operating systems, so you may have better luck asking this in a 
forum related to your operating system.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/63B8E65D-E694-4F22-AA89-4732430FD840%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Erik Cederstrand]

> Den 20/04/2015 kl. 19.46 skrev SHINTO PETER :
> 
> Working with linux/unix

If unix includes FreeBSD, check out 
https://www.freebsd.org/doc/en/books/handbook/security-resourcelimits.html

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/452CBB5B-7837-45A8-94B9-0CBBAA62ECAD%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: how to access manager from a model instance?

[Erik Cederstrand]

> Den 25/03/2015 kl. 20.20 skrev felix :
> 
> Yes I know I can't acces the Manager from a model instance, but I need to 
> check the value of a field saved in the database before updating it. Can I do 
> it from the Model clean() method?

Depending on your requirements, there's a couple of options.

Use refresh_from_db() in up-coming Django 1.8 
(https://docs.djangoproject.com/en/dev/ref/models/instances/#django.db.models.Model.refresh_from_db)
or the poor man's version:

   new_obj = MyModel.objects.get(pk=obj.pk)

Remember to audit your use of transactions.

If you always want to update a value, e.g. increment a counter, you can use an 
F() expression: 
https://docs.djangoproject.com/en/1.7/ref/models/queries/#f-expressions

If none of these are appropriate, you need to describe your requirements in 
more detail.


Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/29A8DA8B-9C59-4742-8172-EC119EFD2426%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Re-run only tests that failed last time

[Erik Cederstrand]

> Den 24/03/2015 kl. 11.27 skrev Gergely Polonkai :
> 
> I have a pretty extended test suite for my application, which can run for 
> about 15 minutes. Now when I introduce an error, I will be notified of it 
> pretty late, and a one-liner fix needs another 15 minutes to check. Of course 
> if only one of my tests fail, I can add the name of the test case as a 
> parameter to manage.py test, but in case of many failures it’s pretty hard to 
> do the same.
> 
> Is there a way to run only the tests that failed last time, even if I have to 
> install a separate app/module for that? A quick Google search gave me no 
> usable results.

Only re-running failed tests partly defeats the purpose of testing. Your fix 
for failing test case A may introduce a failure in test case B.

Instead, I would look into either tweaking you very slow test so they run 
faster, or splitting up your test cases into fast and slow tests, so you can 
run your slow tests less frequently depending on some setting.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4A592356-7DB3-44B5-8ACC-332B4811E40D%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Fwd: attaching an app to site via apphook fails. server crashes only (beginner question)

[Erik Cederstrand]

> Start på videresendt besked:
> 
> Dato: 6. mar. 2015 kl. 09.49.01 CET
> Fra: inoyon artlover KLANGRAUSCH 
> Til: django-users@googlegroups.com
> Emne: attaching an app to site via apphook fails. server crashes only 
> (beginner question)
> 
> I created a site and via the django-cms interface. Than edited the site
> via advanced settings: adding the hooked app and defining a 'namespace'
> as reqired.
> 
> result is:
> 
> ImportError at /de/
> 
> No module named 'p'
> Request Method:   GET
> Request URL:  http://127.0.0.1:8000/de/
> Django Version:   1.7.5
> Exception Type:   ImportError
> Exception Value:  
> No module named 'p'


Try starting runserver as "python manage.py runserver --traceback"

That should get you a traceback when you open the URL in your browser, so you 
can see where in your code the ImportError is happening

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/C8E8FD55-0253-4FF5-93F6-D79BBA529FA4%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: django doesnt wait for my time.sleep

[Erik Cederstrand]

> Den 04/03/2015 kl. 18.03 skrev dk :
> 
> i am using matplotlib to generate a plot/graph,  even do that python is 
> generating the file and saving it so I can use it later on in my web page, 
> django show the page before the process finish,
> so I decided to put a time.sleep(3)  so it wait 3 sec while all this happen, 
> but doesn't respect that =(
> 
> any ideas why that might happen or a workaround?  

Please post the relevant code. Without that, it's really just quesswork.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1D63985E-0B2B-4D2E-B25C-8C347349%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Gunicorn sock file is missing?

[Erik Cederstrand]

> Den 26/02/2015 kl. 07.56 skrev Kaloian :
> 
> Hi Erik,
> 
> No it doesn't start at all, running the gunicorn_start script gives: 
> 
> [ERROR] Retrying in 1 second.
> [ERROR] Retrying in 1 second.
> [ERROR] Retrying in 1 second.
> [ERROR] Retrying in 1 second.
> [ERROR] Retrying in 1 second.
> [ERROR] Can't connect to /path/to/my/gunicorn.sock 
> 
> This is why I thought that the missing sock file is failing it to start. Do 
> you think it could be something else? 
> I have already tried to add the absolute path in gunicorn_start but nothing 
> changed. 

A UNIX socket is like a TCP connection, except it's represented as a file. 
gunicorn is supposed to create it on startup and destroy it on shutdown, and it 
shouldn't be present when gunicorn is not running.

Maybe you tried to create /path/to/my/gunicorn.sock yourself, and the file is 
now garbage? Try deleting it. Otherwise, another instance of gunicorn may be 
running already. Try "ps aux | grep gunicorn" or "lsof -U | grep gunicorn" if 
you're on Linux. Also, the user running gunicorn_start must have access to 
create files in /path/to/my/

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/091316F1-9473-4A28-B237-FAA8C371E169%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: TypeError: __str__ returned non-string (type bytes)

[Erik Cederstrand]

> Den 25/02/2015 kl. 05.33 skrev Mike Dewhirst :
> 
> ==
> ERROR: test_checkreference_exp (substance.tests.test_substance.TestSubstance)
> --
> Traceback (most recent call last):
>  File "C:\Users\mike\env\xxdx3\ssds\substance\tests\test_substance.py", line 
> 57, in test_checkreference_exp
>self.assertEqual(subst.checkreference('1'), True)
>  File "C:\Users\mike\env\xxdx3\ssds\substance\models\substance.py", line 449, 
> in checkreference
>if checkit(self, ref):
>  File "C:\Users\mike\env\xxdx3\ssds\substance\models\substance.py", line 441, 
> in checkit
>val = u"{0}".format(obj.__dict__[field])
> TypeError: __str__ returned non-string (type bytes)

"obj.__dict__[field]" returns an object that has a __str__ method (format() 
calls the __str__ method of the object to convert it to a string). The __str__ 
method should only return unicode data but is returning byte strings. What does 
"obj.__dict__[field]" return, and can you show us the __str__ method of that 
class?

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/A41F5598-AC16-411E-8136-52BABB7B3640%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Gunicorn sock file is missing?

[Erik Cederstrand]

> Den 25/02/2015 kl. 07.16 skrev Kaloian :
> 
> I have an ansible provisioned VM based on this one 
> https://github.com/jcalazan/ansible-django-stack but for some reason trying 
> to start Gunicorn gives the following error:
> 
> Can't connect to /path/to/my/gunicorn.sock
> 
> [...]
> 
> Can anyone point me to any direction what could be causing the missing 
> gunicorn.sock file ?

Is gunicorn even running? If not, you ned to find out why it failed to start, 
or why it crashed. As a start, try adding the absolute path to gunicorn in your 
gunicorn_start script instead of just "exec gunicorn".

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/F5273312-7606-45BD-9042-0807A7F030D1%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Logging transaction statements?

[Erik Cederstrand]

> Den 10/02/2015 kl. 23.45 skrev Carl Meyer :
> 
> Yes, that's what I said :-)

Thanks for the great explanations; I didn't see your previous answer before 
posting my latest reply, hence the duplicate explanations :-)
> 
>> My code relies on isolation level "Serializable" for a cache.get()
>> followed by cache.add() to be reliable, but Django uses the default
>> from PostgreSQL.
> 
> I don't think you need full Serializable, Repeatable Read should be
> sufficient. Serializable is dangerous, has a tendency to cause deadlocks
> (as I think you've discovered). Even with Repeatable Read, you'll need
> be a bit careful about any long-running processes that might hold open a
> transaction.

I was thinking Seralizable is needed since cache.get() is a query that could 
result in a phantom read. But since I have multiple long-running management 
jobs and incoming changes from a REST API, I also think I'll bee too vulnerable 
to deadlocks if I change the isolation level.

> But `self.connection.set_isolation_level()` is called in
> `_set_autocommit()`, and that should be sufficient. (The
> `_set_isolation_level()` method is dead code in 1.7 and has since been
> removed, but what really matters is that
> `self.connection.set_isolation_level()` is called.)

It's only called if self.psycopg2_version < (2, 4, 2), at least in the file I'm 
looking at. My version is 2.5.2.

> OPTIONS['autocommit'] has been ignored since Django 1.6, as the docs you
> linked clearly state: "This configuration is ignored and can be safely
> removed."
> 
> You're looking for
> https://docs.djangoproject.com/en/1.7/ref/settings/#autocommit instead.

Right you are :-) I should get some sleep.

> You could switch to Repeatable Read (but leave AUTOCOMMIT on). Or (what
> I would probably do) you could leave the isolation level at the default
> (since it's the default for Django, some parts of Django, such as
> QuerySet.get_or_create(), may not work correctly under a different
> isolation level) and simply update your locking logic to guard against
> that race condition (if the add fails, assume that another process has
> grabbed the lock in the interim).

Yeah, I think I'll do just that. My production code is a bit more complicated 
since it relies on two cache.get()'s before the cache.add(), but that will 
probably just have to go away.

Thanks again for you excellent help,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/C42D1493-F3D4-416E-A6A9-AA78BB589F78%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Logging transaction statements?

[Erik Cederstrand]

Den 10/02/2015 kl. 21.27 skrev Erik Cederstrand <erik+li...@cederstrand.dk>:
> 
> Running this in parallel in two processes on the same machine returns this 
> after a while:
> 
> Process A:
> [...]
> 
> Process B:
> [...]
> Traceback (most recent call last):
>   File "tmp.py", line 30, in 
> assert cache.add(key='foo', value='bar')
> AssertionError
> 
> 
> I don't see how this is possible when I'm using transaction.atomic().

Phew. This situation is actually possible since "Read Committed" is the default 
isolation level in PostgreSQL, which means that non-repeatable reads are 
possible within a transaction.

My code relies on isolation level "Serializable" for a cache.get() followed by 
cache.add() to be reliable, but Django uses the default from PostgreSQL.

The isolation level is supposed to be configurable with the 'isolation_level' 
setting in OPTIONS 
(https://docs.djangoproject.com/en/1.7/ref/databases/#isolation-level). Except 
it doesn't work because 
django/db/backends/postgresql_psycopg2/base.py::_set_isolation_level() is never 
called anywhere, AFAICS.

I tried disabling autocommit (the docs are wrong BTW, 
https://docs.djangoproject.com/en/1.7/ref/databases/#autocommit-mode says to 
put it in OPTIONS, but django/db/backends/__init__.py (line 123) looks at 
settings_dict['AUTOCOMMIT'], not in OPTIONS) and hacking 
postgresql_psycopg2/base.py::_set_autocommit() to call _set_isolation_level(), 
but now process A blocks process B entirely, even when process A is outside the 
atomic() context manager.

Has anyone got isolation_level "Serializable" to work in Django 1.7?


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/12300B76-606A-4977-A402-990D562C7F77%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Logging transaction statements?

[Erik Cederstrand]

Hi Carl,

> Den 10/02/2015 kl. 18.34 skrev Carl Meyer  >:
> 
> I assume you're using the 'db' cache backend? Otherwise, it wouldn't
> make sense to expect transactions to affect cache calls at all.

Yes, I'm using the db backend.

> The difference between the snippets is that you are doing a conditional
> branch based on the return value of the cache add, but your set_val
> helper method always returns None, it doesn't return the return value of
> cache.add().

Wow, *that* was embarrassing :-) You see what you want to see, I guess...

I'm still having issues with this, though. I reduced my own code to the 
following. The intent of the code is to use the cache to implement 
multiprocess/multiserver locking:

from django.db import transaction
from django.core.cache import cache

class CacheError(Exception):
pass

cache.delete('foo')
while True:
print('Trying to cache foo')
try:
with transaction.atomic():
if cache.get('foo'):
raise CacheError()
print('adding foo to cache')
assert cache.add(key='foo', value='bar')
print('foo cached')
time.sleep(random.random())
with transaction.atomic():
if cache.get('foo'):
cache.delete('foo')
except CacheError:
print('Failed to cache foo')
time.sleep(random.random())

Running this in parallel in two processes on the same machine returns this 
after a while:

Process A:
Trying to cache foo
2015-02-10 21:02:25,781 DEBUG(0.001) SELECT cache_key, value, expires FROM 
"dispatch_cache" WHERE cache_key = 'foo'; args=['foo']
adding foo to cache
2015-02-10 21:02:25,782 DEBUG(0.000) SELECT COUNT(*) FROM "dispatch_cache"; 
args=None
2015-02-10 21:02:25,782 DEBUG(0.000) SAVEPOINT "s140735261451008_x1"; 
args=None
2015-02-10 21:02:25,783 DEBUG(0.000) SELECT cache_key, expires FROM 
"dispatch_cache" WHERE cache_key = 'foo'; args=['foo']
2015-02-10 21:02:25,784 DEBUG(0.000) INSERT INTO "dispatch_cache" 
(cache_key, value, expires) VALUES ('foo', 'gASVBwCMA2JhcpQu', 
'-12-31 23:59:59'); args=['foo', 'gASVBwCMA2JhcpQu', '-12-31 
23:59:59']
2015-02-10 21:02:25,784 DEBUG(0.000) RELEASE SAVEPOINT 
"s140735261451008_x1"; args=None
foo cached
2015-02-10 21:02:26,771 DEBUG(0.000) SELECT cache_key, value, expires FROM 
"dispatch_cache" WHERE cache_key = 'foo'; args=['foo']
2015-02-10 21:02:26,772 DEBUG(0.000) DELETE FROM "dispatch_cache" WHERE 
cache_key = 'foo'; args=['foo']


Process B:
Trying to cache foo
2015-02-10 21:02:25,782 DEBUG(0.000) SELECT cache_key, value, expires FROM 
"dispatch_cache" WHERE cache_key = 'foo'; args=['foo']
adding foo to cache
2015-02-10 21:02:25,783 DEBUG(0.000) SELECT COUNT(*) FROM "dispatch_cache"; 
args=None
2015-02-10 21:02:25,784 DEBUG(0.000) SAVEPOINT "s140735261451008_x1"; 
args=None
2015-02-10 21:02:25,784 DEBUG(0.000) SELECT cache_key, expires FROM 
"dispatch_cache" WHERE cache_key = 'foo'; args=['foo']
2015-02-10 21:02:25,791 DEBUG(0.007) INSERT INTO "dispatch_cache" 
(cache_key, value, expires) VALUES ('foo', 'gASVBwCMA2JhcpQu', 
'-12-31 23:59:59'); args=['foo', 'gASVBwCMA2JhcpQu', '-12-31 
23:59:59']
2015-02-10 21:02:25,792 DEBUG(0.000) ROLLBACK TO SAVEPOINT 
"s140735261451008_x1"; args=None
Traceback (most recent call last):
  File "tmp.py", line 30, in 
assert cache.add(key='foo', value='bar')
AssertionError


I don't see how this is possible when I'm using transaction.atomic().


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/FD568626-E3C3-4026-9617-3EDB24F037D8%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Logging transaction statements?

[Erik Cederstrand]

> Den 10/02/2015 kl. 17.15 skrev Erik Cederstrand <erik+li...@cederstrand.dk>:
> 
> Hi list,
> 
> I'm tracking down a bug in my app that shouldn't be happening with the 
> transaction statements I added in my code. In my logging settings, I can set 
> 'django.db.backends' to DEBUG to log the queries (I'm using the postresql 
> backend). I see 'SAVEPOINT' statements logged, but I also need 'BEGIN' and 
> 'COMMIT' so I can see when the transaction started and ended. How do I do 
> that?

Hmm, I'm beginning to think I don't understand Django transactions. I have the 
following two snippets, boiled down from my original code. The first one calls 
cache methods via helper functions, the other one calls the cache methods 
directly. If I run the code in parallel in two different processes, then the 
first example asserts almost instantly, while the second example survives 
several minutes. Can someone explain this?

import random
from django.db import transaction
from django.core.cache import cache

def get_val(key):
cache.get(key)

def set_val(key, val):
cache.add(key, val)

def del_val(key):
cache.delete(key)


# First example, fails
while True:
with transaction.atomic():
if not get_val('foo'):
print('no key found')
time.sleep(random.random())
if set_val('foo', 'bar'):
print('key added')
time.sleep(random.random())
else:
assert False
del_val('foo')
print('key deleted')
time.sleep(random.random())


# Second example, runs correctly
while True:
with transaction.atomic():
if not cache.get('foo'):
print('no key found')
time.sleep(random.random())
if cache.add('foo', 'bar'):
print('key added')
time.sleep(random.random())
else:
assert False
cache.delete('foo')
print('key deleted')
time.sleep(random.random())

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5D7F4F83-FFEB-4658-AF81-B6B94576ADBA%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Logging transaction statements?

[Erik Cederstrand]

Hi list,

I'm tracking down a bug in my app that shouldn't be happening with the 
transaction statements I added in my code. In my logging settings, I can set 
'django.db.backends' to DEBUG to log the queries (I'm using the postresql 
backend). I see 'SAVEPOINT' statements logged, but I also need 'BEGIN' and 
'COMMIT' so I can see when the transaction started and ended. How do I do that?

Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/675FA187-3A66-4F3D-80CE-E6941C1050BC%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Pillow on QPython3

[Erik Cederstrand]

> Den 03/02/2015 kl. 14.36 skrev ADEWALE ADISA :
> 
> Hi guys, am try to setup django tutorial with QPython3 on android device. But 
> the problem is am unable to install Pillow on the Qpython3. Its just giving 
> me various error. I try Qpython forum but with no avail.

Picking a random post here. When asking questions, please:

* Describe what you're trying to achieve, and what you have already tried
* Post the relevant code, if you have code that's not working
* Post the error messages, if you get error messages

These are really simple things that raise the chance you'll get helpful answers 
from this list.


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/D9115517-C421-49D2-B3A7-181064542DF0%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: When do I have to call save() on a Model Instance?

[Erik Cederstrand]

> Den 29/01/2015 kl. 19.40 skrev Tobias Dacoir :
> 
> Thanks for answering all my questions.
> 
> So it's perfectly save to call the save method at a later time? As long as I 
> keep the object and a pointer to it in memory I can freely modify it in 
> several Functions before finally calling save at some point in time?

This depends on your requirements. If there's a chance that another thread has 
modified the same object in the database from the time you fetch the data and 
until you call save(), then it's not necessarily safe. This would depend on 
your use of transactions, whether the data has been submitted from a form etc.

You should audit your code for these things if race conditions are a concern to 
you.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/D46AC3C3-B2A3-4D09-8C9A-370AC0367DFB%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: django select extremely slow on large table

[Erik Cederstrand]

> Den 23/01/2015 kl. 08.19 skrev James Schneider :
> 
> How many results do you typically get back from that query?

There should be only one result, right? Since the "markid" field is defined as 
unique and OP is filtering on that with a single value.

First thing to check is that the actual column in Oracle is also unique, and 
that there is an index on that column alone. AFAIK Django creates an index 
automatically on unique fields.

The cx_oracle client may be cheating by returning when the query completed, but 
before results are actually fetched.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0898444F-6539-4191-B766-BFE248586F37%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Authentication when usernames are not unique

[Erik Cederstrand]

Ok, here's a stripped-down solution.

I ended up creating a new SchoolUser user model with a OneToOne relation to my 
LegacyUser, to keep the LegacyUser model uncluttered. The SchoolUser implements 
all methods from AbstractBaseUser and PermissionsMixin but doesn't inherit from 
them, because I don't want the model fields that they contain.

I also kept the SchoolUser independent from the standard Django User (i.e. no 
AUTH_USER_MODEL='SchoolUser' in settings.py), so I can still create superuser 
accounts for myself and my colleagues, that are not connected to a school user.

Here's the code:


settings.py:
[...]
AUTHENTICATION_BACKENDS = (
'django.contrib.auth.backends.ModelBackend',
'school_auth.backends.SchoolModelBackend',
)


school_auth/backend.py:
from django.contrib.auth.backends import ModelBackend
from .models import SchoolUser, LegacyUser
class SchoolModelBackend(object):
def authenticate(self, school_id=None, username=None, password=None, 
**kwargs):
if LegacyUser.validate(school=school_id, username=username, 
password=password):
# Password hash validation
try:
school_user = SchoolUser.objects.get(user__school=school_id, 
user__name=username)
except SchoolUser.DoesNotExist:
school_user = 
SchoolUser.objects.create_user(school_id=school_id, username=username)
# Annotate the user object with the path of the backend.
school_user.backend = "%s.%s" % (self.__class__.__module__, 
self.__class__.__name__)
return school_user
#
# if LDAP.validate(school=school_id, username=username, 
password=password):
# pass
# if WAYF.validate(school=school_id, username=username, 
password=password):
# pass
return None

def get_group_permissions(self, user_obj, obj=None):
raise NotImplementedError()
def get_all_permissions(self, user_obj, obj=None):
raise NotImplementedError()

def has_perm(self, user_obj, perm, obj=None):
if not user_obj.is_active:
return False
return perm in self.get_all_permissions(user_obj, obj)

def has_module_perms(self, user_obj, app_label):
if not user_obj.is_active:
return False
for perm in self.get_all_permissions(user_obj):
if perm[:perm.index('.')] == app_label:
return True
return False

def get_user(self, user_id):
try:
return SchoolUser.objects.get(pk=user_id)
except SchoolUser.DoesNotExist:
return None


school_auth/forms.py:
from django.contrib.auth.forms import AuthenticationForm, PasswordResetForm, 
PasswordChangeForm
from django import forms
from django.utils.translation import ugettext_lazy as _
from django.utils.text import capfirst
from .models import LegacyUser, School, SchoolUser
from .backends import SchoolModelBackend
class SchoolAuthenticationForm(AuthenticationForm):
school = forms.ModelChoiceField(queryset=School.objects.active(), 
empty_label=_('Please select a school'))
username = forms.CharField(max_length=40)
password = forms.CharField(label=_("Password"), widget=forms.PasswordInput)

class Meta:
model = LegacyUser
fields = ['school', 'name', 'password']

def __init__(self, request=None, *args, **kwargs):
"""
The 'request' parameter is set for custom auth use by subclasses.
The form data comes in via the standard 'data' kwarg.
"""
self.request = request
self.user_cache = None
super().__init__(*args, **kwargs)

# Set the label for the "username" field.
self.username_field = 
LegacyUser._meta.get_field(SchoolUser.USERNAME_FIELD)
if self.fields['username'].label is None:
self.fields['username'].label = 
capfirst(self.username_field.verbose_name)

def clean(self):
school = self.cleaned_data.get('school')
username = self.cleaned_data.get('username')
password = self.cleaned_data.get('password')

if school and username and password:
self.user_cache = 
SchoolModelBackend().authenticate(school_id=school.pk, username=username,
  
password=password)
if self.user_cache is None:
raise forms.ValidationError(
self.error_messages['invalid_login'],
code='invalid_login',
params={'username': self.username_field.verbose_name},
)
else:
self.confirm_login_allowed(self.user_cache)

return self.cleaned_data


school_auth/models.py:
from django.contrib.auth.models import PermissionsMixin, BaseUserManager, 
AbstractBaseUser
from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils import timezone
from django.utils.crypto import 

Re: Authentication when usernames are not unique

[Erik Cederstrand]

Hi guys,

Thanks for a lot of useful answers! My schools use a palette of authentication 
systems; regular hashed password check with a hash I have access to, LDAP auth 
and WAYF (a Danish educational SSO solution using Shibboleth). Those are the 
least of my worries right now, though.

I'll have a look at the AbstractBaseUser and friends. Ideally, I'd like to 
expand on my existing LegacyUser model and avoid creating separate Django users 
that shadow the LegacyUser, as I do a lot of synchronization of LegacyUsers 
with the legacy system. I'll see how it goes and post a solution if I get that 
far.

Erik

> Den 19/01/2015 kl. 23.24 skrev James Schneider :
> 
> For a pure authentication scenario where permission checks never go beyond 
> user.is_authenticated(), that's probably true. If all the OP is doing is 
> displaying data, they may be able to get away with manually associating the 
> campus and user within the session after, and displaying data based on those 
> session keys. Basically you would end up with a boolean layer of protection 
> for each resource, because all you know is the validated username and campus 
> pair. That may work just fine.
> 
> However, if you need any sort of authorization (permission checking) within 
> the app using Django's permission system, you'll probably need a local copy 
> of the user using a custom user model in the database to perform checks 
> against. It sounds like the OP may need that. Otherwise you are also looking 
> at rolling a custom authorization backend as well.
> 
> If they are LDAP services, you can look at django-ldap, which works quite 
> nicely, including group membership restrictions. It also does the overriding 
> of the authentication backend for you. Not sure how it would work with 
> multiple LDAP servers for various campuses though. That would need some 
> research.
> 
> TL;DR; There are a lot of ways to slice this problem, and a primary strategy 
> driver will be the available authentication backends at each campus. 
> Hopefully they are all the same.
> 
> -James
> 
> 
> On Mon, Jan 19, 2015 at 2:06 PM, Stephen J. Butler  
> wrote:
> Shibboleth 2.0 lets you setup a discovery service (or portal would
> perhaps be a better term) letting the user select which ID Provider
> (IdP) they will authenticate to. All you have to do on the Service
> Provider (SP) side is specify the discovery URL and what IdPs you
> allow. Nothing needs to be done in your Django app except support
> Shibboleth.
> 
> Of course, this is all predicated on there being a competent
> Shibboleth setup at your institutions.
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAD4ANxVp4C2QcxAcY6Xui1bc6Z-hcV--sfOSEy%3DmcaW_w%2BpGHw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXW3NoDaTTpaiL2qtD69vkjmSEFfGSeM9-Dk00YMAG6NQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/81081481-BDD7-4247-BDA9-FBD9DDDF7BA7%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Authentication when usernames are not unique

[Erik Cederstrand]

Hello

I'm creating a Django frontend for a legacy school system. The legacy system 
has users, but usernames are only unique together with a school:

class LegacyUser(models.Model):
   school = models.ForeignKey(School)
   username = models.CharField(max_length=40)

   class Meta:
   unique_together = ['school', 'username']


I need to authenticate users using their school name, username and password, so 
I can serve them data connected to the LegacyUser. The legacy system provides 
an authentication service that I want to use to verify the password.

The Django authentication model seems to revolve around the username being 
unique, so I can't just inherit the User model, login forms etc. How do I get 
the School shoe-horned into the Django auth framework, and where do I call the 
external authentication service? Some ideas how to best accomplish this would 
be great!


Thanks,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/26BA41BB-1771-4C5C-9980-A2C49F30280C%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Lock Django DB on 9 of 10 concurrent uwsgi workers - how to?

[Erik Cederstrand]

> Den 19/01/2015 kl. 19.24 skrev Andreas Krueger :
> 
> 
> 
> What is the most elegant way to
> 
> lock the Django DB while I make a complex transaction (read, decide, write)
> 
> ... during which no other uwsgi worker should have access (or at least no 
> write access) to that table?

That's a rather strange requirement. What are you trying to achive? Elegant for 
who?

You could put your site in "maintenance mode" while the process is running, to 
be nice to users. If you mean elegant (as in simple) sysadmin-wise, simply shut 
down uwsgi and run your commands in a custom management command while uwsgi is 
stopped.

But if you can bundle all your tasks in a single DB transaction, you shouldn't 
need to shut down anything.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/036DAEA0-9F6F-4AD3-9A3C-03D5DC14C00A%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: installation of mysqlclient

[Erik Cederstrand]

> Den 19/01/2015 kl. 20.28 skrev th.gran...@free.fr:
> Thanks but i have a problem!
> 
> when i launch the install command i get these errors
> 
> Downloading mysqlclient-1.3.4.tar.gz (77kB)
>100% || 77kB 356kB/s
>/bin/sh: 1: mysql_config: not found
>Traceback (most recent call last):
>  File "", line 20, in 
>  File "/tmp/pip-build-obp7bna6/mysqlclient/setup.py", line 17, in 
>metadata, options = get_config()
>  File "/tmp/pip-build-obp7bna6/mysqlclient/setup_posix.py", line 47, in 
> get_config
>libs = mysql_config("libs_r")
>  File "/tmp/pip-build-obp7bna6/mysqlclient/setup_posix.py", line 29, in 
> mysql_config
>raise EnvironmentError("%s not found" % (mysql_config.path,))
>OSError: mysql_config not found
>Complete output from command python setup.py egg_info:
>/bin/sh: 1: mysql_config: not found

mysqlclient needs mysql_config, which either isn't installed on your system or 
isn't in your PATH. If this is a Linux distro, it's probably missing the 
libmysqlclient-dev package.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5A250D88-352E-4748-B806-3053E06ADEBF%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: installation of mysqlclient

[Erik Cederstrand]

> Den 19/01/2015 kl. 19.55 skrev th.gran...@free.fr:
> 
> Hello
> 
> i am trying to use Dkango with Python3.4.2 and mysql
> 
> i have downloaded mysqlclient-1.3.4 but i don't know how to install it. it's 
> a .whl file
> 
> Can you help me please?

Just use pip instead. This should get you the latest version:

% pip install mysqlclient

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/C95736CF-C519-4DD8-B21B-0CE34F9E3DBB%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: filter

[Erik Cederstrand]

You're overwriting one QuerySet with another:

new_leave =newleave.objects.filter(department_head_authorization="Approved" )
new_leave = newleave.objects.filter(department="FMKD")

Instead, do this:

new_leave = newleave.objects.filter(department_head_authorization="Approved", 
department="FMKD")


Erik


> Den 16/01/2015 kl. 04.01 skrev sum abiut :
> 
> i expect to see only the data that have 
> department_head_authorization="Approved" and department="FMKD" showing but 
> instead the data that the department_head_authorization is not Approved is 
> also showing.
> 
> 
> 
> On Fri, Jan 16, 2015 at 11:58 AM, Vijay Khemlani  wrote:
> What are you expecting to see? What are you actually seeing?
> 
> On Thu, Jan 15, 2015 at 9:48 PM, sum abiut  wrote:
> Ok thanks,
> 
> i just released that i have made a mistake. so basically here are my 
> template.html and view.py
> i think there is a mistake in my view.py the display result is not what i was 
> looking for. can't seem to figure it out.
> 
> template.html
> 
> 
> 
> Select to approve leave
>   First Name
>   Last Name
>   Position
>   Department
>   Leave Type
>   Details
>   Start Date
>   End Date
>   Total working days
>   # of leave left
>Department Head Authorization
>   Authorize By
>Remarks
>   Authorized Date
>   
> 
> {%for a in new_leave%}
> 
>  onClick="parent.location='#'" >  
> {{a.first_name}}
>   {{a.last_name}}
>   {{a.position}}
>   {{a.department}}
> {{a.leave_type}}
>   {{a.specify_details}}
>   {{a.start_date}}
>   {{a.end_date}}
>   {{a.total_working_days}}
>   {{a.total_Leave_Left}}
>{{a.department_head_authorization}}
> {{a.authorized_by}}
>  {{a.remarks}}
>  {{a.authorization_date}}
> 
>   
> {%endfor%}
> 
> 
> 
> 
> view.py
> 
> 
> def FMKD1_leave_to_authorize(request):
> new_leave 
> =newleave.objects.filter(department_head_authorization="Approved" )
> new_leave = newleave.objects.filter(department="FMKD")
> if new_leave.exists() and new_leave.exists():
> return render_to_response('FMKD1_display_approve_leave.html', 
> locals()) 
> 
> 
> 
> 
> 
> 
> On Fri, Jan 16, 2015 at 10:52 AM, Vijay Khemlani  wrote:
> in your view new_leave and a are QuerySet objects, and then you are comparing 
> them to a string ("True") not a bolean (True without quotes), so it's always 
> False.
> 
> Even if you change "True" to True it won't work, try it like this
> 
> if new_leave.exists() and a.exists():
>   return ...
> 
> 
> On Thu, Jan 15, 2015 at 8:40 PM, sum abiut  wrote:
> 
> 
> Hi,
> I am trying to two column and display some result if two conditions are meet 
> but i am getting this error: 
> The view eLeave.views.FMKD1_leave_to_authorize didn't return an HttpResponse 
> object. It returned None instead.
> 
> I can seems to figure out the issue, here is my view.py file
> 
> def FMKD1_leave_to_authorize(request):
> new_leave 
> =newleave.objects.filter(department_head_authorization="Approved" )
> a = newleave.objects.filter(department="FMKD")
> if new_leave=="True"and a =="True":
> return render_to_response('FMKD1_display_approve_leave.html', 
> locals())
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAPCf-y7uRsM8Vx6Jj5QD4ZY%2BCk24SkgCHZv-xMGZr49icPrmrA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CALn3ei0QaQbwb%3DvkGJ6%2B4DB19p2ZDjTzJb1fSmoAhWUGg_H5PA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAPCf-y6MXc2tR5wBBv9358TMoyBV5K5Ay8vHOSji104fBz1Ovg%40mail.gmail.com.

Re: [HELP] Custom 2 step authentication where the user's password is a public key

[Erik Cederstrand]

> Den 09/01/2015 kl. 12.28 skrev Alon Muroch :
> 
> Hey everyone, i've been thinking of implementing the following custom 
> authentication scheme:
>   • User generates a public and private key pair
>   • when creating a new user, the user name is as usual but the password 
> is the public key (in clear hex)
>   • For login:
>   • the user asks the server to generate a challenge string
>   • the user signs the challenge string and passes it to the 
> server
>   • the user is considered logged in if the returned signed 
> challenge can be verified by the server.
> How i propose to do that: The user sends a GET request for the server which 
> returns a randomly generated challenge and saves it in relation to the 
> requesting user. The user then sends a login request, with the difference 
> that the password param is the signed challenge. 
> Problems with what i propose: How do i verify that who ever requests to 
> generate the challenge is the actual user ?

This looks an awful lot like TLS Client Authentication 
(http://en.wikipedia.org/wiki/Transport_Layer_Security#Client-authenticated_TLS_handshake).

If you're serious about authenticating your clients, I'd suggest looking at 
that instead of rolling your own. Most browsers can handle the client 
certificate securely and automatically, the support is well-tested and there 
are tools for key management, certificate revocation etc.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/36DCC049-E0C6-43BD-8166-CA398A8D1BC1%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with for and if

[Erik Cederstrand]

> Den 05/01/2015 kl. 22.45 skrev Dariusz Mysior :
> 
> I try develop this code to get a number of position this letter in word, and 
> I have it but for example if i search letter "t" I get "1, 5, b", why I get b 
> instead of 11???
> 
> def gen2():
>count=0
>count2=0
>for a in zmienna:
>count2+=1
>if a==szukana:
>count+=1
>pozycja.append(count2)
> 
>return pozycja
> print("3. Literka '",szukana,"' w słowie ",zmienna,
>  "wystąpiła " ,gen(),"razy w kolejności")
> #print(gen2())
> 
> print(", ".join(["%x" % a for a in gen2()]))

Whoa, that code is really convoluted. It won't even compile, as far as I can 
see. Try this instead:

import re
def indices(needle, haystack):
return [match.start() for match in re.finditer(re.escape(needle), 
haystack)]


If your search strings can be overlapping, you'll need another solution.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4551A371-1AAA-405C-BC30-2C7B3044D88B%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Thinking about EAV database model for flexibility. How (in)compatible is it with the django model?

[Erik Cederstrand]

> Den 22/12/2014 kl. 11.27 skrev Felipe Faraggi :
> 
> I'd like to 're-open' this question to ask another (maybe) short one:
> 
> Therefore, is django not very suitable for NOSQL databases like mongo or 
> couch or others in general?
> Or is the problem specifically using RDBMS in a NoSQL manner?

I haven't used Django with NoSQL, so I can't answer that specific question. The 
Django wiki has a page on NoSQL: 
https://code.djangoproject.com/wiki/NoSqlSupport

Which "problem" you are referring to? If you mean using an EAV in Django then 
the problem is not in Django as such, but rather that an EAV usually ends up 
being a miserable way of storing your data, regardless of which software you 
choose to implement it with.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7888B586-E853-4B5D-A325-34CF166CF1D2%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Raw access to cache table

[Erik Cederstrand]

Hi Russel,

> Den 22/12/2014 kl. 00.40 skrev Russell Keith-Magee :
> 
> If you *do* want to do complex queries on the database cache table, the 
> approach suggested by Collin is as good an approach as any. A managed table 
> will give you ORM operations over an arbitrary table - include Django's own 
> internal tables.
> 
> That said, I'll also concur that the database cache backend is the wrong 
> answer here. If you're writing a cron script, the approach I've always used 
> is PIDfile based lock

Thanks for taking the time to explain the design considerations for the cache 
mechanism.

I agree that using the cache to hold locks is a bit awkward, but DB 
transactions would be running way too long for my taste, and I can't use 
pidfile locks because my dataset can get updates from both cron jobs and my 
REST API. But I should probably take a hard look at my design again.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0D2D0B45-A5A7-454D-8173-5F71566AB3C8%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.