Massive import in Django database

[John Carlo]

Hello everybody,

I've fallen in love with Django two years ago and I've been using it for my 
job projects. In the past I found very useful information in this group, so 
a big thank you guys!

I have a little doubt.
I have to import in Django db (sqlite for local development, mySql on the 
server) about 1.000.000 xml documents.

The model class is the following:

class Doc(models.Model):
doc_code =  models.CharField(max_length=20, unique=True, 
primary_key=True, db_index = True) 
doc_text = models.TextField(null=True, blank=True) 
related_doc= models.ManyToManyField('self', null=True, blank=True, 
db_index = True) 

>From what I know bulk insertion is not possibile because I have a 
ManyToManyField relation.

So I have this simple loop (in pseudo code)

for each xml:
   extract from the xml  date-> mydoc_code, mydoc_text, myRelated_doc_codes

   myDoc = Doc.object.get_or_create(doc_code = mydoc_code)[0]
   myDoc.doc_text = mydoc_text
   
   for reldoc_code in myRelated_doc_codes:
myRelDoc =  Doc.object.get_or_create(doc_code = reldoc_code )[0]
myDoc.related_doc.add(myRelDoc )

  myDoc.save()


I'm doing it right? Do you have some suggestions, recommendation? I fear 
that since I have 1.000.000 docs to import, it will take a lt of time, 
especially during the get_or_create routines

thank you in advance everybody!

John




 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5b88deaf-d806-4a64-9e8d-528d95599c80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: onetoone field and memory issue

[John Carlo]

great, now it works like a sharm Thank you very very much!


2013/11/26 James Bennett 

>
> https://docs.djangoproject.com/en/1.6/ref/contrib/admin/#django.contrib.admin.ModelAdmin.raw_id_fields
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-users/IZbXwisADwo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAL13Cg-gtiSsD8%2BS9AcFZKnony5BdD6SsRMZagWY759kHKaDjw%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAMo%2BLG%3Dt7a7KX_hetWCH-2XUbk9sGVhNALRx3-6_%2B0t%3DZmiuGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

onetoone field and memory issue

[John Carlo]

Hello everybody,

I have a memory issue and I'm looking for some suggestions.

I have the following model

class Entity(models.Model):
   
item= models.OneToOneField(Bigdata, blank = True,null=True)


Bigdata is a table with 150.000+ items
In the admin interface, when I try to add a new Entity item, I have a big 
memory increment, I think because django load all the Bigdata table in the 
dropdown list of the fied item (I'm pretty sure, because if the table 
Bigdata is small I have no problem).
So the problem seems the filling of the Bigdata dropdown list, that loads 
all the items at once.

How can I deal with this situation? Pagination of Bigdata? Autocomplete 
instead dropdown list? Any feedback will be very appreciated.
thank you  everyone!

Carlo
  


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/83b025d3-1ce1-4d9b-964f-1caa173d2132%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Sessions memory issue

[John Carlo]

Hello Francois, 

thank you very much for your reply. Not it's all clear.

What about using shelve?

import shelve

db = shelve.open("database", "c")
db["one"] = 1
db["two"] = 2
db["three"] = 3
db.close()

db = shelve.open("database", "r")
for key in db.keys():
print repr(key), repr(db[key])


regards


Il giorno giovedì 24 ottobre 2013 23:10:43 UTC+2, François Schiettecatte ha 
scritto:
>
> John 
>
> There are a couple of ways you can handle this, either you store the files 
> in a database as a TEXT blob, or as a temporary file somewhere. 
>
> And you can identify your users with request.user if they have to have an 
> account, or request.session.session_key if they don't, the session_key is 
> the cookie. For either to work the client has to accept cookies. 
>
> The temporary file approach will required a database table to link the 
> file name to the user. 
>
> I have used all of these and they all work well. 
>
> If you need the text to be persistent across sessions I would store it in 
> a database, if it is around for an hour then just store it in a temporary 
> file. 
>
> And make sure you have a process to delete old data. 
>
> Finally you could also compress the text when you save it. 
>
> Hopes this helps. 
>
> François 
>
>
> On Oct 24, 2013, at 3:51 PM, John Carlo <johncar...@gmail.com> 
> wrote: 
>
> > Hello everybody, 
> > 
> > I'm a newbie with Django, I love it but something it's not clear to me. 
> So I'm here to make a question. Thank you in advance. 
> > 
> > I have an application that has some istances of custom classes I wrote. 
> At every client request, every istance creates a big list of strings. Then, 
> a function combines the lists generated from the istances and send them to 
> the client. 
> > I store the istances in the session, and the combining function get the 
> istances through session. The problem is that the lists of strings comsume 
> a  lot of memory... 
> > I know that sessions are stored in database, I can see them in the table 
> django_session, but they are kept also in RAM, and this is my problem. 
> > How can I reduce the RAM memory consumption? 
> > 
> > I thought 2 ways: 
> > 1) Find a way to move the lists from the RAM and put them in the db, I 
> hope there is something built-in in django session, but I did not find them 
> > 2) Instead building the lists of strings, I create a temp file in which 
> I will append every string. In the end the combining function reads the 
> temp files and combines them 
> > 
> > Could you please help me? I'm really I'm really stuck on this... 
> > 
> > thank you very much! 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-users...@googlegroups.com . 
> > To post to this group, send email to 
> > django...@googlegroups.com. 
>
> > Visit this group at http://groups.google.com/group/django-users. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/c7118798-84fe-4a32-bba0-53452871b6ae%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/groups/opt_out. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/cbb30ae1-d903-4028-b8fa-7388c55d24d6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sessions memory issue

[John Carlo]

Hello everybody,

I'm a newbie with Django, I love it but something it's not clear to me. So 
I'm here to make a question. Thank you in advance.

I have an application that has some istances of custom classes I wrote. At 
every client request, every istance creates a big list of strings. Then, a 
function combines the lists generated from the istances and send them to 
the client.
I store the istances in the session, and the combining function get the 
istances through session. The problem is that the lists of strings comsume 
a  lot of memory... 
I know that sessions are stored in database, I can see them in the table 
django_session, but they are kept also in RAM, and this is my problem.
How can I reduce the RAM memory consumption?

I thought 2 ways:
1) Find a way to move the lists from the RAM and put them in the db, I hope 
there is something built-in in django session, but I did not find them
2) Instead building the lists of strings, I create a temp file in which I 
will append every string. In the end the combining function reads the temp 
files and combines them

Could you please help me? I'm really I'm really stuck on this...

thank you very much!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c7118798-84fe-4a32-bba0-53452871b6ae%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.