Re: Importing file to Models - Temporary store to allow confirming?

[Shawn Milochik]

On 06/28/2011 02:43 AM, Victor Hooi wrote:


Shawn,

Thanks for the quick reply =).

If we go with the third approach, what advantages are there to 
persisting the models in MongoDB (or something similar like Redid.or 
Cassandra), as opposed to a traditional RDBMS? (I just want to get a 
good handle on the issues).



The schema-less nature will give you the ability to "just work."

Furthermore, is there a particular reason you picked MongoDB over the 
other NoSQL solutions?


No. I heard about it on FLOSS Weekly, checked out, and I love it. Also, 
pymongo is great and I'm a Python guy. I've read some comparisons among 
the other "NoSQL" databases and some are better for some uses but 
MongoDB is great for me.


Also, in terms of actually persisting the temporary models to MongoDB, 
I can just use PyMongo and store the dicts themselves - and just user 
a nested dict to represent all the model instances from a given import 
file. Any issues with that approach?




No issues that I can see. If you're using pymongo then you can just 
treat everything as a Python dictionary and MongoDB will consume it no 
problem (assuming you stick to certain data types).


Thanks for the tip about using asynchronous processing for the import 
file - I didn't think the file would be that big/complex to process, 
but I suppose it's probably the better approach to do it all 
asynchronously, instead of just hoping it won't grow larger I the 
future. In this case, I suppose I can just use Ajax on the page itself 
to check on the status of the queue?


Sure, although Comet would be better for this use-case. You could also 
just e-mail the user from the background process after it's done with a 
link to the URL for the next step. AJAX is easier than Comet (no service 
to install or set up), and it probably wouldn't generate enough extra 
bandwidth to really bother you.



Cheers,
Victor

--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/CvCjWNrV4pUJ.

To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Importing file to Models - Temporary store to allow confirming?

[Victor Hooi]

Shawn,

Thanks for the quick reply =).

If we go with the third approach, what advantages are there to persisting 
the models in MongoDB (or something similar like Redid.or Cassandra), as 
opposed to a traditional RDBMS? (I just want to get a good handle on the 
issues).

Furthermore, is there a particular reason you picked MongoDB over the other 
NoSQL solutions?

Also, in terms of actually persisting the temporary models to MongoDB, I can 
just use PyMongo and store the dicts themselves - and just user a nested 
dict to represent all the model instances from a given import file. Any 
issues with that approach?

Thanks for the tip about using asynchronous processing for the import file - 
I didn't think the file would be that big/complex to process, but I suppose 
it's probably the better approach to do it all asynchronously, instead of 
just hoping it won't grow larger I the future. In this case, I suppose I can 
just use Ajax on the page itself to check on the status of the queue?

Cheers,
Victor

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/CvCjWNrV4pUJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Importing file to Models - Temporary store to allow confirming?

[Victor Hooi]

Shawn,

Thanks for the quick reply =).

If we go with the third approach, what advantages are there to persisting
the models in MongoDB (or something similar like Redid.or Cassandra), as
opposed to a traditional RDBMS? (I just want to get a good handle on the
issues).

Furthermore, is there a particular reason you picked MongoDB over the other
NoSQL solutions?

Also, in terms of actually persisting the temporary models to MongoDB, I can
just use PyMongo and store the dicts themselves - and just user a nested
dict to represent all the model instances from a given import file. Any
issues with that approach?

Thanks for the tip about using asynchronous processing for the import file -
I didn't think the file would be that big/complex to process, but I suppose
it's probably the better approach to do it all asynchronously, instead of
just hoping it won't grow larger I the future. In this case, I suppose I can
just use Ajax on the page itself to check on the status of the queue?

Cheers,
Victor
On Mon, Jun 27, 2011 at 04:36, Shawn Milochik  wrote:

> If you're using Django 1.2 or higher you can take advantage of the
> multi-database support by adding the 'using' kwarg to your model.save().
> This will allow you to ensure that the model saves successfully (is valid)
> in the 'holding' database, and move it to your 'live' database later.
>
> You could add a field to the model indicating whether it's 'live' or
> 'pending,' load all the temp models as pending, then just flip the flag as
> each one is "approved." That would front-load all the processing.
>
> You can use a ModelForm to accept the CSV data (preferably in dict form,
> from csv.DictReader) to validate the data, then just check for is_valid()
> without saving anything to the database. You could then store those
> dictionaries somewhere (such as MongoDB) for retrieval when you actually
> want to save them to the database.
>
> Each of these options has different advantages. I don't know about
> performance. In any case, you may have to use asynchronous processing (via
> ZeroMQ or django-celery) so the page can refresh before you Web server's
> timeout.
>
> Use whichever seems best for your needs, and maybe someone else has another
> option. I'd prefer option #3, because it keeps all the temporary data out of
> your production database and makes good use of the validation properties of
> the ModelForm class.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to django-users+unsubscribe@**
> googlegroups.com .
> For more options, visit this group at http://groups.google.com/**
> group/django-users?hl=en
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Importing file to Models - Temporary store to allow confirming?

[Shawn Milochik]

If you're using Django 1.2 or higher you can take advantage of the 
multi-database support by adding the 'using' kwarg to your model.save(). 
This will allow you to ensure that the model saves successfully (is 
valid) in the 'holding' database, and move it to your 'live' database later.


You could add a field to the model indicating whether it's 'live' or 
'pending,' load all the temp models as pending, then just flip the flag 
as each one is "approved." That would front-load all the processing.


You can use a ModelForm to accept the CSV data (preferably in dict form, 
from csv.DictReader) to validate the data, then just check for 
is_valid() without saving anything to the database. You could then store 
those dictionaries somewhere (such as MongoDB) for retrieval when you 
actually want to save them to the database.


Each of these options has different advantages. I don't know about 
performance. In any case, you may have to use asynchronous processing 
(via ZeroMQ or django-celery) so the page can refresh before you Web 
server's timeout.


Use whichever seems best for your needs, and maybe someone else has 
another option. I'd prefer option #3, because it keeps all the temporary 
data out of your production database and makes good use of the 
validation properties of the ModelForm class.



--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Importing file to Models - Temporary store to allow confirming?

[Victor Hooi]

heya,

We have a CSV file that we are importing into a Django application, and then 
creating the appropriate models and relationships.

At the first page, we have a file upload form where the user selects a file.

We then parse the file, and return a second page showing them what would be 
created, any validation errors etc.

The user can then decide whether to proceed or not (or possibly to correct 
any areas on-screen).

What would be the best way of storing the temporary interim models, before 
it actually hits the database proper?

The CSV file will be fairly big, possibly around 200 Kb in size, and create 
several hundred models.

Should I store this in the database somewhere, and label those models 
"temporary"? It seems a bit heavy just for a confirm, and I'm not sure if 
it's appropriate use of the database. Or is there some way we could store it 
in Django sessions? Or any other way to do it?

Cheers,
Victor

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/zBHlA9Q2t7MJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.