Thanks for the reply. My teammate that writing the scrapers is a java guy, and he's planning to use that rather than django/python. I haven't looked at the kinds of databases that django creates, but do you think that will cause any problems?
If not, I think we'll proceed with plan B, below. Michael Lissner [email protected] 909-576-4123 Rishabh Manocha wrote on 02/18/2010 07:16 PM: > > > On Fri, Feb 19, 2010 at 10:04 AM, mjlissner <[email protected] > <mailto:[email protected]>> wrote: > > I'm using django for a final project in my masters program at UC > Berkeley, and I'm trying to sort out exactly how the database works. I > would spend a bunch of time figuring this out myself, but I am working > in a team, and my teammates need this info asap. > > My teammates are making scrapers that pull information from > various .gov websites, and then dump that information into a mysql > database. > > It's my job to use django to make queries on that database, and to > build a web front end to it. > > We've developed a data model for the database, but my question is > what's the best way to interface between the data they're dumping in, > and django? I have two thoughts about this...both of which are > probably wrong: > 1. They can build a separate database, and I can pull from it using > custom sql queries within my django code; or > 2. I can use django models to flesh out the database, and then they > can dump into the tables that were created by the syncdb command. > > Can people opine as to what the best way to do this might be, or what > kinds of problems we can anticipate if we do the above? > > -- > You received this message because you are subscribed to the Google > Groups "Django users" group. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > To unsubscribe from this group, send email to > [email protected] > <mailto:django-users%[email protected]>. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > > I've done something similar with my project. I setup up my database > tabels such that they work with both my scraping code as well as my > django code. To insert data from my scraping code, I used sqlalchemy > (though, you can just as easily use django's ORM for this - this > method would probably be better in the longer run 'cause you can > define your models in one place and just include them in both your > scraping code as well as your django site) and then just accessed that > data from my django site accessing the same database (no replication, > parsing etc.). This method has worked out quiet well for me, so far. > > -- > > Best, > > R > -- > You received this message because you are subscribed to the Google > Groups "Django users" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
