Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Bloody thing, had to send it directly to the list cos it doesn't have the option of bulk inviting people. How annoying :X Anyone who didn't receive the invite email, please let me know. Cal On Sat, Aug 27, 2011 at 9:50 PM, Cal Leeming [Simplicity Media Ltd] <

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Nope, we're allowed up to 100 attendees per time slot, and we are at 55 at the moment, so feel free to register for both. On Sat, Aug 27, 2011 at 9:46 PM, william ratcliff < william.ratcl...@gmail.com> wrote: > Any problem if we register for both? > > > On Sat, Aug 27, 2011 at 4:33 PM, Cal

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Any problem if we register for both? On Sat, Aug 27, 2011 at 4:33 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hey guys, > > I'm sending out the webcast invitations now (should receive them in about > 10 minutes from GoToMeeting). Last chance to jump on

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hey guys, I'm sending out the webcast invitations now (should receive them in about 10 minutes from GoToMeeting). Last chance to jump on if you haven't already. Cal On Sat, Aug 20, 2011 at 12:32 AM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hi all, > >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi all, Okay, good news and bad news. The bad news, is that the site/project which sparked this webcast to be made, has had some legal complications and is being shut down - less than 1 month after being released :L The good news, is that the webcast will still be going ahead. For those of you

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Last call on this guys: https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0 If you want to register your attendance, please do so now. Thanks On Mon, Jul 25, 2011 at 2:00 PM, nicolas HERSOG wrote: > I

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

I won t be able to follow the live session but I can't wait to watch your record on YT. Thx! On Mon, Jul 25, 2011 at 2:42 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Second call for register your attendance on this webcast, if you haven't > already

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Second call for register your attendance on this webcast, if you haven't already done so. Thanks Cal On Tue, Jul 12, 2011 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hi all, > > Great response to this, 45 registered votes in total. > > The

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Unfortunately, I'm on GMT-0300 and none of those timezones work for me. Looking forward to the recorded version, though! Cheers, André On Tue, Jul 12, 2011 at 10:48 AM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hi all, > > Great response to this, 45

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

> If you're interested, please reply on-list so others can see. Would be very interested in attending. Thanks! -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Of course :) I'll make sure to take a high quality recording and stick it on YT. Cal On Tue, Jul 12, 2011 at 6:31 PM, Venkatraman S wrote: > Can you please ..please...please..please record this session!? > > -V > > -- > You received this message because you are subscribed

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Can you please ..please...please..please record this session!? -V -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi all, Great response to this, 45 registered votes in total. The webcast will take place on Monday 29th August 2011 - (Minimum resolution 1920x1080) However, because the time zone results are almost split 50/50, I'm going to allow users to select from two time slots, if the results are still

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Oh, sorry :). 2011/7/3 Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> > This has already been discussed earlier in the thread :) > > On Sun, Jul 3, 2011 at 4:28 PM, Andre Santos wrote: > > Maybe someone can record it and upload somewhere for the

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

This has already been discussed earlier in the thread :) On Sun, Jul 3, 2011 at 4:28 PM, Andre Santos wrote: > Maybe someone can record it and upload somewhere for the ones who cant be > online when you do the presentation... > > 2011/7/3 gradja >> >>

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Maybe someone can record it and upload somewhere for the ones who cant be online when you do the presentation... 2011/7/3 gradja > > > If you're interested, please reply on-list so others can see. > > Count me in, and thanks to share your experience with others. >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

> If you're interested, please reply on-list so others can see. Count me in, and thanks to share your experience with others. Graziella -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Oops, sorry guys, I pasted the wrong link. Here you go: https://spreadsheets0.google.com/spreadsheet/viewanalytics?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ On Sat, Jul 2, 2011 at 11:35 AM, Cal Leeming [Simplicity Media Ltd] wrote: > Hi guys, > Alright,

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi guys, Alright, vote is closed and here are the results: https://spreadsheets.google.com/spreadsheet/gform?key=0AiciLOVZqDWqdENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c=en_GB=0#chart It looks like I might have to do two separate webcasts to cater for all timezones, but we'll see. I'll post up a fixed

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

> Hey all, > > Last call for registering your interest in this webcast (28 votes so far). > > If you haven't done so already, please visit the following URL: > > https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0 >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hey all, Last call for registering your interest in this webcast (28 votes so far). If you haven't done so already, please visit the following URL: https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0 Poll ends tomorrow, at which point we will

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Please, count me in. On Jun 22, 2011 10:15 AM, "Cal Leeming [Simplicity Media Ltd]" < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hi all, > > Some of you may have noticed, in the last few months I've done quite a few > posts/snippets about handling large data sets in Django. At the end of this

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

If I am unable to attend at the scheduled time I absolutely must watch the resulting recording. Thanks for your efforts. On Jun 22, 5:50 pm, "Cal Leeming [Simplicity Media Ltd]" wrote: > Nice! > > On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Wow, that sucks! :X Never known a corp to block google docs On Wed, Jun 29, 2011 at 2:16 PM, Andre Terra wrote: > Can't access google spreadsheets through my corporate proxy (go figure), > but I will vote later today. > > Really interested in watching this! > > > Andre > >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Can't access google spreadsheets through my corporate proxy (go figure), but I will vote later today. Really interested in watching this! Andre On Tue, Jun 28, 2011 at 12:30 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Second call for anyone who

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Second call for anyone who wants to attend this webcast, 3 days left to place your vote. Cal On Fri, Jun 24, 2011 at 7:34 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Really glad to see there has been so much interest in this! > > If possible, can

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Really glad to see there has been so much interest in this! If possible, can everyone please use the following form to indicate what week day / timeslot they are available, along with their screen resolution. This will be open until 1st July.

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Yes, very interested. +1 On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers < bruno.desthuilli...@gmail.com> wrote: > +1 > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

+1 -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi, Cal and everybody, I'd like to see the webcast too, thanks very much for that :) On Jun 23, 2:50 am, "Cal Leeming [Simplicity Media Ltd]" wrote: > Nice! > > On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt > wrote: > > > > > > >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

I would like to attend, too. Thank you in advance! CU Fatrix -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

+1 Very interested. On Jun 22, 3:47 pm, "Cal Leeming [Simplicity Media Ltd]" wrote: > Nice, seems like there is a lot of positive feedback. > > Here's some further info about the webcast/webinar: > > *Expected Date:* July/August > *Expected Time:* Somewhere

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

(sorry, should have sent this reply on-list) You know, I was actually thinking about that, but the flights would cost way too much (like $1400+) ;/ If the project has interest next year, I'll probably ask about presenting at DjangoCon EU though. On Thu, Jun 23, 2011 at 6:13 PM, Steve Holden

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

And, as luck would have it, the US Call For Papers was just published: http://djangocon.us/blog/2011/06/22/call-papers/ regards Steve On Wed, Jun 22, 2011 at 9:37 AM, Shawn Milochik wrote: > Cal, > > That sounds awesome. I wish you could present it at DjangoCon US too.

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Me On Jun 23, 2011, at 7:49, Chris Calitz wrote: > Sounds really cool. I'm definitely in. > > On 22 Jun 2011, at 14:16, "Cal Leeming [Simplicity Media Ltd]" > wrote: > >> Hi all, >> >> Some of you may have noticed, in the last

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Sounds really cool. I'm definitely in. On 22 Jun 2011, at 14:16, "Cal Leeming [Simplicity Media Ltd]" wrote: > Hi all, > > Some of you may have noticed, in the last few months I've done quite a few > posts/snippets about handling large data sets in

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi Derek, I've had a look and apparently 4-5mbit should be sufficient enough for a HD webinar. If quite a few of you don't have this capability (or screen size), I could look into reducing it down to 1440x900 perhaps? Let me know your thoughts guys. (maybe it's something I need to do a web vote

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Carl Please post here when the recording becomes available... some of us do not have access to HD-capable bandwidth yet! Thanks Derek -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

> If you're interested, please reply on-list so others can see. > would love to see this. thanks for posting. -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Definitely interested. Struggling with scaling a Django/Celery/Postgres/CouchDB/Memcach project now. Would love to see your approach. Cheers Vlad -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

I was thinking.. what would be really nice would be a monkey patch that could catch any inserts/updates/saves from the ORM, and defer them until the user triggered the bulk update some how.. That way, DSE would be completely interchangeable with existing code, making it much more likely for people

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Definitely looking forward to it! On Wed, Jun 22, 2011 at 6:50 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Nice! > > On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt > wrote: > >> On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Nice! On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt wrote: > On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd] > wrote: > > > > > > On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra > wrote: >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Nice, seems like there is a lot of positive feedback. Here's some further info about the webcast/webinar: *Expected Date:* July/August *Expected Time:* Somewhere between 9AM - 9PM GMT+0 (UK Time), I'll probably run a webpage vote so you guys can decide. *Via:* GoToWebinar - screen share with

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd] wrote: > > > On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra wrote: >> >> Hello, Cal >> >> First of all, congrats on the newborn! The Django community will surely >>

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

I really like the idea, hope i can see it soon. Interessting thanks On 22 jun, 15:15, "Cal Leeming [Simplicity Media Ltd]" wrote: > Hi all, > > Some of you may have noticed, in the last few months I've done quite a few > posts/snippets about handling large

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Great idea. I also would like to see this webcast On Jun 22, 6:00 pm, "Cal Leeming [Simplicity Media Ltd]" wrote: > On Wed, Jun 22, 2011 at 4:55 PM, creecode wrote: > > Hello SleepyCal, > > > On Wednesday, June 22, 2011 6:15:48 AM UTC-7,

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On 22 June 2011 14:15, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > Hi all, > > Therefore, I'd like to see if there would be any interest in webcast in > which I would explain how we handle such large amounts of data, the trial > and error processes we went

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hello SleepyCal, On Wednesday, June 22, 2011 9:00:15 AM UTC-7, SleepyCal wrote: Yup, I'm planning on recording in 1080p and posting on Youtube shortly > afterwards. > Fantastic! Thank you, Toodle-l... creecode -- You received this message because you are subscribed to the

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

+1 for making it viewable after the fact On Wed, Jun 22, 2011 at 11:55 AM, creecode wrote: > Hello SleepyCal, > > > On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote: > > If you're interested, please reply on-list so others can see. > > > +1 > > Also if the

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 4:55 PM, creecode wrote: > Hello SleepyCal, > > > On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote: > > If you're interested, please reply on-list so others can see. > > > +1 > > Also if the webcast could be stored for later viewing that

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hello SleepyCal, On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote: If you're interested, please reply on-list so others can see. +1 Also if the webcast could be stored for later viewing that would be grand. Toodle-loo.. creecode -- You received this

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

I am in :-) On Wed, Jun 22, 2011 at 11:39 AM, Ivan Aleman wrote: > > > On 22 June 2011 08:15, Cal Leeming [Simplicity Media Ltd] < > cal.leem...@simplicitymedialtd.co.uk> wrote: > >> >> If you're interested, please reply on-list so others can see. >> >> >> > Sweet! Count

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On 22 June 2011 08:15, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > > If you're interested, please reply on-list so others can see. > > > Sweet! Count me in :) > > -- Iván -- You received this message because you are subscribed to the Google Groups

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 11:47 AM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > > > On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra wrote: > >> Hello, Cal >> >> First of all, congrats on the newborn! The Django community will surely >>

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra wrote: > Hello, Cal > > First of all, congrats on the newborn! The Django community will surely > benefit from having yet another success story, especially considering how > big this project sounds. Is there any chance you could

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hmm, that's odd, the grouping (map/reduce/filter/lambda) is extremely quick for me (even on a heavy data set). My guess is that grouping would need to be done on a combination of field name+value, and would need to allow the user to specify what bulk to use (to prevent MemoryError exception - or

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hello, Cal First of all, congrats on the newborn! The Django community will surely benefit from having yet another success story, especially considering how big this project sounds. Is there any chance you could open-source some of your custom made improvements so that they could eventually be

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 3:52 PM, Cal Leeming [Simplicity Media Ltd] wrote: > Sorry, let me explain a little better. > (51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: > 72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked', >

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Also, the 13.8 minutes per million, is basically a benchmark based on the amount of db writes, and the total amount of time it took to execute (which was 51s). Please also note, this code is doing a *heavy* amount of content analysis, but if you were to strip that out, the only overheads would be

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Sorry, let me explain a little better. (51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: 72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)] map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems())

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd] wrote: > Hey Thomas, > Yeah we actually spoke a little while ago about DSE. In the end, we actually > used a custom approach which analyses data in blocks of 50k rows, builds a > list of rows

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Cal, That sounds awesome. I wish you could present it at DjangoCon US too. :o/ Shawn -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hey Thomas, Yeah we actually spoke a little while ago about DSE. In the end, we actually used a custom approach which analyses data in blocks of 50k rows, builds a list of rows which need changing to the same value, then applied them in bulk using update() + F(). Here's our benchmark: (42.11s)

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Yes! I'm in. Out of curiosity: When inserting lots of data, how do you do it? Using the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I wrote DSE to solve inserting/updating huge sets of data, but if there's a better way to do it that would be especially interesting to hear more

Re: Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Dnia 2011-06-22, śro o godzinie 14:15 +0100, Cal Leeming [Simplicity Media Ltd] pisze: > If you're interested, please reply on-list so others can see. Sure, I'd attend. -- Michał (Saviq) Sawicz signature.asc Description: This is a digitally signed message part

Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

Hi all, Some of you may have noticed, in the last few months I've done quite a few posts/snippets about handling large data sets in Django. At the end of this month (after what seems like a lifetime of trial and error), we're finally going to be releasing a new site which holds around 40mil+ rows