On Mon, 08 Dec 2008 08:55:40 -0800, Lawrence Wickline wrote: > On Dec 6, 2008, at 12:41 AM, Lie Ryan wrote: >> >> In most cases, in processing involving networking, the bottleneck is >> the >> network speed itself. To speed things up by optimizing your own code >> might not make your download significantly faster (getting 60 seconds >> faster is great for scripts that usually runs for 70 seconds, but is a >> waste of development time for scripts that usually run for 1 hour) >> >> Usually a multi-threading downloader might be a better chance to >> improvement, especially for 1) downloading from different site, 2) the >> remote sites have speed limit, 3) you have faster download link than >> the >> server can gives > > > In this particular case everything is on the local network. This is > actually part of a hadoop map/reduce system I am learning, so reducing > cpu is of high value. if network pull times become and issue the cluster > can be expanded and the time between pulls can be reduced. As of this > morning I am being directed to make the reducer usable both in the > mapper and then again as a reducer. This has forced me to rework > everything to work so that it can be called as a module. >
Just don't forget about the profile and cProfile module, it might reveal hidden bottlenecks. _______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
