Re: Another attempt at speeding up yaouh, with persistent connections
Al Johnson wrote: On Friday 06 March 2009, Helge Hafting wrote: Al Johnson wrote: On Thursday 05 March 2009, Helge Hafting wrote: Fixing this in advance is a better approach - but I'd rather spend time on a much better fix using libcurl. That would speed yaouh up further also. But this have to wait until someone ports pycurl to the freerunner. No porting needed - pycurl is in OE. bitbake python-pycurl I haven't yet set up any development tools, which is why I worked on a python program. Is there an ipk/opk file, or at least something that can be installed with tar? http://openmoko.truebox.co.uk/repos/mazikeen/fso-milestone5/ipk/armv4t/ I've a little housekeeping (and a lot of upload time) to do before it's a full repository, but individual python packages are there at the moment. Thanks a lot! Downloading individual packages is fine with me. I'll see what I can do - pycurl will let me do the entire version check and upgrade through one or a handful persistent connections, that will remove delays that currently happen with small map subdirectories. Filling the network bandwith should be easy then, unless md5 calculations become the new limit. But that can be handled by storing the md5sums, so they only need to be calculated for new/updated files. Helge Hafting ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
Al Johnson wrote: On Thursday 05 March 2009, Helge Hafting wrote: Fixing this in advance is a better approach - but I'd rather spend time on a much better fix using libcurl. That would speed yaouh up further also. But this have to wait until someone ports pycurl to the freerunner. No porting needed - pycurl is in OE. bitbake python-pycurl I haven't yet set up any development tools, which is why I worked on a python program. Is there an ipk/opk file, or at least something that can be installed with tar? Helge Hafting ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
On Friday 06 March 2009, Helge Hafting wrote: Al Johnson wrote: On Thursday 05 March 2009, Helge Hafting wrote: Fixing this in advance is a better approach - but I'd rather spend time on a much better fix using libcurl. That would speed yaouh up further also. But this have to wait until someone ports pycurl to the freerunner. No porting needed - pycurl is in OE. bitbake python-pycurl I haven't yet set up any development tools, which is why I worked on a python program. Is there an ipk/opk file, or at least something that can be installed with tar? http://openmoko.truebox.co.uk/repos/mazikeen/fso-milestone5/ipk/armv4t/ I've a little housekeeping (and a lot of upload time) to do before it's a full repository, but individual python packages are there at the moment. ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
2009/3/3 Helge Hafting helge.haft...@hist.no: The tiles have 5-digit names, so the max would be 10 ? i think it only uses as many digits as necessary - i.e. there are never any triling zeroes. if you look at antarctica, there are numbers up to 130,000 for zoom 17. if tango ever supports zoom 18, then there's potential for 260,000 per directory. is it likely, that anyone will have more than 250 per directory? i dunno, but i recall a certain prophecy that 640kb of ram...something, something Anyway, that could be a problem. With hundred thousand tiles and about 60 byte per URL, pythm would send a 7.5MB parameter list to a invocation of curl. That can easily be trimmed down to 950kB by only sending file names. Still, it'd be interesting to know if the freerunner can handle that at all. A pc usually can. A loop can be used to limit this to something sane, like 1000 files or so. But do anyone actually have that many files? I just checked my 50.000 tiles, and found no more than 250 tiles in any directory. Perhaps a large square area at zoom 17? ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
Robin Paulson wrote: 2009/3/3 Helge Hafting helge.haft...@hist.no: The tiles have 5-digit names, so the max would be 10 ? i think it only uses as many digits as necessary - i.e. there are never any triling zeroes. if you look at antarctica, there are numbers up to 130,000 for zoom 17. if tango ever supports zoom 18, then there's potential for 260,000 per directory. is it likely, that anyone will have more than 250 per directory? i dunno, but i recall a certain prophecy that 640kb of ram...something, something There could be problems. If that happens, I can program a loop to solve it. So far, nobody reported such problems. Perhaps nobody has huge collections of zoom 17 tiles - or maybe python has no problem invoking curl with a 7MB parameter list. Fixing this in advance is a better approach - but I'd rather spend time on a much better fix using libcurl. That would speed yaouh up further also. But this have to wait until someone ports pycurl to the freerunner. Helge Hafting ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
On Thursday 05 March 2009, Helge Hafting wrote: Fixing this in advance is a better approach - but I'd rather spend time on a much better fix using libcurl. That would speed yaouh up further also. But this have to wait until someone ports pycurl to the freerunner. No porting needed - pycurl is in OE. bitbake python-pycurl ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
Robin Paulson wrote: 2009/3/3 Helge Hafting helge.haft...@hist.no: too. It looks like openstreetmap can have up to 50 files in a directory. So far this program has checked 5100 files in 12 minutes, and downloaded 260. It looks like it will get through all my 50.000 tiles in under 2 hours. I use wifi, the usb connection may be slower. If anyone want to try, here is the download link. http://www.aitel.hist.no/~helgehaf/openmoko/yaouh_new.py awesome, that's lots faster by the way, i don't know if it's important or not, but iirc osm folder structure has up to 128000 files per folder, for zoom 17 The tiles have 5-digit names, so the max would be 10 ? Anyway, that could be a problem. With hundred thousand tiles and about 60 byte per URL, pythm would send a 7.5MB parameter list to a invocation of curl. That can easily be trimmed down to 950kB by only sending file names. Still, it'd be interesting to know if the freerunner can handle that at all. A pc usually can. A loop can be used to limit this to something sane, like 1000 files or so. But do anyone actually have that many files? I just checked my 50.000 tiles, and found no more than 250 tiles in any directory. Perhaps a large square area at zoom 17? Further speedups are possible. File check data for 50.000 tiles is 12MB, the limit for such a dataset is how fast 12MB can be downloaded, and how fast all those files can be checksummed. And then there is the transfer of changed tiles. My code is inefficient when there are many small directories, because it starts a new transfer for each directory. To get the best possible speed, pycurl should be used. Everything can then be done from inside python, avoiding process creation overhead. Persistent connections can be utilized for the entire transfer, eliminating the small-directory problem entirely. I am not sure if there is a pycurl package yet though. Installing python-modules dodn't bring it in. Helge Hafting ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Another attempt at speeding up yaouh, with persistent connections
My last attempt at speeding up yaouh was not only clunky to use, it also drove the cpu load up to 12-15, which made the phone extremely sluggish during the update. This time I took a different approach that keeps the load in the 0.3-0.5 range. There is no more parallelism than what you find in standard yaouh v. 0.4, most of the speedup comes from avoiding network latency. Standard yaouh uses curl to fetch the checksum for one file at a time. My approach have curl downloading all the files in a directory in one operation. This helps a lot, because curl will use persistent connections instead of making a new connection per file. Not having to start a new process for each file helps too. It looks like openstreetmap can have up to 50 files in a directory. The same trick is used for md5sum, as md5sum also is capable of processing several files in one go. This avoid some process startup time as well. Finally, files that aren't up-to-date are downloaded with curl instead of wget. This way one invocation can download all stale tiles in a directory in one go, again utilizing persistent connections. So far this program has checked 5100 files in 12 minutes, and downloaded 260. It looks like it will get through all my 50.000 tiles in under 2 hours. I use wifi, the usb connection may be slower. If anyone want to try, here is the download link. http://www.aitel.hist.no/~helgehaf/openmoko/yaouh_new.py Install yaouh v 0.4 from opkg.org if you don't already have it, so that all dependencies are in place. My modified script should then work. Helge Hafting -- View this message in context: http://n2.nabble.com/Another-attempt-at-speeding-up-yaouh%2C-with-persistent-connections-tp2411310p2411310.html Sent from the Openmoko Community mailing list archive at Nabble.com. ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
O Luns, 2 de Marzo de 2009, Helge Hafting escribiu: My last attempt at speeding up yaouh was not only clunky to use, it also It's really fast now!!! Thank you very much for this great program! -- David Garabana Barro jabber google talk ID:da...@garabana.com Clave pública PGP/GPG: http://davide.garabana.com/pgp.html signature.asc Description: This is a digitally signed message part. ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: Another attempt at speeding up yaouh, with persistent connections
2009/3/3 Helge Hafting helge.haft...@hist.no: too. It looks like openstreetmap can have up to 50 files in a directory. So far this program has checked 5100 files in 12 minutes, and downloaded 260. It looks like it will get through all my 50.000 tiles in under 2 hours. I use wifi, the usb connection may be slower. If anyone want to try, here is the download link. http://www.aitel.hist.no/~helgehaf/openmoko/yaouh_new.py awesome, that's lots faster by the way, i don't know if it's important or not, but iirc osm folder structure has up to 128000 files per folder, for zoom 17 ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
