Re: [Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
You define the path to the mbtiles in tilestache.cfg, so yes, they could be in any location (as long as the access rights / permissions are right) On Tue, Aug 11, 2015 at 8:28 AM, Tim Moody t...@timmoody.com wrote: This looks great and not overly complex. XSCE uses one convention that you are probably not aware of and that is that we put data in /library, in this case probably /library/osm or tilestache or some combination. Can tilestache.cfg point to a location different than the html and config files? *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com] *On Behalf Of *Adam Holt *Sent:* Tuesday, August 11, 2015 12:19 AM *To:* xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org *Subject:* [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Don't stop pushing Jerome- I hope you can demonstrate Nepal-or-larger working models during our Toronto summit Oct 10-12. See you 7AM EDT Thursday on our weekly call, if you have time for our unusually early meeting this week: http://tinyurl.com/xsceminutes (Several of our early design discussions on offline mapping began on unleashk...@googlegroups.com, so am bringing that thread full circle.) On Mon, Aug 10, 2015 at 11:01 PM, Anish Mangal anis...@umich.edu wrote: adding server-devel (the publicly archived mailing list) as well. Great work. Thanks! Happy to help with the testing. I have an intel NUC running an atom processor 2 Ghz, and 2GB RAM - not exactly low end, but still a viable appliance to deploy the xsce. Could you tell me how to get the mbtiles from your ftp. If this works, I can help with creating a playbook - and possibly admin console integration. Thanks, Anish On Tue, Aug 11, 2015 at 8:19 AM, Jérôme Gagnon-Voyer gagno...@gmail.com wrote: Hi all This is about the project to bring a new way of storing OpenStreetMaps offline for internet in a box (schoolserver). (Apologies if I'm using the wrong mailing list, feel free to add more) I wanted to test what it would look like to run Tilestache to serve .mbtiles using the schoolserver. So I installed XSCE on a local VM with Fedora 21 here, installed Tilestache, moved the .mbtiles in it, and did some testing. So far I only tested with Nepal zoom 0-13 and Nepal zoom 14-15 (to see how combining multiple files was working) I reduced the VM to use only 1 core at 50% (I'm on MBP Intel i5 2.6Ghz, 2 cores). Speed was decent when viewing with leaflet in my browser from my laptop. I've created a gist that explains all the steps required to do it https://gist.github.com/jeromegv/cb64bc2c994e80af The reason I wanted to test it was also to see if the integration would be relatively straightforward. I assume you would eventually want to do the install through the schoolserver admin console. When comes the time to do that, I might not be the person to do this integration, but I think that the steps I've done so far should at least prove that the XSCE stack should be compatible. - I might not have the best testing setup or what is similar to a typical schoolserver, if you have the proper hardware to test, let me know, I can send you the .mbtiles on a FTP for nepal and you can test it (about 1Gb) - Feel free to provide feedback on the GIST and if the integration seems to make sense - There's an extra optimization that is possible for tilestache, to cache the tiles to the HD once they were retrieved and served from the .mbtiles once. It's currently disabled. Not so sure if it's worth it and how much we would gain considering MBtiles are already pre-rendered files, perhaps better to test once we have a full planet. The downside is that it stores on HD a lot of PNG files and this might eventually baloon if you get tons of people to use those maps. Benefit of sticking entirely to mbtiles is we know exactly how much disk space. Might be optional setting from admin console when there's plenty of HD available. - Next step for me will be to test generation of MBtiles for bigger country and eventually get to the Planet. Will need some type of cloud setup to manage/store all that. Thanks -- Jérôme Gagnon-Voyer -- Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org ! -- Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org ! --- You received this message because you are subscribed to the Google Groups Unleash Kids group. To unsubscribe from this group and stop receiving emails from it, send an email to unleashkids+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org ! --- You received this message because you are subscribed to the Google Groups Unleash Kids group. To unsubscribe from this group and stop
Re: [Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
This looks great and not overly complex. XSCE uses one convention that you are probably not aware of and that is that we put data in /library, in this case probably /library/osm or tilestache or some combination. Can tilestache.cfg point to a location different than the html and config files? From: unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com] On Behalf Of Adam Holt Sent: Tuesday, August 11, 2015 12:19 AM To: xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org Subject: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Don't stop pushing Jerome- I hope you can demonstrate Nepal-or-larger working models during our Toronto summit Oct 10-12. See you 7AM EDT Thursday on our weekly call, if you have time for our unusually early meeting this week: http://tinyurl.com/xsceminutes (Several of our early design discussions on offline mapping began on unleashk...@googlegroups.com mailto:unleashk...@googlegroups.com , so am bringing that thread full circle.) On Mon, Aug 10, 2015 at 11:01 PM, Anish Mangal anis...@umich.edu mailto:anis...@umich.edu wrote: adding server-devel (the publicly archived mailing list) as well. Great work. Thanks! Happy to help with the testing. I have an intel NUC running an atom processor 2 Ghz, and 2GB RAM - not exactly low end, but still a viable appliance to deploy the xsce. Could you tell me how to get the mbtiles from your ftp. If this works, I can help with creating a playbook - and possibly admin console integration. Thanks, Anish On Tue, Aug 11, 2015 at 8:19 AM, Jérôme Gagnon-Voyer gagno...@gmail.com mailto:gagno...@gmail.com wrote: Hi all This is about the project to bring a new way of storing OpenStreetMaps offline for internet in a box (schoolserver). (Apologies if I'm using the wrong mailing list, feel free to add more) I wanted to test what it would look like to run Tilestache to serve .mbtiles using the schoolserver. So I installed XSCE on a local VM with Fedora 21 here, installed Tilestache, moved the .mbtiles in it, and did some testing. So far I only tested with Nepal zoom 0-13 and Nepal zoom 14-15 (to see how combining multiple files was working) I reduced the VM to use only 1 core at 50% (I'm on MBP Intel i5 2.6Ghz, 2 cores). Speed was decent when viewing with leaflet in my browser from my laptop. I've created a gist that explains all the steps required to do it https://gist.github.com/jeromegv/cb64bc2c994e80af The reason I wanted to test it was also to see if the integration would be relatively straightforward. I assume you would eventually want to do the install through the schoolserver admin console. When comes the time to do that, I might not be the person to do this integration, but I think that the steps I've done so far should at least prove that the XSCE stack should be compatible. * I might not have the best testing setup or what is similar to a typical schoolserver, if you have the proper hardware to test, let me know, I can send you the .mbtiles on a FTP for nepal and you can test it (about 1Gb) * Feel free to provide feedback on the GIST and if the integration seems to make sense * There's an extra optimization that is possible for tilestache, to cache the tiles to the HD once they were retrieved and served from the .mbtiles once. It's currently disabled. Not so sure if it's worth it and how much we would gain considering MBtiles are already pre-rendered files, perhaps better to test once we have a full planet. The downside is that it stores on HD a lot of PNG files and this might eventually baloon if you get tons of people to use those maps. Benefit of sticking entirely to mbtiles is we know exactly how much disk space. Might be optional setting from admin console when there's plenty of HD available. * Next step for me will be to test generation of MBtiles for bigger country and eventually get to the Planet. Will need some type of cloud setup to manage/store all that. Thanks -- Jérôme Gagnon-Voyer -- Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org ! -- Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org ! --- You received this message because you are subscribed to the Google Groups Unleash Kids group. To unsubscribe from this group and stop receiving emails from it, send an email to unleashkids+unsubscr...@googlegroups.com mailto:unleashkids+unsubscr...@googlegroups.com . For more options, visit https://groups.google.com/d/optout. ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
Hey Anish Not sure if you read what I wrote in the design document for the architecture, but the idea was to have it cloud based. Generate the tiles in EC2, and then host it in S3. Of course we would eventually need to estimate how much that would cost. Not sure yet if that's the best solution but that's what I'm proposing We also have to keep in mind that if we generate the tiles on the hardware that you mentioned, we still need to transfer the output somewhere to host it (and pay for the bandwidth fee to upload at that point). If we generate on EC2, we pay for the usage of EC2, but the transfer of the output to S3 is entirely free. My initial idea was to have something that is more maintainable in the cloud, where the entire tile generation can be re-trigerred when you desire it and something that can easily be changed/updated based on your need. I'm just afraid that if it's a private server, this might not be there in the long term, and not necessarily easy to give access to it. Just trying to come up with a solution that is not too manual (ie: moving files around through SSH through the server of someone else that only few people understand), which eventually means the tiles don't get updated often and might stay stale for a long term. Of course, open to other ideas, just my proposition here. On Tue, Aug 11, 2015 at 9:35 AM, Anish Mangal anis...@umich.edu wrote: Hi Here is the data size zoom 28K 0 48K 1 80K 2 184K 3 340K 4 788K 5 2.1M 6 5.8M 7 19M 8 58M 9 188M10 673M11 17M12 54M13 134M14 375M15 1.1G16 3.0G17 9.6G18 As is obvious, for zoom levels 0-11 I had generated tiles for the entire world, which was essentially empty tiles except the shape of continents since the only data source I used was the nepal osm data Zoom level 12 and onwards, tiles were generated only for nepal. The bigger question i'd be asking is the setup required to generate tiles, and what it would take to do for the entire planet. I guess Braddock would be happy to give access to Jerome on hadron if he wants to give it a try. It is a machine with a 500G SSD and 32G RAM and an 8 core processor. On Tue, Aug 11, 2015 at 6:38 PM, Tim Moody t...@timmoody.com wrote: Anish, you generated tiles for Nepal. How does the size compare to what you got? *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com] *On Behalf Of *Tim Moody *Sent:* Tuesday, August 11, 2015 9:04 AM *To:* 'Jérôme Gagnon-Voyer' gagno...@gmail.com *Cc:* 'Adam Holt' h...@laptop.org; 'xsce-devel' xsce-de...@googlegroups.com; 'Unleash Kids!' unleashk...@googlegroups.com; 'server-devel' server-devel@lists.laptop.org *Subject:* RE: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Can I get the data, maybe a dropbox link? not sure of a place to ftp unless you have one. *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com] *On Behalf Of *Jérôme Gagnon-Voyer *Sent:* Tuesday, August 11, 2015 9:01 AM *To:* Tim Moody t...@timmoody.com *Cc:* Adam Holt h...@laptop.org; xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org *Subject:* Re: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) You define the path to the mbtiles in tilestache.cfg, so yes, they could be in any location (as long as the access rights / permissions are right) On Tue, Aug 11, 2015 at 8:28 AM, Tim Moody t...@timmoody.com wrote: This looks great and not overly complex. XSCE uses one convention that you are probably not aware of and that is that we put data in /library, in this case probably /library/osm or tilestache or some combination. Can tilestache.cfg point to a location different than the html and config files? *From:* unleashk...@googlegroups.com [mailto:unleashk...@googlegroups.com] *On Behalf Of *Adam Holt *Sent:* Tuesday, August 11, 2015 12:19 AM *To:* xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org *Subject:* [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Don't stop pushing Jerome- I hope you can demonstrate Nepal-or-larger working models during our Toronto summit Oct 10-12. See you 7AM EDT Thursday on our weekly call, if you have time for our unusually early meeting this week: http://tinyurl.com/xsceminutes (Several of our early design discussions on offline mapping began on unleashk...@googlegroups.com, so am bringing that thread full circle.) On Mon, Aug 10, 2015 at 11:01 PM, Anish Mangal anis...@umich.edu wrote: adding server-devel (the publicly archived mailing list) as well. Great work. Thanks! Happy to help with the testing. I have an intel NUC running an atom processor 2 Ghz, and 2GB RAM - not exactly low end, but still a viable appliance
Re: [Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
I see i see, so the disk space can't be exactly compared to what we see in .mbtiles, since if you had to transfer that to another server and did not want to do rendering on fly, this would be a different file size (much bigger). Thanks On Tue, Aug 11, 2015 at 12:49 PM, Anish Mangal anishmangal2...@gmail.com wrote: I just used the default settings in renderd. As far as I know, I didn't generate any png or jpg tiles, only metatiles were generated, which are renderd as png on the fly. On Tue, Aug 11, 2015 at 8:10 PM, Jérôme Gagnon-Voyer gagno...@gmail.com wrote: Hi Anish Regarding file size Would you mind sharing how you generated those tiles? Like the settings used, if its PNG 8 bit, full png or JPG, metatiles size, style used, etc... I'll be sending you and Tim in separate email the link to download the .mbtiles file On Tue, Aug 11, 2015 at 9:35 AM, Anish Mangal anis...@umich.edu wrote: Hi Here is the data size zoom 28K 0 48K 1 80K 2 184K 3 340K 4 788K 5 2.1M 6 5.8M 7 19M 8 58M 9 188M10 673M11 17M12 54M13 134M14 375M15 1.1G16 3.0G17 9.6G18 As is obvious, for zoom levels 0-11 I had generated tiles for the entire world, which was essentially empty tiles except the shape of continents since the only data source I used was the nepal osm data Zoom level 12 and onwards, tiles were generated only for nepal. The bigger question i'd be asking is the setup required to generate tiles, and what it would take to do for the entire planet. I guess Braddock would be happy to give access to Jerome on hadron if he wants to give it a try. It is a machine with a 500G SSD and 32G RAM and an 8 core processor. On Tue, Aug 11, 2015 at 6:38 PM, Tim Moody t...@timmoody.com wrote: Anish, you generated tiles for Nepal. How does the size compare to what you got? *From:* unleashk...@googlegroups.com [mailto: unleashk...@googlegroups.com] *On Behalf Of *Tim Moody *Sent:* Tuesday, August 11, 2015 9:04 AM *To:* 'Jérôme Gagnon-Voyer' gagno...@gmail.com *Cc:* 'Adam Holt' h...@laptop.org; 'xsce-devel' xsce-de...@googlegroups.com; 'Unleash Kids!' unleashk...@googlegroups.com; 'server-devel' server-devel@lists.laptop.org *Subject:* RE: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Can I get the data, maybe a dropbox link? not sure of a place to ftp unless you have one. *From:* unleashk...@googlegroups.com [ mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com] *On Behalf Of *Jérôme Gagnon-Voyer *Sent:* Tuesday, August 11, 2015 9:01 AM *To:* Tim Moody t...@timmoody.com *Cc:* Adam Holt h...@laptop.org; xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org *Subject:* Re: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) You define the path to the mbtiles in tilestache.cfg, so yes, they could be in any location (as long as the access rights / permissions are right) On Tue, Aug 11, 2015 at 8:28 AM, Tim Moody t...@timmoody.com wrote: This looks great and not overly complex. XSCE uses one convention that you are probably not aware of and that is that we put data in /library, in this case probably /library/osm or tilestache or some combination. Can tilestache.cfg point to a location different than the html and config files? *From:* unleashk...@googlegroups.com [mailto: unleashk...@googlegroups.com] *On Behalf Of *Adam Holt *Sent:* Tuesday, August 11, 2015 12:19 AM *To:* xsce-devel xsce-de...@googlegroups.com; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org *Subject:* [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Don't stop pushing Jerome- I hope you can demonstrate Nepal-or-larger working models during our Toronto summit Oct 10-12. See you 7AM EDT Thursday on our weekly call, if you have time for our unusually early meeting this week: http://tinyurl.com/xsceminutes (Several of our early design discussions on offline mapping began on unleashk...@googlegroups.com, so am bringing that thread full circle.) On Mon, Aug 10, 2015 at 11:01 PM, Anish Mangal anis...@umich.edu wrote: adding server-devel (the publicly archived mailing list) as well. Great work. Thanks! Happy to help with the testing. I have an intel NUC running an atom processor 2 Ghz, and 2GB RAM - not exactly low end, but still a viable appliance to deploy the xsce. Could you tell me how to get the mbtiles from your ftp. If this works, I can help with creating a playbook - and possibly admin console integration. Thanks, Anish On Tue, Aug 11, 2015 at 8:19 AM, Jérôme Gagnon-Voyer gagno...@gmail.com wrote: Hi all This is about the project to bring a new way of storing OpenStreetMaps offline for internet in a box (schoolserver). (Apologies if I'm using the wrong mailing list,
Re: [Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
I think rendering on the fly is a bit of a misnomer as the metatiles are just concatenated pngs to save on the number of files in the file system, but they aren't compressed and are just rendered tiles. http://wiki.openstreetmap.org/wiki/Meta_tiles. I think Braddock used 4096 tiles per metatile. From: xsce-de...@googlegroups.com [mailto:xsce-de...@googlegroups.com] On Behalf Of Jérôme Gagnon-Voyer Sent: Tuesday, August 11, 2015 1:18 PM To: Anish Mangal anishmangal2...@gmail.com Cc: xsce-devel xsce-de...@googlegroups.com; Anish Mangal anis...@umich.edu; Adam Holt h...@laptop.org; Unleash Kids! unleashk...@googlegroups.com; server-devel server-devel@lists.laptop.org Subject: Re: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) I see i see, so the disk space can't be exactly compared to what we see in .mbtiles, since if you had to transfer that to another server and did not want to do rendering on fly, this would be a different file size (much bigger). Thanks On Tue, Aug 11, 2015 at 12:49 PM, Anish Mangal anishmangal2...@gmail.com mailto:anishmangal2...@gmail.com wrote: I just used the default settings in renderd. As far as I know, I didn't generate any png or jpg tiles, only metatiles were generated, which are renderd as png on the fly. On Tue, Aug 11, 2015 at 8:10 PM, Jérôme Gagnon-Voyer gagno...@gmail.com mailto:gagno...@gmail.com wrote: Hi Anish Regarding file size Would you mind sharing how you generated those tiles? Like the settings used, if its PNG 8 bit, full png or JPG, metatiles size, style used, etc... I'll be sending you and Tim in separate email the link to download the .mbtiles file On Tue, Aug 11, 2015 at 9:35 AM, Anish Mangal anis...@umich.edu mailto:anis...@umich.edu wrote: Hi Here is the data size zoom 28K 0 48K 1 80K 2 184K 3 340K 4 788K 5 2.1M 6 5.8M 7 19M 8 58M 9 188M10 673M11 17M12 54M13 134M14 375M15 1.1G16 3.0G17 9.6G18 As is obvious, for zoom levels 0-11 I had generated tiles for the entire world, which was essentially empty tiles except the shape of continents since the only data source I used was the nepal osm data Zoom level 12 and onwards, tiles were generated only for nepal. The bigger question i'd be asking is the setup required to generate tiles, and what it would take to do for the entire planet. I guess Braddock would be happy to give access to Jerome on hadron if he wants to give it a try. It is a machine with a 500G SSD and 32G RAM and an 8 core processor. On Tue, Aug 11, 2015 at 6:38 PM, Tim Moody t...@timmoody.com mailto:t...@timmoody.com wrote: Anish, you generated tiles for Nepal. How does the size compare to what you got? From: mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com [mailto: mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com] On Behalf Of Tim Moody Sent: Tuesday, August 11, 2015 9:04 AM To: 'Jérôme Gagnon-Voyer' mailto:gagno...@gmail.com gagno...@gmail.com Cc: 'Adam Holt' mailto:h...@laptop.org h...@laptop.org; 'xsce-devel' mailto:xsce-de...@googlegroups.com xsce-de...@googlegroups.com; 'Unleash Kids!' mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com; 'server-devel' mailto:server-devel@lists.laptop.org server-devel@lists.laptop.org Subject: RE: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) Can I get the data, maybe a dropbox link? not sure of a place to ftp unless you have one. From: mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com [ mailto:unleashk...@googlegroups.com mailto:unleashk...@googlegroups.com] On Behalf Of Jérôme Gagnon-Voyer Sent: Tuesday, August 11, 2015 9:01 AM To: Tim Moody mailto:t...@timmoody.com t...@timmoody.com Cc: Adam Holt mailto:h...@laptop.org h...@laptop.org; xsce-devel mailto:xsce-de...@googlegroups.com xsce-de...@googlegroups.com; Unleash Kids! mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com; server-devel mailto:server-devel@lists.laptop.org server-devel@lists.laptop.org Subject: Re: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline) You define the path to the mbtiles in tilestache.cfg, so yes, they could be in any location (as long as the access rights / permissions are right) On Tue, Aug 11, 2015 at 8:28 AM, Tim Moody t...@timmoody.com mailto:t...@timmoody.com wrote: This looks great and not overly complex. XSCE uses one convention that you are probably not aware of and that is that we put data in /library, in this case probably /library/osm or tilestache or some combination. Can tilestache.cfg point to a location different than the html and config files? From: mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com [mailto: mailto:unleashk...@googlegroups.com unleashk...@googlegroups.com] On Behalf Of Adam Holt Sent: Tuesday, August 11, 2015 12:19 AM To: xsce-devel
