Hi All, India data from Microsoft's ML Buildings Footprints data release is deployed on both vector and raster tile layers: https://server.nikhilvj.co.in/buildings1/ This is a TileServer GL instance.
Recipe for how this was done and the configs etc reqd is shared here: https://github.com/answerquest/maptiles_recipe_buildings This process may be useful for working with other large datasets also. To overlay on JOSM, pls use this PNG layer: https://server.nikhilvj.co.in/buildings1/styles/basic/{z}/{x}/{y}.png For raster overlay in browser, pls use webp format which is smaller and faster: https://server.nikhilvj.co.in/buildings1/styles/basic/{z}/{x}/{y}.webp For vector tiles (will work in QGIS also), pls use this: https://server.nikhilvj.co.in/buildings1/data/india_buildings_z14/{z}/{x}/{y}.pbf There was a lot of troubleshooting needed for getting the raster output working - big thanks to Deepak Sharda for help in that. Meanwhile, I've also completed importing the data to a postgis DB. If anyone wants data for a limited region, pls reach me on nikhil.js [at] gmail.com -- Cheers, Nikhil VJ https://nikhilvj.co.in On Thu, May 26, 2022 at 7:07 PM Deepak Sharda <[email protected]> wrote: > To make folder structure mbtile you might need to add ''-e'' in the > recipe. > > or tile is made you can use Mbutil to create folder structure from single > file. > > > To your second query to create building footprints with rater tiles there > are two ways two approach it. > > You can create raster tiles from geotiff by using tiles2py in folder > structure. > > But in your case you can use Tileserver-gl what it does is it can host > vector tiles but serve them as raster. > > You can check config file and change as per your requirements such as > 256x256 or 512x512. > > > let me know if you still face issue. > > Note - Go by vector tiles its fast and you can give 3D effect as you have > seen in our app. > > > Thanks and Regards > > On Thu, 26 May, 2022, 6:39 pm Nikhil VJ, <[email protected]> wrote: > >> Hi Deepak, >> >> Thanks for the suggestion. >> >> I came across tippecanoe again : https://github.com/mapbox/tippecanoe >> and finally seeing a use case for it. So, while the DB is still being >> loaded, started off tippecanoe program on the original .geojsonl. >> >> The program was quite fast - in about 6 hrs it had generated a 1.4GB >> .mbtiles file and it didn't even take too much RAM. >> >> Then, with the help of https://github.com/maptiler/tileserver-php: >> This vector tiles URL is working at my end in QGIS: >> >> https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf >> >> I was looking for options to generate raster .png etc tiles, but haven't >> found one yet. >> >> I also saw an option in tippecanoe to output .pbf tiles to a folder >> instead of a single .mbtiles file. Did a couple of runs, but that hasn't >> worked out. >> >> Meanwhile, Sreeram-K posted in OSM-India telegram group: folks at >> mapwith.ai have incorporated the data and are now showing it in their >> rapID editor alongwith the AI roads data, as suggestions for openstreetmap >> editing. One link: >> >> https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361 >> >> >> -- >> Cheers, >> Nikhil VJ >> https://nikhilvj.co.in >> >> >> >> On Tue, May 24, 2022 at 1:48 PM Deepak Sharda <[email protected]> >> wrote: >> >>> Hello Nikhil, >>> >>> I would definitely love to collobrate. The best i would suggest is to >>> make Mbtiles and host them as data is not going to change a lot so most of >>> the time static. >>> >>> Rendering from hosted Mbtiles will not only fast but easy to handle on >>> small web server. >>> >>> How to proceed. >>> >>> First creation of Mbtiles >>> second Hosting of Mbtiles >>> Third Styling at client side. >>> >>> >>> Thanks >>> >>> >>> >>> On Tue, 24 May, 2022, 1:03 pm Nikhil VJ, <[email protected]> wrote: >>> >>>> Hi Bhibhash, >>>> >>>> Short answer: No. >>>> There is no metadata. I took the top 1000 lines and visualized it : the >>>> shapes were spread all over the country and there wasn't any order in them. >>>> >>>> With an intern's help I've started import of them into a portable >>>> postgresql DB (dockerized with a persistent volume) on my webserver. >>>> Started yesterday and it's 10 million shapes in; from size comparisons >>>> it'll probably be around 79 million shapes, so should be imported in a >>>> week's time. >>>> >>>> I've set an auto-increment primary key column in the DB table, so once >>>> imported at least the data can be pulled out in batches. >>>> >>>> There are 2 paths ahead once I've imported it: >>>> ------- >>>> >>>> *1. Serving vector tiles* >>>> I want to serve MVT vector tiles from this DB - and then have a map >>>> display the same. That way one can directly have this stuff render on a map >>>> as they browse it - fetching only one small region at a time and hopefully >>>> not being too slow. >>>> >>>> Finding some references here: >>>> https://blog.jawg.io/how-to-make-mvt-with-postgis/ >>>> https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis >>>> >>>> https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084 >>>> >>>> I'm new to this MVT business, so if anybody wants to collaborate pls >>>> feel free to jump in. >>>> >>>> ------ >>>> >>>> *2. Tag with region ids* >>>> Add columns to the DB table and tag each shape by state, district ids >>>> etc using Within operations either in the DB or in python-geopandas. >>>> >>>> >>>> -- >>>> Cheers, >>>> Nikhil VJ >>>> https://nikhilvj.co.in >>>> >>>> >>>> On Tue, May 24, 2022 at 10:14 AM Bibhash Nath <[email protected]> >>>> wrote: >>>> >>>>> Thanks for sharing. >>>>> Do we have state-wise datasets? It seems quite large to process on my >>>>> computer. >>>>> >>>>> Best, >>>>> Bibhash >>>>> >>>>> On Wed, May 18, 2022 at 4:58 AM Nikhil VJ <[email protected]> wrote: >>>>> >>>>>> Thanks Justin for sharing! >>>>>> >>>>>> I've downloaded and extracted the India.geojsonl on a webserver. >>>>>> >>>>>> ref about .geojsonl : >>>>>> https://www.interline.io/blog/geojsonl-extracts/ >>>>>> -> pretty useful! One can loop through a huge file without having to >>>>>> load it all into RAM. >>>>>> >>>>>> top lines look like: >>>>>> {"type": "Feature", "properties": {},"geometry": {"type": >>>>>> "Polygon","coordinates": [[[83.06380515611697, 25.34167404697847], >>>>>> [83.06380909901775, 25.341635591519122], [83.06386494585949, >>>>>> 25.341640268589657], [83.06386100295869, 25.341678724047526], >>>>>> [83.06380515611697, 25.34167404697847]]]}} >>>>>> {"type": "Feature", "properties": {},"geometry": {"type": >>>>>> "Polygon","coordinates": [[[87.87555977691633, 22.397660095199], >>>>>> [87.8754256865811, 22.397658576750985], [87.87542690191553, >>>>>> 22.397566835386712], [87.87556099225075, 22.39756835383578], >>>>>> [87.87555977691633, 22.397660095199]]]}} >>>>>> ... >>>>>> So we have just basic polygons, one for each building, no properties >>>>>> or categorization. >>>>>> >>>>>> Here are some ideas on what to do with this: >>>>>> >>>>>> 1. Postgresql DB: >>>>>> - Load all of these into a PostGreSQL DB >>>>>> - Setup an api that will take a lat/lon and give all shapes within >>>>>> 1km radius >>>>>> - Next possible api: send a bounding polygon and get all buildings in >>>>>> that >>>>>> >>>>>> 2. Split up by district or lower level admin boundaries: >>>>>> - Load the admin boundaries into a Postgresql DB >>>>>> - Loop through each line (ie each building) >>>>>> - Find out which place it's in by ST_Within query >>>>>> - Dump it into a separate .geojsonl for that place >>>>>> - We've now split it into multiple smaller files that are more >>>>>> usable, can be loaded up by OSM mappers etc. >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> Nikhil VJ >>>>>> https://nikhilvj.co.in >>>>>> >>>>>> >>>>>> On Wed, May 18, 2022 at 3:51 AM Justin <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> [image: lg.png] >>>>>>> >>>>>>> https://github.com/microsoft/GlobalMLBuildingFootprints >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>> more about us by visiting http://datameet.org >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "datameet" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>> more about us by visiting http://datameet.org >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "datameet" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>> more about us by visiting http://datameet.org >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "datameet" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> Datameet is a community of Data Science enthusiasts in India. Know more >>>> about us by visiting http://datameet.org >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "datameet" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> Datameet is a community of Data Science enthusiasts in India. Know more >>> about us by visiting http://datameet.org >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "datameet" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> Datameet is a community of Data Science enthusiasts in India. Know more >> about us by visiting http://datameet.org >> --- >> You received this message because you are subscribed to the Google Groups >> "datameet" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/datameet/CAH7jeuMh1dDpd%2BH8RNDGUPbitt5hr3ZEZMSf6w7qehjQyB9Gyg%40mail.gmail.com >> <https://groups.google.com/d/msgid/datameet/CAH7jeuMh1dDpd%2BH8RNDGUPbitt5hr3ZEZMSf6w7qehjQyB9Gyg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/CA%2B4SsQDbj2pQ8Ehk8JTHCMmoTAqMCBTEBevunpgYCMkon8oC6Q%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CA%2B4SsQDbj2pQ8Ehk8JTHCMmoTAqMCBTEBevunpgYCMkon8oC6Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuPi2aX0ruuLHobpdZWqCayqfKSGFd66yo8BcncxfhrgqA%40mail.gmail.com.
