Hi Deepak, Thanks for the suggestion.
I came across tippecanoe again : https://github.com/mapbox/tippecanoe and finally seeing a use case for it. So, while the DB is still being loaded, started off tippecanoe program on the original .geojsonl. The program was quite fast - in about 6 hrs it had generated a 1.4GB .mbtiles file and it didn't even take too much RAM. Then, with the help of https://github.com/maptiler/tileserver-php: This vector tiles URL is working at my end in QGIS: https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf I was looking for options to generate raster .png etc tiles, but haven't found one yet. I also saw an option in tippecanoe to output .pbf tiles to a folder instead of a single .mbtiles file. Did a couple of runs, but that hasn't worked out. Meanwhile, Sreeram-K posted in OSM-India telegram group: folks at mapwith.ai have incorporated the data and are now showing it in their rapID editor alongwith the AI roads data, as suggestions for openstreetmap editing. One link: https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361 -- Cheers, Nikhil VJ https://nikhilvj.co.in On Tue, May 24, 2022 at 1:48 PM Deepak Sharda <deepsha...@gmail.com> wrote: > Hello Nikhil, > > I would definitely love to collobrate. The best i would suggest is to make > Mbtiles and host them as data is not going to change a lot so most of the > time static. > > Rendering from hosted Mbtiles will not only fast but easy to handle on > small web server. > > How to proceed. > > First creation of Mbtiles > second Hosting of Mbtiles > Third Styling at client side. > > > Thanks > > > > On Tue, 24 May, 2022, 1:03 pm Nikhil VJ, <nikhil...@gmail.com> wrote: > >> Hi Bhibhash, >> >> Short answer: No. >> There is no metadata. I took the top 1000 lines and visualized it : the >> shapes were spread all over the country and there wasn't any order in them. >> >> With an intern's help I've started import of them into a portable >> postgresql DB (dockerized with a persistent volume) on my webserver. >> Started yesterday and it's 10 million shapes in; from size comparisons >> it'll probably be around 79 million shapes, so should be imported in a >> week's time. >> >> I've set an auto-increment primary key column in the DB table, so once >> imported at least the data can be pulled out in batches. >> >> There are 2 paths ahead once I've imported it: >> ------- >> >> *1. Serving vector tiles* >> I want to serve MVT vector tiles from this DB - and then have a map >> display the same. That way one can directly have this stuff render on a map >> as they browse it - fetching only one small region at a time and hopefully >> not being too slow. >> >> Finding some references here: >> https://blog.jawg.io/how-to-make-mvt-with-postgis/ >> https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis >> >> https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084 >> >> I'm new to this MVT business, so if anybody wants to collaborate pls feel >> free to jump in. >> >> ------ >> >> *2. Tag with region ids* >> Add columns to the DB table and tag each shape by state, district ids etc >> using Within operations either in the DB or in python-geopandas. >> >> >> -- >> Cheers, >> Nikhil VJ >> https://nikhilvj.co.in >> >> >> On Tue, May 24, 2022 at 10:14 AM Bibhash Nath <bibhas...@gmail.com> >> wrote: >> >>> Thanks for sharing. >>> Do we have state-wise datasets? It seems quite large to process on my >>> computer. >>> >>> Best, >>> Bibhash >>> >>> On Wed, May 18, 2022 at 4:58 AM Nikhil VJ <nikhil...@gmail.com> wrote: >>> >>>> Thanks Justin for sharing! >>>> >>>> I've downloaded and extracted the India.geojsonl on a webserver. >>>> >>>> ref about .geojsonl : https://www.interline.io/blog/geojsonl-extracts/ >>>> -> pretty useful! One can loop through a huge file without having to >>>> load it all into RAM. >>>> >>>> top lines look like: >>>> {"type": "Feature", "properties": {},"geometry": {"type": >>>> "Polygon","coordinates": [[[83.06380515611697, 25.34167404697847], >>>> [83.06380909901775, 25.341635591519122], [83.06386494585949, >>>> 25.341640268589657], [83.06386100295869, 25.341678724047526], >>>> [83.06380515611697, 25.34167404697847]]]}} >>>> {"type": "Feature", "properties": {},"geometry": {"type": >>>> "Polygon","coordinates": [[[87.87555977691633, 22.397660095199], >>>> [87.8754256865811, 22.397658576750985], [87.87542690191553, >>>> 22.397566835386712], [87.87556099225075, 22.39756835383578], >>>> [87.87555977691633, 22.397660095199]]]}} >>>> ... >>>> So we have just basic polygons, one for each building, no properties or >>>> categorization. >>>> >>>> Here are some ideas on what to do with this: >>>> >>>> 1. Postgresql DB: >>>> - Load all of these into a PostGreSQL DB >>>> - Setup an api that will take a lat/lon and give all shapes within 1km >>>> radius >>>> - Next possible api: send a bounding polygon and get all buildings in >>>> that >>>> >>>> 2. Split up by district or lower level admin boundaries: >>>> - Load the admin boundaries into a Postgresql DB >>>> - Loop through each line (ie each building) >>>> - Find out which place it's in by ST_Within query >>>> - Dump it into a separate .geojsonl for that place >>>> - We've now split it into multiple smaller files that are more usable, >>>> can be loaded up by OSM mappers etc. >>>> >>>> -- >>>> Cheers, >>>> Nikhil VJ >>>> https://nikhilvj.co.in >>>> >>>> >>>> On Wed, May 18, 2022 at 3:51 AM Justin <justinelliotmey...@gmail.com> >>>> wrote: >>>> >>>>> [image: lg.png] >>>>> >>>>> https://github.com/microsoft/GlobalMLBuildingFootprints >>>>> >>>>> >>>>> -- >>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>> more about us by visiting http://datameet.org >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "datameet" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to datameet+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> Datameet is a community of Data Science enthusiasts in India. Know more >>>> about us by visiting http://datameet.org >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "datameet" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to datameet+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> Datameet is a community of Data Science enthusiasts in India. Know more >>> about us by visiting http://datameet.org >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "datameet" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to datameet+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> Datameet is a community of Data Science enthusiasts in India. Know more >> about us by visiting http://datameet.org >> --- >> You received this message because you are subscribed to the Google Groups >> "datameet" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to datameet+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com >> <https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to the Google Groups > "datameet" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to datameet+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuMh1dDpd%2BH8RNDGUPbitt5hr3ZEZMSf6w7qehjQyB9Gyg%40mail.gmail.com.
