Hi Drishtie, thanks for your extensive reply. In general it looks good to me. One point however. Earlier in this thread I mentioned: ' I would be more interested to know whether community buy-in is okay. I haven't seen a response from the Thai community on this posting. I think they should be responding in this discussion before any import.'
Is the local Thai community involved? (still miss them in this thread) Kind regards, Johan 2017-03-21 22:02 GMT+01:00 osm <[email protected]>: > Hello OSM Enthusiasts, > > Firstly, thank you for all the feedback. We really appreciate it. Sorry > for the delay in responding. I have spent the last few days gathering all > the questions and have come up with a list of answers that I hope brings > some clarity. I tried to send this e-mail yesterday but had too many > example pics so it didn’t go through. Please click on the “Example” links > to see pictures we shared to help clarify some of the question around > process. J > > *Can you share DG imagery?* > Our license does not currently permit this. However, to reiterate what > Kevin from Digital Globe said, earlier on this thread, this will not be an > issue in a few weeks when all of OSM will get an imagery refresh. > > *Facebook License with Digital Globe* > This license is necessary to comply with our obligations to Digital Globe, > and we think the benefits of supplying this data outweigh the licensing > restraints. The data will be under ODBL once it has been contributed to OSM > in compliance with the terms. > > There is no contradiction between the OSM terms and the Facebook license > with Digital Globe. Our license terms do not apply to data once contributed > to OSM and termination of the agreement with Digital Globe would not impact > the data that has already been contributed to OSM. > > *Is the Imagery free of Cloud Cover?* > The intended output of +Vivid is cloud free, and in most cases, DG meets > this. There are some highly cloudy parts of the world where it’s > unavoidable. On average though, this product maintains less than 1% cloud > cover. > > *Who are your editors?* > Our team consists of 3 engineers and 14 editors, and usernames can be > found on the Import wiki. All editors have also updated their profile to > say they map for Facebook. You can see their names on their profile. All > our mappers are qualified for editing with backgrounds in GIS, geography > and are extremely technically savvy. We have also spent a considerable > amount of time over the last 5 months training specifically for OSM. > > The mappers are not bound by time limits and metrics as the focus is > primarily on quality. We do however track time taken to edit, density of > tile and who mapped what tile to use this information to plan for future > mapping. > > Facebook does not control these accounts and all users manage their own > profiles and passwords. Once we start mapping we also expect each mapper to > respond to questions from the community. All feedback is discussed as a > group to ensure we are all aware of the feedback and can learn as team. No > other accounts other than the ones listed here and the wiki have been used > to make edits for Facebook. > > Our team includes: > Jaclyn <https://www.openstreetmap.org/user/VLD001> - VLD001 > Annie <https://www.openstreetmap.org/user/VLD002> - VLD002 > Alexandra <https://www.openstreetmap.org/user/VLD003> - VLD003 > Sheffield <https://www.openstreetmap.org/user/VLD004> - VLD004 > Mei <https://www.openstreetmap.org/user/VLD005> - VLD005 > Joseph <https://www.openstreetmap.org/user/VLD006> - VLD006 > Kurt <https://www.openstreetmap.org/user/VLD007> - VLD007 > Pablo Presti-Mohr <https://www.openstreetmap.org/user/VLD008> VLD008 > Goerge <https://www.openstreetmap.org/user/RVR001> - RVR001 > Jeff <https://www.openstreetmap.org/user/RVR002> - RVR002 > David <https://www.openstreetmap.org/user/RVR003> - RVR003 > Yunzhi <https://www.openstreetmap.org/user/RVR004> - RVR004 > Adrian <https://www.openstreetmap.org/user/RVR005> - RVR005 > Stefani <https://www.openstreetmap.org/user/RVR006> - RVR006 > Mohamed <https://www.openstreetmap.org/user/RVR007> - RVR007 > > *Who is behind [email protected] <[email protected]>?* > There are 4 people behind this email. This helps us collaborate as a team > and is the most efficient way for everyone to be on the same page. More > people also ensures that no emails are missed. I (Drishtie Patel) as the > Program Manager primarily watch this email and respond to queries. The > other 3 people are our Engineers namely, Ming Gao and Saikat Basu and the > Product Manager for Maps at Facebook, Sadi Khan. > > *Can the Chageset be more specific?* > Yes we can absolutely change that to import=facebook-ai-____. We don't > specifically have versions but can include the area or task number for > example. Open to suggestions. > > As for adding more details to our edits: there were no changes to current > OSM data in this sample and we plan to add notes when we upload as would > normally happen when using the Tasking Manager and iD. The sample shared > has not been uploaded yet J > > *Can you Publish Source Code for ML* > We cannot share the source code for Machine Learning at this time, but we > can and will share our internal tools. > > *Can you Publish Source Code for internal tools?* > Yes. We are currently using a version of iD Editor > <https://github.com/openstreetmap/iD> and the HOT Tasking Manager > <https://github.com/hotosm/osm-tasking-manager2>. We are absolutely going > to share our tooling. We are currently working on creating a Github where > we will post soon. > > *Can Facebook Share other data?* > This is clear violation of privacy so we cannot share this kind of > information. We are exploring ways to crowdsource road names for OSM and > will share that as we have a more concrete plan. > > *Are you going to Map other countries?* > Yes. We are moving slowly to focus efforts on one country at a time to > make sure our process is accepted by the OSM community out of respect for > the process. > > *Yes we made a mistake* > We've learned from a number of missteps in our initial efforts in Egypt > and Thailand last year. We have since refined our processes, hired a team > of well experienced mappers and are working to follow all the processes and > requests being asked of us. > > *What is the Overall Process?* > *PHASE 1 - Generating Road Masks * > > 1. Training data is created by editors. Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16781235_1170760256386350_6655657355411193856_n.jpg?oh=2a84c16359c99bec01f9d1d23b9241ee&oe=595CF8FC> > > 2. Training data used by engineers in ML to produce road masks. > Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17365429_1592481874110291_3999181234856001536_n.png?oh=df901589c651edc1883bf0472eb1d4ca&oe=59670662> > > 3. Road masks are processed to remove low confidence predictions > and add connections between short breaks. > > > > *PHASE 2 - Creating Road Vectors (.osm files)* > > 1. Road masks are then processed using an algorithm from a black > and white image to a vector format specifically a .osm file. Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16781103_1267857896629755_8804482991712632832_n.png?oh=cb25f91d0e6159b08f6ec342353ae77a&oe=593F3547> > > 2. Our .osm file is then merged with current OSM changesets. During > this process the following things take place > > · All current OSM data for an area is merged, keeping full > history. > > · No changes are made to the current OSM data. > > · Duplicated roads are deleted from the ML generated data. > Current OSM data is always taken over facebook generated data. > > · New ML identified roads are connected to current OSM roads. > > 3. We then have a locally stored file containing both our generated > roads and the current OSM data set for a specific area. > > > *PHASE 3 - Human Validation* > > 1. We use the Tasking manager to divide up tasks and create a flow > of editing and validation for the Facebook mapping team. Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17365380_1448559118508805_536801470968758272_n.jpg?oh=3cb38653fa004f2a20eeaabc2cfa583c&oe=5971703D> > > 2. The Mapper picks a task and loads the locally stored .osm file > generated from the post processing steps outlined above. Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17433416_1595081350521384_2351112230801506304_n.jpg?oh=304c8670744ee2baf27c78043cc6943a&oe=595ECAF0> > > 3. The task is opened up in our enhanced version of iD. (This > includes both the roads generated by us and the current OSM data). Example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16686335_1118257054950682_8271679018604953600_n.jpg?oh=b033e8c22c7b7248a1def85d1733fadc&oe=596EA8D8> > > 4. Our generated roads are highlighted in a different color so > editors can inspect it for issues like crossing highways, disconnected > roads, incorrect intersections, short road stubs, road types, etc., and fix > these issues manually one by one, using Digital Globe satellite imagery > background. > > 5. We do not delete current OSM data, but make typical OSM editing > changes where necessary to ensure high quality of the data (more precisely > follow imagery, consistent tagging, etc). > > 6. Our modified iD tool is equipped with data validation > functionalities similar to JOSM and osmlint https://github.com/osmlab/ > osmlint/. This allows us to check for quality and conflicts with current > OSM data. Some examples of what we check for include: > > · isolated roads that are not connected to rest of the map. > Example-Before > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16781243_1457031487701473_198753992489041920_n.jpg?oh=a3413fae5079f7ce38519d9977b28a9b&oe=5971B86C>. > Example-After > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17433411_1244723028968011_867537480591605760_n.jpg?oh=f5a7cbeebec559fc6c80bfbfeef2d1b6&oe=59255F3A> > > · roads crossing waterways. > > · roads with repeating node IDs. > > · roads with possible connections after a short extension. > Example-Before > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17365393_1561733437200741_5520973609088057344_n.jpg?oh=486151bca24ba30880738c15a1770f87&oe=596A8307>. > Example-After > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17351734_1279516578799625_2079794241860534272_n.jpg?oh=5eb3c2118ae35ab40ddae75f225c7ad8&oe=595C2079> > > 7. In case of conflicts between our newly created roads and other > OSM editor's mapping, our mapper will almost always choose “keep their > edits”. If we think our newly added roads are better aligned with the > latest version of DG satellite imagery, we'll contact the other mapper > offline to reach resolution. If we end up changing other mapper's edits, we > will leave detailed notes to explain why. > > 8. Until all errors are fixed mappers will not be able to save. > Here is an example > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16685255_1430090083699471_1532498821462884352_n.png?oh=8438b1e92957b809f59b874507521bbc&oe=5973316B> > of a highlighted issue prompting the mapper to fix it. > > 9. After fixing all the issues detected, our mapper clicks the > “Save Local” button in the iD tool to save their edited roads locally for > validation. > > 10. A second person, the validator then goes into the same task to > verify that the data looks correct, makes necessary changes as needed and > clicks “Submit” to finally upload the tile to OSM. > > > *Can you share more data samples?* > We have attached a few samples that we are sharing based on the 3 Phases > explained above. We do these in small batches and are not sitting on a > bunch of data for the whole country but are happy to share what we have > created over the last couple of days. > > > > 1. *Phase 1 - Generating Road Masks.* These are the masks of the > initial output from machine learning. These are not processed to a vector > format. We have offered to share our tile service > <https://wiki.openstreetmap.org/wiki/AI-Assisted_Road_Tracing> to anyone. > Sample > 1 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16685603_1205005339598346_4564649276063350784_n.jpg?oh=b2619991eef35541c7a0fb1fd7d3e19a&oe=59276BD6> > | Sample 2 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17351717_1626376154057536_1688258658015641600_n.jpg?oh=ac96567463786eb6047cae69fb9a6c19&oe=59277824> > | Sample 3 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/16686305_1494502210621428_618761533014409216_n.jpg?oh=3316b9f6798e3367c678fb8b93cb808d&oe=595F35AB> > | Sample 4 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17119819_1384749424908927_2865149829640617984_n.jpg?oh=a812255978a6b91933392451b693f5ae&oe=59255A6B> > > 2. *Phase 2 - Creating Road Vectors. *This is the processed road > vectors merged with current OSM data locally - These .osm files that have > NOT been edited or validated by people and would not be uploaded as is. > (Please see attached) > > 3. *Phase 3 - Human Validation. *This is the final output validated > by mappers twice in order to be uploaded to OSM. While we are happy to > share this please note the time taken to validate, share and wait for > feedback would be counter productive because the current OSM data may be > changed during this time. This increases the possibility for conflicts. We > plan to map small areas at a time to ensure we are working with current OSM > data and validating conflicts quickly and smoothly as we add our generated > roads. Sample 1 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17351465_1228329023883542_5416323939192274944_n.jpg?oh=d34ccfb8c092940bdadf4b322a9bb33f&oe=59720FD0> > | Sample 2 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17365486_1399125510131415_7405155781655396352_n.jpg?oh=786797c91760722e701eabb66ad37a65&oe=5962161B> > | Sample 3 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17365310_1444333455611506_7249929757997924352_n.jpg?oh=a5f90f3d7f0ec4c6f5224d766da0be5c&oe=596697D0> > | Sample 4 > <http://scontent.xx.fbcdn.net/v/t39.10482-6/17351405_1379847552085633_543677743674425344_n.jpg?oh=6f9f2c7c4e18736c28a7da8e5f2f46ea&oe=59581CCC> > > > > For the attached samples, please note these are not edited and would not > in any way be uploaded to OSM as is. We have also provided screenshots for > Phase 1 and Phase 3 as it would appear in our internal tools with the DG > imagery before being edited. > > Look forward to more feedback :) > > Best, > Drishtie Patel on behalf of the OSM at Facebook Team > > OSM Profile <https://www.openstreetmap.org/user/DrishT> > > > > _______________________________________________ > Imports mailing list > [email protected] > https://lists.openstreetmap.org/listinfo/imports > >
_______________________________________________ Imports mailing list [email protected] https://lists.openstreetmap.org/listinfo/imports
