[OSM-dev] Problem with hourly/daily diffs! was: Re: osmium check-refs failing on latest assembled PBF with hourly changes
Hi! There was a problem with hourly and daily diffs missing some changes on Nov 11 2020. If you are digesting hourly or daily diffs and updating a planet file or database regularly, you are affected by this and need to take action or your planet file/database will be incomplete. If you are using minutely diffs, you are okay. This doesn't apply there. The diffs have been fixed by the sysadmins now (Thanks Tom!). This means that everybody who is affected by this can fix their systems by going back and re-applying diffs. To fix your planet file/database, you have to go back to hourly diff 72011 and re-apply the hourly diffs from there. Or go back to daily diff 2999 and re-apply the daily diffs from there. It is not enough to just apply the problematic minutely diffs or the one hourly/daily diff, you have to reapply all diffs after that in the right order! Jochen On Wed, Dec 02, 2020 at 11:38:39AM +0100, Jochen Topf wrote: > Date: Wed, 2 Dec 2020 11:38:39 +0100 > From: Jochen Topf > To: Manohar Erikipati via dev > Subject: Re: [OSM-dev] osmium check-refs failing on latest assembled PBF > with hourly changes > > Hi! > > There is a problem with the hourly diffs. From my analysis it looks like > minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925 > is in hourly 72013, but the two minutely diffs in between (4304923 and > 4304924) are in neither. So you are definitely missing some data and > osmium check-refs found part of that. > > Sysadmins are aware of the problem and maybe we going to get a fixed > hourly file. If not you should be able to fix this by applying those two > minutely diffs and then re-applying all the hourly diffs from that > point. > > Jochen > > On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote: > > Date: Tue, 1 Dec 2020 11:53:49 -0500 > > From: Manohar Erikipati via dev > > To: dev@openstreetmap.org > > Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with > > hourly changes > > > > Hi, we have been applying new hourly changesets to create a daily PBF. This > > works very well and it is continuously stable. Recently, when the hourly > > changes were applied, I noticed a failure in osmium check-refs. > > > > Hourly file changes applied are from these files > > > > from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz > > > > Updating sequence number from 71849 to 72051 > > > > Running a osmium check-refs * on the updated PBF results in several > > referential check failures for nodes. Nodes that belong to below OSM ways. > > When I looked into these OSM ways through JOSM, I could not spot any > > problems. > > > > Ways > > > > https://www.openstreetmap.org/way/185306812 > > > > https://www.openstreetmap.org/way/36960460 > > > > https://www.openstreetmap.org/way/326450137 > > > > https://www.openstreetmap.org/way/872325295 > > > > https://www.openstreetmap.org/way/878901824 > > > > https://www.openstreetmap.org/way/878968187 > > > > https://www.openstreetmap.org/way/878968195 > > > > Example nodes > > > > https://www.openstreetmap.org/node/8176907685 > > > > https://www.openstreetmap.org/node/8176907689 > > > > https://www.openstreetmap.org/node/8176907690 > > > > https://www.openstreetmap.org/node/8176907686 > > > > https://www.openstreetmap.org/node/8176968661 > > > > https://www.openstreetmap.org/node/8176907653 > > > > Could someone help if they have observed any missing data in the hourly > > files or if osmium check-refs itself is not highlighting any problematic > > relation. > > > > Any help is appreciated, thanks. > > > > Manohar > > > > https://www.openstreetmap.org/user/manoharuss > > > ___ > > dev mailing list > > dev@openstreetmap.org > > https://lists.openstreetmap.org/listinfo/dev > > > -- > Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium check-refs failing on latest assembled PBF with hourly changes
Hi Jochen, applying the 2 minutely files to the PBF worked nicely and passed osmium check-refs. Seems that is the only cause. Thanks for looking into it and sharing the minutely files. Manohar On Wed, Dec 2, 2020 at 5:41 AM Jochen Topf wrote: > Hi! > > There is a problem with the hourly diffs. From my analysis it looks like > minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925 > is in hourly 72013, but the two minutely diffs in between (4304923 and > 4304924) are in neither. So you are definitely missing some data and > osmium check-refs found part of that. > > Sysadmins are aware of the problem and maybe we going to get a fixed > hourly file. If not you should be able to fix this by applying those two > minutely diffs and then re-applying all the hourly diffs from that > point. > > Jochen > > On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote: > > Date: Tue, 1 Dec 2020 11:53:49 -0500 > > From: Manohar Erikipati via dev > > To: dev@openstreetmap.org > > Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with > > hourly changes > > > > Hi, we have been applying new hourly changesets to create a daily PBF. > This > > works very well and it is continuously stable. Recently, when the hourly > > changes were applied, I noticed a failure in osmium check-refs. > > > > Hourly file changes applied are from these files > > > > from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz > > > > Updating sequence number from 71849 to 72051 > > > > Running a osmium check-refs * on the updated PBF results in several > > referential check failures for nodes. Nodes that belong to below OSM > ways. > > When I looked into these OSM ways through JOSM, I could not spot any > > problems. > > > > Ways > > > > https://www.openstreetmap.org/way/185306812 > > > > https://www.openstreetmap.org/way/36960460 > > > > https://www.openstreetmap.org/way/326450137 > > > > https://www.openstreetmap.org/way/872325295 > > > > https://www.openstreetmap.org/way/878901824 > > > > https://www.openstreetmap.org/way/878968187 > > > > https://www.openstreetmap.org/way/878968195 > > > > Example nodes > > > > https://www.openstreetmap.org/node/8176907685 > > > > https://www.openstreetmap.org/node/8176907689 > > > > https://www.openstreetmap.org/node/8176907690 > > > > https://www.openstreetmap.org/node/8176907686 > > > > https://www.openstreetmap.org/node/8176968661 > > > > https://www.openstreetmap.org/node/8176907653 > > > > Could someone help if they have observed any missing data in the hourly > > files or if osmium check-refs itself is not highlighting any problematic > > relation. > > > > Any help is appreciated, thanks. > > > > Manohar > > > > https://www.openstreetmap.org/user/manoharuss > > > ___ > > dev mailing list > > dev@openstreetmap.org > > https://lists.openstreetmap.org/listinfo/dev > > > -- > Jochen Topf joc...@remote.org https://www.jochentopf.com/ > +49-351-31778688 > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osmium check-refs failing on latest assembled PBF with hourly changes
Hi! There is a problem with the hourly diffs. From my analysis it looks like minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925 is in hourly 72013, but the two minutely diffs in between (4304923 and 4304924) are in neither. So you are definitely missing some data and osmium check-refs found part of that. Sysadmins are aware of the problem and maybe we going to get a fixed hourly file. If not you should be able to fix this by applying those two minutely diffs and then re-applying all the hourly diffs from that point. Jochen On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote: > Date: Tue, 1 Dec 2020 11:53:49 -0500 > From: Manohar Erikipati via dev > To: dev@openstreetmap.org > Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with > hourly changes > > Hi, we have been applying new hourly changesets to create a daily PBF. This > works very well and it is continuously stable. Recently, when the hourly > changes were applied, I noticed a failure in osmium check-refs. > > Hourly file changes applied are from these files > > from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz > > Updating sequence number from 71849 to 72051 > > Running a osmium check-refs * on the updated PBF results in several > referential check failures for nodes. Nodes that belong to below OSM ways. > When I looked into these OSM ways through JOSM, I could not spot any > problems. > > Ways > > https://www.openstreetmap.org/way/185306812 > > https://www.openstreetmap.org/way/36960460 > > https://www.openstreetmap.org/way/326450137 > > https://www.openstreetmap.org/way/872325295 > > https://www.openstreetmap.org/way/878901824 > > https://www.openstreetmap.org/way/878968187 > > https://www.openstreetmap.org/way/878968195 > > Example nodes > > https://www.openstreetmap.org/node/8176907685 > > https://www.openstreetmap.org/node/8176907689 > > https://www.openstreetmap.org/node/8176907690 > > https://www.openstreetmap.org/node/8176907686 > > https://www.openstreetmap.org/node/8176968661 > > https://www.openstreetmap.org/node/8176907653 > > Could someone help if they have observed any missing data in the hourly > files or if osmium check-refs itself is not highlighting any problematic > relation. > > Any help is appreciated, thanks. > > Manohar > > https://www.openstreetmap.org/user/manoharuss > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
