Bug#775638: IPv6 database is corrupt
Okay, I sorted it out. The reason the city DB was corrupted is because we were putting so many locations into the location file that we were overflowing the addressable places to put them. The Maxmind format uses 3 bytes to store offsets, so when we exceed offset 0xFF our offsets wrap around to 0 and nothing good happens after that. I added a check for the overflow condition, so the code that creates city DBs errors out instead of silently making a corrupted database. Coincidentally, we were just barely over the size limit for Maxmind's format, which is why a fairly small number of entries were affected. That left the issue of why we were overflowing in the first place. It turns out that a large fraction of the locations in the CSV data aren't actually needed (aren't associated with any IP address). In order to fit the data into a .dat file in Maxmind's format, we have to scan for those and remove them. There are a huge number of them, as I said -- removing them shrinks the .dat file from 31 MB to 20 MB. I've attached a patch which does both of those things. I've built gdnsd successfully with this patch applied and the DBs rebuilt. I've also tested it by running geoiplookup against the database for several addresses. The patch also fixes the area code and metro code for US addresses, which I noticed were backwards (switched with one another). Hope this helps, let me know if you see any issues. -Andrew On Fri, Jan 23, 2015 at 11:52 AM, Andrew Moise andrew.mo...@gmail.com wrote: Hi Patrick. I tracked it down to a few broken entries in the city DB. It looks like the DB creation code makes almost all the entries correctly, but there are a couple of them that come out corrupted: (jessie)moise@localhost:~$ geoiplookup 1.120.146.170 GeoIP Country Edition: AU, Australia GeoIP City Edition, Rev 1: GP, ���-��-��%��%��%��%��- �- �-Ɣ-��%��%�-��-x�-v�-, N/A, N/A, N/A, -180.00, -179.993500, 0, 0 GeoIP ASNum Edition: AS30722 Vodafone Omnitel B.V. Note that's with a local build of the database -- that particular IP address may not be corrupted in the actual jessie database. In that local DB, as in the actual jessie database, almost all the entries are correct, just there are a handful that look like that one. I've been busy the last couple days, but I expect that I'll get enough time today to actually track down what's going wrong and fix it. -Andrew On Jan 23, 2015 9:43 AM, Patrick Matthäi pmatth...@debian.org wrote: Hi Andrew, do you have got any news? :) Am 20.01.2015 um 18:43 schrieb Andrew Moise: Thanks Patrick. One note - just like the v6 issue, the gdnsd test suite is detecting a genuine problem in the city DB. It is something wrong with the city DB creation tools that's causing it (i.e. not just an issue with the gdnsd tests). I just haven't finished tracking down exactly what the issue is yet. -Andrew On Jan 20, 2015 12:22 PM, Patrick Matthäi pmatth...@debian.org wrote: tag #775638 + confirmed clone #775638 -1 reassign -1 geoip-bin retitle -1 geoip-generator produces faulty v6/city database severity -1 grave found -1 1.6.2-3 thanks Hi Am 18.01.2015 um 05:21 schrieb Debian Bug Tracking System: Processing commands for cont...@bugs.debian.org: reassign 775638 geoip-database 20141027-1 Bug #775638 [src:gdnsd] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Bug reassigned from package 'src:gdnsd' to 'geoip-database'. No longer marked as found in versions gdnsd/2.1.0-1. Ignoring request to alter fixed versions of bug #775638 to the same values previously set Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Marked as found in versions geoip-database/20141027-1. retitle 775638 IPv6 database is corrupt Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Changed Bug title to 'IPv6 database is corrupt' from 'gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2' severity 775638 grave Bug #775638 [geoip-database] IPv6 database is corrupt Severity set to 'grave' from 'serious' thanks thanks for spotting it. Curious that no one (also myself!) spotted it. The patch for the v6 database is just: --- geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-19 18:50:04 UTC (rev 5693) +++ geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-20 08:31:03 UTC (rev 5694) @@ -959,6 +959,7 @@ address_family = AF_INET; break; case '6': + database_type = GEOIP_COUNTRY_EDITION_V6; address_family = AF_INET6; break; case 'i': But Andrew found out that there is also an issue with the city database, which is working in general, but the gdnsd tests also fail. -- /* Mit
Bug#775638: IPv6 database is corrupt
Hi Patrick. I tracked it down to a few broken entries in the city DB. It looks like the DB creation code makes almost all the entries correctly, but there are a couple of them that come out corrupted: (jessie)moise@localhost:~$ geoiplookup 1.120.146.170 GeoIP Country Edition: AU, Australia GeoIP City Edition, Rev 1: GP, ���-��-��%��%��%��%��-�-�-Ɣ-��%��%�-��-x�-v�-, N/A, N/A, N/A, -180.00, -179.993500, 0, 0 GeoIP ASNum Edition: AS30722 Vodafone Omnitel B.V. Note that's with a local build of the database -- that particular IP address may not be corrupted in the actual jessie database. In that local DB, as in the actual jessie database, almost all the entries are correct, just there are a handful that look like that one. I've been busy the last couple days, but I expect that I'll get enough time today to actually track down what's going wrong and fix it. -Andrew On Jan 23, 2015 9:43 AM, Patrick Matthäi pmatth...@debian.org wrote: Hi Andrew, do you have got any news? :) Am 20.01.2015 um 18:43 schrieb Andrew Moise: Thanks Patrick. One note - just like the v6 issue, the gdnsd test suite is detecting a genuine problem in the city DB. It is something wrong with the city DB creation tools that's causing it (i.e. not just an issue with the gdnsd tests). I just haven't finished tracking down exactly what the issue is yet. -Andrew On Jan 20, 2015 12:22 PM, Patrick Matthäi pmatth...@debian.org wrote: tag #775638 + confirmed clone #775638 -1 reassign -1 geoip-bin retitle -1 geoip-generator produces faulty v6/city database severity -1 grave found -1 1.6.2-3 thanks Hi Am 18.01.2015 um 05:21 schrieb Debian Bug Tracking System: Processing commands for cont...@bugs.debian.org: reassign 775638 geoip-database 20141027-1 Bug #775638 [src:gdnsd] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Bug reassigned from package 'src:gdnsd' to 'geoip-database'. No longer marked as found in versions gdnsd/2.1.0-1. Ignoring request to alter fixed versions of bug #775638 to the same values previously set Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Marked as found in versions geoip-database/20141027-1. retitle 775638 IPv6 database is corrupt Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Changed Bug title to 'IPv6 database is corrupt' from 'gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2' severity 775638 grave Bug #775638 [geoip-database] IPv6 database is corrupt Severity set to 'grave' from 'serious' thanks thanks for spotting it. Curious that no one (also myself!) spotted it. The patch for the v6 database is just: --- geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-19 18:50:04 UTC (rev 5693) +++ geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-20 08:31:03 UTC (rev 5694) @@ -959,6 +959,7 @@ address_family = AF_INET; break; case '6': + database_type = GEOIP_COUNTRY_EDITION_V6; address_family = AF_INET6; break; case 'i': But Andrew found out that there is also an issue with the city database, which is working in general, but the gdnsd tests also fail. -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */ -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#775638: IPv6 database is corrupt
Hi Andrew, do you have got any news? :) Am 20.01.2015 um 18:43 schrieb Andrew Moise: Thanks Patrick. One note - just like the v6 issue, the gdnsd test suite is detecting a genuine problem in the city DB. It is something wrong with the city DB creation tools that's causing it (i.e. not just an issue with the gdnsd tests). I just haven't finished tracking down exactly what the issue is yet. -Andrew On Jan 20, 2015 12:22 PM, Patrick Matthäi pmatth...@debian.org mailto:pmatth...@debian.org wrote: tag #775638 + confirmed clone #775638 -1 reassign -1 geoip-bin retitle -1 geoip-generator produces faulty v6/city database severity -1 grave found -1 1.6.2-3 thanks Hi Am 18.01.2015 um 05:21 schrieb Debian Bug Tracking System: Processing commands for cont...@bugs.debian.org mailto:cont...@bugs.debian.org: reassign 775638 geoip-database 20141027-1 Bug #775638 [src:gdnsd] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Bug reassigned from package 'src:gdnsd' to 'geoip-database'. No longer marked as found in versions gdnsd/2.1.0-1. Ignoring request to alter fixed versions of bug #775638 to the same values previously set Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Marked as found in versions geoip-database/20141027-1. retitle 775638 IPv6 database is corrupt Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Changed Bug title to 'IPv6 database is corrupt' from 'gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2' severity 775638 grave Bug #775638 [geoip-database] IPv6 database is corrupt Severity set to 'grave' from 'serious' thanks thanks for spotting it. Curious that no one (also myself!) spotted it. The patch for the v6 database is just: --- geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-19 18:50:04 UTC (rev 5693) +++ geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-20 08:31:03 UTC (rev 5694) @@ -959,6 +959,7 @@ address_family = AF_INET; break; case '6': + database_type = GEOIP_COUNTRY_EDITION_V6; address_family = AF_INET6; break; case 'i': But Andrew found out that there is also an issue with the city database, which is working in general, but the gdnsd tests also fail. -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org mailto:pmatth...@debian.org patr...@linux-dev.org mailto:patr...@linux-dev.org */ -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */
Bug#775638: IPv6 database is corrupt
Thanks Patrick. One note - just like the v6 issue, the gdnsd test suite is detecting a genuine problem in the city DB. It is something wrong with the city DB creation tools that's causing it (i.e. not just an issue with the gdnsd tests). I just haven't finished tracking down exactly what the issue is yet. -Andrew On Jan 20, 2015 12:22 PM, Patrick Matthäi pmatth...@debian.org wrote: tag #775638 + confirmed clone #775638 -1 reassign -1 geoip-bin retitle -1 geoip-generator produces faulty v6/city database severity -1 grave found -1 1.6.2-3 thanks Hi Am 18.01.2015 um 05:21 schrieb Debian Bug Tracking System: Processing commands for cont...@bugs.debian.org: reassign 775638 geoip-database 20141027-1 Bug #775638 [src:gdnsd] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Bug reassigned from package 'src:gdnsd' to 'geoip-database'. No longer marked as found in versions gdnsd/2.1.0-1. Ignoring request to alter fixed versions of bug #775638 to the same values previously set Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Marked as found in versions geoip-database/20141027-1. retitle 775638 IPv6 database is corrupt Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2 Changed Bug title to 'IPv6 database is corrupt' from 'gdnsd: FTBFS in jessie: dh_auto_test: make -j1 test returned exit code 2' severity 775638 grave Bug #775638 [geoip-database] IPv6 database is corrupt Severity set to 'grave' from 'serious' thanks thanks for spotting it. Curious that no one (also myself!) spotted it. The patch for the v6 database is just: --- geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-19 18:50:04 UTC (rev 5693) +++ geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-20 08:31:03 UTC (rev 5694) @@ -959,6 +959,7 @@ address_family = AF_INET; break; case '6': + database_type = GEOIP_COUNTRY_EDITION_V6; address_family = AF_INET6; break; case 'i': But Andrew found out that there is also an issue with the city database, which is working in general, but the gdnsd tests also fail. -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */
