Just finished chatting with Frank Warmerdam on #gdal. I'll paste below the full conversion for the details.
The summary is that just as Jon notes, the shapefile dbf has bogus/ null characters perhaps due to creation from geoserver/geotools. This caused mapnik to fail in properly filter the attributes, while the shapelib derived shapefile reader in MapServer is able to handle this type of situation. The conversion/copying of the file using ogr2ogr also effectively fixed the shapefile because GDAL/OGR will correct any 'winding' issues with polygons. So, for now I feel confident continuing to use an 'OGR fixed' version of this shapefile to develop benchmarking tests of mapnik against mapserver and geoserver. I'll also file a mapnik ticket now to capture this issue so that future effort on the shapefile reason to attempt to handle this situation. done: https://trac.mapnik.org/ticket/132 Cheers, Dane FrankW: springmeyer: I'll take a peek. [4:41pm] springmeyer: FrankW: cool: looks like john burgess is getting close: http://www.nabble.com/Mapnik-Filter-Expression-Failure-tt20141006.html#a20141526 [4:41pm] FrankW: springmeyer: I don't see obvious differences. I will say that the .dbf header structure is pretty much identical. [4:41pm] FrankW: I will say that ogr2ogr will try and correct any winding issues with polygons. [4:42pm] FrankW: Ah null chars in a .dbf are inappropriate. [4:42pm] FrankW: Whatever wrote that dbf was not being well behaved. [4:44pm] FrankW: I wonder if the .dbf was written by geoserver or geotools based code. [4:44pm] • springmeyer is back... [4:44pm] springmeyer: yes, very likely [4:44pm] springmeyer: I just downloaded it yesterday linked off pramsey's post [4:44pm] springmeyer: it is the geoserver 'test shapefile' for the gs vs. mapserver trials at foss4g2008 [4:45pm] springmeyer: it seems like mapserver handles it fine [4:45pm] springmeyer: perhaps that is because mapserver uses ogr to read shapefiles? [4:45pm] springmeyer: mapnik has its own shapefile reader... ugh. [4:46pm] FrankW: MapServer normally uses it's own shapefile reader derived from an early version of shapelib, though it can also do so through OGR with some loss of speed. [4:46pm] springmeyer: ah, interesting. [4:46pm] FrankW: Perhaps MapServer just has a better shapefile reader than mapnik. [4:46pm] springmeyer: likely [4:47pm] springmeyer: curious though to thinking about the 'definitive' tests benchmarking mapserver vs. geoserver using a shapefile with a suspect dbf... [4:47pm] springmeyer: thinking/think [4:48pm] FrankW: Indeed [4:49pm] springmeyer: so the shapelib based driver for mapserver is default then? [4:50pm] FrankW: yes [4:50pm] FrankW: To be clear it is a reader *derived* from an early version of shapelib and then modified substantially. [4:50pm] springmeyer: okay, cool. well I don't see any indication of using ogr in the mapfile the geoserver guys created: http://mapnik-utils.googlecode.com/svn/trunk/sandbox/bench.map [4:51pm] springmeyer: thats my local copy of it for testing with mapnik [4:53pm] • springmeyer feels confident in simply fixing the shapefile with ogr2ogr now for running the tests... [4:53pm] springmeyer: thanks so much! On Oct 23, 2008, at 4:25 PM, Jon Burgess wrote: > On Fri, 2008-10-24 at 00:18 +0100, Jon Burgess wrote: >> On Thu, 2008-10-23 at 15:42 -0700, Dane Springmeyer wrote: >>> Mapnik Users, >>> >>> I've run into a repeatable failure (on both mac 10.5 and Ubuntu >>> 8.04) >>> of the mapnik Filter expression and have a test case. >>> >>> >>> You can download the test case as a zip archive from here: >>> http://mapnik-utils.googlecode.com/svn/trunk/sandbox/mapnik_filter_failure_test_case.zip >>> >>> Inside the zip archive are two shapefiles, two XML mapfiles, and a >>> test script. >>> >>> The run_test.sh script utilizes nik2img to generate a simple graphic >>> using each mapfile. >>> >>> The only difference between the mapfiles is that one points to a >>> 'states.shp' shapefile which mapnik renders fully green highlighting >>> the failure of the <Filter> expression, while the other points to >>> 'states_working.shp' which for me renders with states in shades of >>> green, blue or red, highlighting a working Filter on the [PEOPLE] >>> attribute. >> >> The test script isn't working for me, but doing a simple binary >> comparison between the files shows that only the dbf file are >> different. >> We can discount the prj file since that is not read by Mapnik. >> >> If I dump the the two dbf files with dbfdump (from shapelib >> utilities) >> then the results are the same for the two files. The difference >> must be >> in something very subtle like the padding bytes. >> >> It either needs some extra debug code in the Mapnik shapefile >> reader to >> compare exactly what is read from the two files. Or if someone is >> familiar with the DBF format might be able to decode the differences >> byte-by-byte. > > Looking closer the padding on the numeric fields is definitely > different > nul (broken) vs space (working): > > > 0001320 13 09 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >................< > 0001340 0d 20 49 6c 6c 69 6e 6f 69 73 20 20 20 20 20 20 >. > Illinois < > 0001360 20 20 20 20 20 20 20 20 20 20 20 31 37 45 20 4e > > 17E N< > -0001400 20 43 65 6e 49 4c 31 34 33 39 38 36 2e 36 31 30 > > CenIL143986.610< > -0001420 30 30 30 30 30 30 00 00 00 31 39 39 33 2e 33 33 > >000000...1993.33< > -0001440 35 30 30 30 30 30 30 00 00 00 00 00 31 31 34 33 > >5000000.....1143< > -0001460 30 36 30 32 2e 30 30 30 30 30 30 30 30 30 00 32 > >0602.000000000.2< > -0001500 39 32 34 38 38 30 2e 30 30 30 30 30 30 30 30 30 > >924880.000000000< > -0001520 00 00 34 32 30 32 32 34 30 2e 30 30 30 30 30 30 >.. > 4202240.000000< > -0001540 30 30 30 00 00 35 35 35 32 32 33 33 2e 30 30 30 > >000..5552233.000< > ... > +0001400 20 43 65 6e 49 4c 20 20 20 31 34 33 39 38 36 2e > CenIL > 143986.< > +0001420 36 31 30 30 30 30 30 30 30 20 20 20 20 20 31 39 > >610000000 19< > +0001440 39 33 2e 33 33 35 30 30 30 30 30 30 20 31 31 34 > >93.335000000 114< > +0001460 33 30 36 30 32 2e 30 30 30 30 30 30 30 30 30 20 > >30602.000000000 < > +0001500 20 32 39 32 34 38 38 30 2e 30 30 30 30 30 30 30 > > 2924880.0000000< > +0001520 30 30 20 20 34 32 30 32 32 34 30 2e 30 30 30 30 >00 > 4202240.0000< > +0001540 30 30 30 30 30 20 20 35 35 35 32 32 33 33 2e 30 >00000 > 5552233.0< > +0001560 30 30 30 30 30 30 30 30 20 20 35 38 37 38 33 36 >00000000 > 587836< > > Not sure why this should confuse Mapnik though. > > Jon > _______________________________________________ Mapnik-users mailing list [email protected] https://lists.berlios.de/mailman/listinfo/mapnik-users
