[mapserver-users] how does mapserver determine shapefile encoding?
Hello, I have a shapefile from which I don't really know the encoding. It might be latin1 or ISO-8559-5. The good thing is, when MapServer is accessing the shapefile, the labels are shown correct. QGis is also displaying everything fine. But when I try to convert with shp2pgsql or ogr I get only garbage back. I tried different source encodings. Since I need the data to be in the database I'm now trying to find out where the problem is. I tried converting the data on different systems (Ubuntu, RedHat), with postgis/shp2pgsql from repository as well as self compiled. Same results. Also iconv and recode wasn't a help. As I said QGis is displaying everything fine it fails in converting the shapefile too. Opening the dbf in LibreOffice doesn't helped either. Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? Thanks in advance for any pointers. Lars ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] how does mapserver determine shapefile encoding?
Lars, When you try to load some data to a PostgreSQL server, you can specify the encoding to use. This can be done with client_encoding. For example, if you want to populate a table with some CSV file contents, you have to do the following: SET client_encoding = 'LATIN1'; COPY my_table FROM '/usr/file.csv' WITH CSV Now, when you use shp2pgsql you can't set the client encoding, but you can set the default client_encoding of your database and every data transfer between some client and your database will be encoded properly: ALTER DATABASE my_database SET client_encoding = 'LATIN1' Hope this helps Cheers from Guadalajara, México IC Carlos Ruiz From: Lars Lingner gislars+l...@googlemail.com To: 'mapserver-users@lists.osgeo.org' mapserver-users@lists.osgeo.org Sent: Tuesday, August 2, 2011 2:45 PM Subject: [mapserver-users] how does mapserver determine shapefile encoding? Hello, I have a shapefile from which I don't really know the encoding. It might be latin1 or ISO-8559-5. The good thing is, when MapServer is accessing the shapefile, the labels are shown correct. QGis is also displaying everything fine. But when I try to convert with shp2pgsql or ogr I get only garbage back. I tried different source encodings. Since I need the data to be in the database I'm now trying to find out where the problem is. I tried converting the data on different systems (Ubuntu, RedHat), with postgis/shp2pgsql from repository as well as self compiled. Same results. Also iconv and recode wasn't a help. As I said QGis is displaying everything fine it fails in converting the shapefile too. Opening the dbf in LibreOffice doesn't helped either. Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? Thanks in advance for any pointers. Lars ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] how does mapserver determine shapefile encoding?
Lars, Looking at the shp2pgsql documentation, it is possible to specify the shape attributes encoding with the -W parameter. IC Carlos Ruiz From: Lars Lingner gislars+l...@googlemail.com To: 'mapserver-users@lists.osgeo.org' mapserver-users@lists.osgeo.org Sent: Tuesday, August 2, 2011 2:45 PM Subject: [mapserver-users] how does mapserver determine shapefile encoding? Hello, I have a shapefile from which I don't really know the encoding. It might be latin1 or ISO-8559-5. The good thing is, when MapServer is accessing the shapefile, the labels are shown correct. QGis is also displaying everything fine. But when I try to convert with shp2pgsql or ogr I get only garbage back. I tried different source encodings. Since I need the data to be in the database I'm now trying to find out where the problem is. I tried converting the data on different systems (Ubuntu, RedHat), with postgis/shp2pgsql from repository as well as self compiled. Same results. Also iconv and recode wasn't a help. As I said QGis is displaying everything fine it fails in converting the shapefile too. Opening the dbf in LibreOffice doesn't helped either. Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? Thanks in advance for any pointers. Lars ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] how does mapserver determine shapefile encoding?
Hello Carlos, I should have been be more precise. By using all the tools, shp2pgsql, iconv, recode I set the source encoding parameter. And I tried different encodings. My database is configured for utf8. But the problem arise before the data is send to the database. With shp2pgsql I generated an sql file, containing wrong character strings. With ogr2ogr I tried to convert to csv or json just for testing, but without success. Depending on the encoding I get garbage or the error message that the data couldn't be converted to utf8. Thank you for your answer. Lars Am 02.08.2011 22:01, schrieb Carlos Ruiz: Lars, When you try to load some data to a PostgreSQL server, you can specify the encoding to use. This can be done with client_encoding. For example, if you want to populate a table with some CSV file contents, you have to do the following: SET client_encoding = 'LATIN1'; COPY my_table FROM '/usr/file.csv' WITH CSV Now, when you use shp2pgsql you can't set the client encoding, but you can set the default client_encoding of your database and every data transfer between some client and your database will be encoded properly: ALTER DATABASE my_database SET client_encoding = 'LATIN1' Hope this helps Cheers from Guadalajara, México IC Carlos Ruiz From: Lars Lingner gislars+l...@googlemail.com To: 'mapserver-users@lists.osgeo.org' mapserver-users@lists.osgeo.org Sent: Tuesday, August 2, 2011 2:45 PM Subject: [mapserver-users] how does mapserver determine shapefile encoding? Hello, I have a shapefile from which I don't really know the encoding. It might be latin1 or ISO-8559-5. The good thing is, when MapServer is accessing the shapefile, the labels are shown correct. QGis is also displaying everything fine. But when I try to convert with shp2pgsql or ogr I get only garbage back. I tried different source encodings. Since I need the data to be in the database I'm now trying to find out where the problem is. I tried converting the data on different systems (Ubuntu, RedHat), with postgis/shp2pgsql from repository as well as self compiled. Same results. Also iconv and recode wasn't a help. As I said QGis is displaying everything fine it fails in converting the shapefile too. Opening the dbf in LibreOffice doesn't helped either. Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? Thanks in advance for any pointers. Lars ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] how does mapserver determine shapefile encoding?
On 11-08-02 03:45 PM, Lars Lingner wrote: Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? MapServer has no way to tell the encoding of a shapefile. However it can usually deal with data in either UTF-8 or in the native encoding of your TTF font without specifying any encoding in the LABEL. In those cases, it does not try to figure the encoding, it just lets the bytes go straight through without looking at them (and TrueType does the rest). -- Daniel Morissette http://www.mapgears.com/ Provider of Professional MapServer Support since 2000 ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] how does mapserver determine shapefile encoding?
On 11-08-02 04:59 PM, Daniel Morissette wrote: On 11-08-02 03:45 PM, Lars Lingner wrote: Finally my (odd) questions are: Why does MapServer do everything right? How is the encoding determined? Can I see somewhere which encoding is used by MapServer? MapServer has no way to tell the encoding of a shapefile. However it can usually deal with data in either UTF-8 or in the native encoding of your TTF font without specifying any encoding in the LABEL. In those cases, it does not try to figure the encoding, it just lets the bytes go straight through without looking at them (and TrueType does the rest). Err... I mean FreeType does the rest... -- Daniel Morissette http://www.mapgears.com/ Provider of Professional MapServer Support since 2000 ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
