Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Stephen Woodbridge]

I think it would be interesting to see how/if ESRI responded to a change 
in shapelib that allow it to detect standard files or large files and 
allow application built on it to work with both. If enough FOSS 
application supported that then ESRI might be forced to deal with it or 
have a lot of clients with files that they can not read.


From my usage, I almost always have a problem with the DBF getting too 
large mostly because of a lot of attribute columns. A lot of my data 
processing script have an option to split the output into numbered parts 
like:


file-01.*
file-02.*
etc

and then I use a tileindex to reassemble the parts in mapserver or ogr. 
If would be nice if we didn't have to worry about this and could just 
make large files. Personally, shapefiles are a convenient and fast 
format to work with and only on rare occasions do I have to deal with an 
ESRI client. Since I have a work around for them, that is not a problem.


-Steve W

On 5/9/2013 1:35 PM, Frank Warmerdam wrote:

Folks,

The changes I made were mostly focused on supporting large DBF files
which don't have the 32bit offsets issue.

The offsets for shapefiles are actually to 16bit words, so in theory if
the offsets were treated as unsigned 32bit values they could address up
to 8GB .shp files.  I don't recall however, if I ever got that working.
  Even if it was done it is not clear to me these files would work
smoothly with other applications.

I'd been hoping someone else might have authoritative comment in this
thread. :-)

Best regards,
Frank


On Wed, May 8, 2013 at 9:49 PM, Stephen Woodbridge
mailto:wood...@swoodbridge.com>> wrote:

In general the shapefile spec is owned by ERSI and unless they
change the spec you will have a problem. Last I knew, there are
pointers/offsets inside the shapefile that are only 32bits wide per
the spec. it is because of these that you are limited to 2GB.

At some point I thought Frank W or someone was working on a version
of shapelib that could support greater than 2GB, but only programs
that used that version would be able to read/write these larger
files. I have not heard anything on that front in a while so I don't
know if that was just talk, or code ever materialized.

-Steve W


On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:

Hi,

I am not sure either if reading >2GB shapefiles works with the
native
shp reader on all platforms, but as the ticket says, you should be
able to use OGR connection in the mapfile even with your current
version. If you use OGR you can also split the shapefile into pieces
and read them through the ogr tileindex system which may be faster
sometimes.

-Jukka Rahkonen-

__ Andrea Peri wrote:

If it don't work you should evalute the option to use
spatialite.
It work well with dataset greater than 2GB . I use it with
mapserver 6.3dev and gdal 1.10.




2013/5/9 Roger André mailto:ran...@gmail.com>>>
Pretty sure I just hit
this> problem in my 6.0.3
local stack.  I haven't tested it in the current release, but does
anyone know for sure that a shapefile with any component larger than
2GB works in 6.2?

Roger

_ mapserver-users
mailing list
mapserver-users@lists.osgeo.__org

>


http://lists.osgeo.org/__mailman/listinfo/mapserver-__users






-- - Andrea Peri . . . . . . . . . qwerty àèìòù
- _
mapserver-users mailing list mapserver-users@lists.osgeo.__org

http://lists.osgeo.org/__mailman/listinfo/mapserver-__users



_
mapserver-users mailing list
mapserver-users@lists.osgeo.__org

http://lists.osgeo.org/__mailman/listinfo/mapserver-__users





--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com 
light and sound - activate the windows | http://pobox.com/~warmerdam

and watch the world go round - Rush| Geospatial Software Developer


_

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Frank Warmerdam]

Folks,

The changes I made were mostly focused on supporting large DBF files which
don't have the 32bit offsets issue.

The offsets for shapefiles are actually to 16bit words, so in theory if the
offsets were treated as unsigned 32bit values they could address up to 8GB
.shp files.  I don't recall however, if I ever got that working.  Even if
it was done it is not clear to me these files would work smoothly with
other applications.

I'd been hoping someone else might have authoritative comment in this
thread. :-)

Best regards,
Frank


On Wed, May 8, 2013 at 9:49 PM, Stephen Woodbridge
wrote:

> In general the shapefile spec is owned by ERSI and unless they change the
> spec you will have a problem. Last I knew, there are pointers/offsets
> inside the shapefile that are only 32bits wide per the spec. it is because
> of these that you are limited to 2GB.
>
> At some point I thought Frank W or someone was working on a version of
> shapelib that could support greater than 2GB, but only programs that used
> that version would be able to read/write these larger files. I have not
> heard anything on that front in a while so I don't know if that was just
> talk, or code ever materialized.
>
> -Steve W
>
>
> On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:
>
>> Hi,
>>
>> I am not sure either if reading >2GB shapefiles works with the native
>> shp reader on all platforms, but as the ticket says, you should be
>> able to use OGR connection in the mapfile even with your current
>> version. If you use OGR you can also split the shapefile into pieces
>> and read them through the ogr tileindex system which may be faster
>> sometimes.
>>
>> -Jukka Rahkonen-
>>
>> __**__ Andrea Peri wrote:
>>
>>  If it don't work you should evalute the option to use spatialite.
>>> It work well with dataset greater than 2GB . I use it with
>>> mapserver 6.3dev and gdal 1.10.
>>>
>>
>>
>>
>> 2013/5/9 Roger André 
>> mailto:randr**e...@gmail.com
>> >>
>> Pretty sure I just hit
>> this>
>> problem in my 6.0.3
>> local stack.  I haven't tested it in the current release, but does
>> anyone know for sure that a shapefile with any component larger than
>> 2GB works in 6.2?
>>
>> Roger
>>
>> __**_ mapserver-users
>> mailing list
>> mapserver-users@lists.osgeo.**org 
>> > >
>>
>>
>>  
>> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>
>>
>>
>>
>>
>> -- - Andrea Peri . . . . . . . . . qwerty àèìòù
>> - __**_
>> mapserver-users mailing list 
>> mapserver-users@lists.osgeo.**org
>> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>>
>>
> __**_
> mapserver-users mailing list
> mapserver-users@lists.osgeo.**org 
> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>



-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Software Developer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Jeff McKenna]

Very very offtopic, but it may help someone else someday...

Using CONNECTIONTYPE OGR for shapefile connections is a power user trick
that comes in handy more times than you think.  During my Advanced
MapServer workshop at FOSS4G-BA, we tackled international character
labeling ("international" not for them, for local users there if they
want labeling in Spanish some characters will cause grief), and the
quick solution for labeling was: make sure the mapfile is saved as UTF8
encoding [especially if you plan to refer to special characters in
EXPRESSIONs), and then make sure for the LAYER to set CONNECTIONTYPE
OGR.  Bingo, that easy. (attendees saw me sweat for a few minutes, but
this trick saved me)

So keep that in your back pocket always.

-jeff




-- 
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/


On 2013-05-09 9:30 AM, Jeff McKenna wrote:
> Hi Roger,
> 
> It would be great if you could make available your test shp, possibly
> point to it [or upload it to a shared OSGeo server] and your tiny
> mapfile directly in a comment in the ticket (
> https://github.com/mapserver/mapserver/issues/3514).  Once you do that
> I'll test your mapfile with master/trunk.
> 
> Thanks,
> 
> -jeff
> 
> 


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Jeff McKenna]

Hi Roger,

It would be great if you could make available your test shp, possibly
point to it [or upload it to a shared OSGeo server] and your tiny
mapfile directly in a comment in the ticket (
https://github.com/mapserver/mapserver/issues/3514).  Once you do that
I'll test your mapfile with master/trunk.

Thanks,

-jeff


-- 
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/




On 2013-05-09 2:39 AM, Roger André wrote:
> Very interesting to know this. Thank you all for the feedback.
> 
> I did use the OGR connector and it worked fine, but I assumed the native
> reader was faster.
> 
> On May 8, 2013 9:53 PM, "Stephen Woodbridge"  > wrote:
> 
> In general the shapefile spec is owned by ERSI and unless they
> change the spec you will have a problem. Last I knew, there are
> pointers/offsets inside the shapefile that are only 32bits wide per
> the spec. it is because of these that you are limited to 2GB.
> 
> At some point I thought Frank W or someone was working on a version
> of shapelib that could support greater than 2GB, but only programs
> that used that version would be able to read/write these larger
> files. I have not heard anything on that front in a while so I don't
> know if that was just talk, or code ever materialized.
> 
> -Steve W
> 
> On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:
> 
> Hi,
> 
> I am not sure either if reading >2GB shapefiles works with the
> native
> shp reader on all platforms, but as the ticket says, you should be
> able to use OGR connection in the mapfile even with your current
> version. If you use OGR you can also split the shapefile into pieces
> and read them through the ogr tileindex system which may be faster
> sometimes.
> 
> -Jukka Rahkonen-
> 
> __ Andrea Peri wrote:
> 
> If it don't work you should evalute the option to use
> spatialite.
> It work well with dataset greater than 2GB . I use it with
> mapserver 6.3dev and gdal 1.10.
> 
> 
> 
> 
> 2013/5/9 Roger André   >>
> Pretty sure I just hit
> this > problem in my 6.0.3
> local stack.  I haven't tested it in the current release, but does
> anyone know for sure that a shapefile with any component larger than
> 2GB works in 6.2?
> 
> Roger

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Roger André]

Very interesting to know this. Thank you all for the feedback.

I did use the OGR connector and it worked fine, but I assumed the native
reader was faster.
On May 8, 2013 9:53 PM, "Stephen Woodbridge" 
wrote:

> In general the shapefile spec is owned by ERSI and unless they change the
> spec you will have a problem. Last I knew, there are pointers/offsets
> inside the shapefile that are only 32bits wide per the spec. it is because
> of these that you are limited to 2GB.
>
> At some point I thought Frank W or someone was working on a version of
> shapelib that could support greater than 2GB, but only programs that used
> that version would be able to read/write these larger files. I have not
> heard anything on that front in a while so I don't know if that was just
> talk, or code ever materialized.
>
> -Steve W
>
> On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:
>
>> Hi,
>>
>> I am not sure either if reading >2GB shapefiles works with the native
>> shp reader on all platforms, but as the ticket says, you should be
>> able to use OGR connection in the mapfile even with your current
>> version. If you use OGR you can also split the shapefile into pieces
>> and read them through the ogr tileindex system which may be faster
>> sometimes.
>>
>> -Jukka Rahkonen-
>>
>> __**__ Andrea Peri wrote:
>>
>>  If it don't work you should evalute the option to use spatialite.
>>> It work well with dataset greater than 2GB . I use it with
>>> mapserver 6.3dev and gdal 1.10.
>>>
>>
>>
>>
>> 2013/5/9 Roger André 
>> mailto:randr**e...@gmail.com
>> >>
>> Pretty sure I just hit
>> this>
>> problem in my 6.0.3
>> local stack.  I haven't tested it in the current release, but does
>> anyone know for sure that a shapefile with any component larger than
>> 2GB works in 6.2?
>>
>> Roger
>>
>> __**_ mapserver-users
>> mailing list
>> mapserver-users@lists.osgeo.**org 
>> > >
>>
>>
>>  
>> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>
>>
>>
>>
>>
>> -- - Andrea Peri . . . . . . . . . qwerty àèìòù
>> - __**_
>> mapserver-users mailing list 
>> mapserver-users@lists.osgeo.**org
>> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>>
>>
> __**_
> mapserver-users mailing list
> mapserver-users@lists.osgeo.**org 
> http://lists.osgeo.org/**mailman/listinfo/mapserver-**users
>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Stephen Woodbridge]

In general the shapefile spec is owned by ERSI and unless they change 
the spec you will have a problem. Last I knew, there are 
pointers/offsets inside the shapefile that are only 32bits wide per the 
spec. it is because of these that you are limited to 2GB.


At some point I thought Frank W or someone was working on a version of 
shapelib that could support greater than 2GB, but only programs that 
used that version would be able to read/write these larger files. I have 
not heard anything on that front in a while so I don't know if that was 
just talk, or code ever materialized.


-Steve W

On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:

Hi,

I am not sure either if reading >2GB shapefiles works with the native
shp reader on all platforms, but as the ticket says, you should be
able to use OGR connection in the mapfile even with your current
version. If you use OGR you can also split the shapefile into pieces
and read them through the ogr tileindex system which may be faster
sometimes.

-Jukka Rahkonen-

 Andrea Peri wrote:


If it don't work you should evalute the option to use spatialite.
It work well with dataset greater than 2GB . I use it with
mapserver 6.3dev and gdal 1.10.




2013/5/9 Roger André mailto:ran...@gmail.com>>
Pretty sure I just hit
this problem in my 6.0.3
local stack.  I haven't tested it in the current release, but does
anyone know for sure that a shapefile with any component larger than
2GB works in 6.2?

Roger

___ mapserver-users
mailing list
mapserver-users@lists.osgeo.org



http://lists.osgeo.org/mailman/listinfo/mapserver-users





-- - Andrea Peri . . . . . . . . . qwerty àèìòù
- ___
mapserver-users mailing list mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users



___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Rahkonen Jukka]

Hi,

I am not sure either if reading >2GB shapefiles works with the native shp 
reader on all platforms, but as the ticket says, you should be able to use OGR 
connection in the mapfile even with your current version. If you use OGR you 
can also split the shapefile into pieces and read them through the ogr 
tileindex system which may be faster sometimes.

-Jukka Rahkonen-


Andrea Peri wrote:

> If it don't work you should evalute the option to use spatialite.
> It work well with dataset greater than 2GB .
> I use it with mapserver 6.3dev and gdal 1.10.



2013/5/9 Roger André mailto:ran...@gmail.com>>
Pretty sure I just hit this 
problem in my 6.0.3 local stack.  I haven't tested it in the current release, 
but does anyone know for sure that a shapefile with any component larger than 
2GB works in 6.2?

Roger

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users




--
-
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Is there still a 2GB size limit on shapefiles?

[Andrea Peri]

If it don't work you should evalute the option to use spatialite.
It work well with dataset greater than 2GB .
I use it with mapserver 6.3dev and gdal 1.10.



2013/5/9 Roger André 

> Pretty sure I just hit this  
> problem
> in my 6.0.3 local stack.  I haven't tested it in the current release, but
> does anyone know for sure that a shapefile with any component larger than
> 2GB works in 6.2?
>
> Roger
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>


-- 
-
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Is there still a 2GB size limit on shapefiles?

[Roger André]

Pretty sure I just hit this
 problem
in my 6.0.3 local stack.  I haven't tested it in the current release, but
does anyone know for sure that a shapefile with any component larger than
2GB works in 6.2?

Roger
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users