On Tue, Jan 4, 2011 at 4:42 PM, <[email protected]> wrote: > > >>>Large file support does not help here because 8.5 billion points > >>>exceeds the number of supported features in GRASS vectors which is > >>>about 2 billion (2,147,483,647 to be precise). > >>I'll chop the inputfile into sections less than 2 billion then. Is > there any reason that the 2 billion limit on GRASS vectors >>cannot be > raised? ( variable change vs types of variable and lots of ugly fixes in > various places?) If it's something simple, I can play >>with it, but I'm not > a C programmer :-(. > > >The limit can be raised, but, unfortunately, this is not simple, because > >portability across platforms must be maintained. It boils down to the > >problem that it is not simple to have a portable >64 bit integer type on at > >least Linux, Mac, Windows, both 32 bit and 64 bit architectures. I would > >like to have a portable 64 bit integer type for grass 7 though... > > I see GRASS7 in my future :-) > > In the short term, Can this limit be gotten around by putting the data in > Postgresql/Postgis/Spatialite and using v.external or v.db.connect for the > inputpoints for v.surf.bspline? > I guess in this case v.external is worth a try, if Postgresql/Postgis/Spatialite support that many entries. Make sure you use v.external -b (no topology building).
Markus M > > Markus Metz <[email protected]> > > 01/04/2011 04:43 AM > > To > [email protected] > cc > [email protected], Markus Neteler <[email protected]> > Subject > Re: [GRASS-dev] How many points in a point layer and v.surf.bspline questions > > > > > > On Mon, Jan 3, 2011 at 4:01 PM, <[email protected]> wrote: > > Markus M., > > >Large file support does not help here because 8.5 billion points > >exceeds the number of supported features in GRASS vectors which is > >about 2 billion (2,147,483,647 to be precise). > I'll chop the inputfile into sections less than 2 billion then. Is > there any reason that the 2 billion limit on GRASS vectors cannot be raised? > ( variable change vs types of variable and lots of ugly fixes in various > places?) If it's something simple, I can play with it, but I'm not a C > programmer :-(. > > The limit can be raised, but, unfortunately, this is not simple, because > portability across platforms must be maintained. It boils down to the problem > that it is not simple to have a portable 64 bit integer type on at least > Linux, Mac, Windows, both 32 bit and 64 bit architectures. I would like to > have a portable 64 bit integer type for grass 7 though... > > Markus M > > > >Further on, a region with 6.8 billion cells is a bit large since the > >interpolated raster will be held in memory which would require about > >50 GB RAM (that could be fixed for v.surf.bspline by keeping > >intermediate data on disk). > I know some folks with computers with 64GB+ of RAM, so that is not an > insurmountable issue. However, it would probably be better to go the > intermediate route. > > > No idea where the request for > >18446744071812941729 * 8 bytes comes from, this is astronomical. > That was my reaction. Perhaps the computer was screaming in pain.:-) > > > >Apart from that, spline steps of 40 seem ok, since the point distance > >is 5 m or less, spline steps of 20 would also be ok. The larger the > >spline steps, the smoother the resulting surface. Smoothing is also > >controlled through lambda. > > Thanks for the insight, I will need to change my approach to minimize the > smoothing. > > Doug > > > > > Markus M > Doug Newcomb > USFWS > Raleigh, NC > 919-856-4520 ext. 14 [email protected] > --------------------------------------------------------------------------------------------------------- > The opinions I express are my own and are not representative of the official > policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life > is too short for undocumented, proprietary data formats. > > Markus Metz <[email protected]> > > 01/03/2011 09:13 AM > > To > Markus Neteler <[email protected]> > cc > [email protected], [email protected] > Subject > Re: [GRASS-dev] How many points in a point layer and v.surf.bspline questions > > > > > > > On Mon, Jan 3, 2011 at 2:57 PM, Markus Neteler <[email protected]> wrote: > > hi Doug, > > > > On Mon, Jan 3, 2011 at 2:26 PM, <[email protected]> wrote: > >> > >> Hi Folks, > >> > >> I aggregated all of the bare earth lidar points for the state of North > >> Carolina into a single file Imported all 8.5 billion points for NC into > >> one > >> point layer with no topology. I was sort of curious to see if I could > >> generate a seamless 20 ft elevation grid for the State of North Carolina > >> from a single data layerusing bspline and tried the following command: > >> > >> GRASS 6.5.svn (ncstpft_nad83):~ > v.surf.bspline input=all_nc_be_pts2 > >> raster=all_nc_be_20ft_bspline sie=40 sin=40 layer=0 > >> WARNING: Coor files of vector map <all_nc_be_p...@statewide> is larger than > >> it should be (158913789952 bytes excess) > >> Cells for raster map <all_nc_be_20ft_bspline> will be interpolated > >> ERROR: G_calloc: unable to allocate 18446744071812941729 * 8 bytes of > >> memory at dalloc.c:66 > > > > you are hitting the limits :) I dunno if large file support helps, but > > for vector data > > it is only available in GRASS 7 to my knowledge. > > > Large file support does not help here because 8.5 billion points > exceeds the number of supported features in GRASS vectors which is > about 2 billion (2,147,483,647 to be precise). > > Further on, a region with 6.8 billion cells is a bit large since the > interpolated raster will be held in memory which would require about > 50 GB RAM (that could be fixed for v.surf.bspline by keeping > intermediate data on disk). No idea where the request for > 18446744071812941729 * 8 bytes comes from, this is astronomical. > > Apart from that, spline steps of 40 seem ok, since the point distance > is 5 m or less, spline steps of 20 would also be ok. The larger the > spline steps, the smoother the resulting surface. Smoothing is also > controlled through lambda. > > Markus M > > _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
