On 7/20/06, Pagurek,Debbie [NCR] <[EMAIL PROTECTED]> wrote: ..
Funny you should mention all this point in polygon search stuff, because that is exactly what I'm working on right now, but I'm using a .csv file, WFS requests with a DWITHIN filter, and adding the results to the original .csv as new fields. I don't think the performance is going to be very good though ;)
"Algorithms with Perl" aka the wolf book <http://www.oreilly.com/catalog/maperl/reviews.html> has a point-in-poly algorigthm that I have used in the past successfully with shapefiles. I used Geo::ShapeFile <http://search.cpan.org/~jasonk/Geo-ShapeFile-2.51/ShapeFile.pm> to work with the shapefiles. I know that will work, but I have to work on optimizing the speed part because doing a 5 mill points in 200k polys is a cool trillion intersections. By the way, the task is not one-off. Hence the need for a program.
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Thursday, July 20, 2006 10:37 AM To: Pagurek,Debbie [NCR] Cc: [email protected] Subject: Re: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to points Hey Debbie, Thanks for your reply. A few clarifications... On 7/20/06, Pagurek,Debbie [NCR] <[EMAIL PROTECTED]> wrote: > I am working on some similar work with .csv files and OGR right now. > I don't know why your field is being changed like that... > Yup, this is strange, and unacceptable, so I need a workaround for this. > As for debugging, aside from the ogrinfo test, you might want to embed > your <OGRVRTDataSource> into a mapfile and then try the shp2img tool. > If this produces an image for you, all the mapserver stuff is fine. > But you might be having permissions problems on your server, which is > where MY problem lies right now. I'm using IIS5, not apache... So I > have some work ahead of me! I am not doing anything with the web at all here. This is pure command line stuff. The complete process I want to accomplish is as follows -- 1. convert lon-lat into shapefile points. 2. overlay (point-in-poly) #1 shapefile with a shapefile polys to determine, for each point, what poly it falls in... so, if #1 shapefile has cities, and the polys shapefile has states, I want to create an additional attribute for each city and fill it with the states value. This is all command line, and for quite a bit of data (upward of 5 million), so speed is an issue, although CPU and memory are not an issue (good hardware). I am using Perl for all the automation. I would like to keep everything in the realm of text files and dbf files as much as possible, but I am not averse to using SQLite or Berkeley DB. If push-comes-to-shove, I might be open to experimenting with PostGIS, but only if push really comes to shove. Many thanks, (and hope to see you again soon somewhere... maybe I will have to make a trip up North). > > You might also want to write a little PHP script to make sure that > your ODBC DSN (a system DSN, right?) is working ok as well. It's when > I did this that I realized that my problem has nothing to do with > mapserver, ogrinfo etc. > > Good luck and let us know what you discover, Debbie Pagurek > > -----Original Message----- > From: UMN MapServer Users List [mailto:[EMAIL PROTECTED] > On Behalf Of P Kishor > Sent: Thursday, July 20, 2006 8:19 AM > To: [email protected] > Subject: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to > points > > Folks, > > I have seen this question asked in the past, and have a similar problem. > I want to convert many lon-lat to a shapefile of points quickly. The > lon-lat are in a csv text file (values.txt) like so > > foo,bar,baz,long,lat,qux > 1, bar1, baz1, -87.796341, 41.907504, s10 ... > ... > > So, I installed FW_Tools 1.05, created a DSN out of the text file > (this is and has to be on a Windows box), and ogrinfo happily gave me > information that I wanted, but with a twist... > > OGRFeature(values.txt):1056 > foo (Integer) = 1 > bar(String) = ba1 > baz(String) = baz1 > long (Real) = -87.796341 > lat (Real) = 41.907504 > qux(String) = 10.0000 > > So, what's with interpreting qux? It converted my "s10" into a string > of value "10.0000". > > Anyway, then I created an OVF file (values.ovf) like so > > <OGRVRTDataSource> > <OGRVRTLayer name="values"> > <SrcDataSource>ODBC:values</SrcDataSource> > <SrcLayer>values.txt</SrcLayer> > <GeometryField encoding="PointFromColumns" x="Long" y="Lat"/> > <GeometryType>wkbPoint</GeometryType> > </OGRVRTLayer> > </OGRVRTDataSource> > > well, > > ogr2ogr -f "ESRI Shapefile" . values.ovf values > > but while that seemed to do something without any error, I got no > output. > > So, is what I am doing the right way of going about this? Is there a > better or another way to try out and benchmark? Oh, did I mention? -- > The values.txt file has about 5.25 million rows, so speed is important. > -- Puneet Kishor http://punkish.eidesis.org/ Nelson Inst. for Env. Studies, UW-Madison http://www.ies.wisc.edu/ Open Source Geospatial Foundation https://edu.osgeo.org/
-- Puneet Kishor http://punkish.eidesis.org/ Nelson Inst. for Env. Studies, UW-Madison http://www.ies.wisc.edu/ Open Source Geospatial Foundation https://edu.osgeo.org/
