Hi Mark, any reason you are using mapping files?
Well, I was think running the process thru the dos prompt would be faster than actually opening up FME workbench. There are 2600 ASCII files and each ASCII file contains 80,000 to 200,000 records. The clipping file is only one file with 5000+ polygons. I did not use dataset fanout to avoid loading all 2600 files into memory before clipping. I'm running on Windows XP 3.0+Mhz with 3.5gigs of RAM. Yes, that's 3.5 GIGs for RAM. I also invoked the /3GB switch. I am using the clipper transformer and I'll try setting to the first setting. I'll try your suggestions and get back to you. --- In [email protected], "mark2atsafe" <[EMAIL PROTECTED]> wrote: > > Hi, > First off, is there any reason you are using mapping files? It would > be a much simpler task to do this using the Batch Deploy in Workbench. > > One issue I'm seeing is that CSV output is folder based, not file > based, so what you should do is supply a folder name instead of a > filename. Of course doing this would lead to each input file > over-writing the same output file (and it might be doing this anyway > in your command file, hence the reason for a lack of output). It's > another reason I'd suggest using Batch Deploy. > > An alternative is to use fanouts - but that would mean reading all the > data at once, which could cause problems in a Clipping process where > you intend to have files processed entirely separately. So I don't > recommend it here. > > So, how many features are you processing, and how long is it taking? > What sort of machine spec do you have? I've noticed that with spatial > feature processing more memory provides a better result than a faster > processor. Also there are a number of tips that might help, for > example pointing the temporary folder to a faster disk. See > performance tuning FME on fmepedia.com > > Finally - are you actually using the Clipper transformer? If so make > sure the Clippers arrive first and that you have the Clippers First > setting turned on. I believe this can make a very drastic difference > in performance (the higher the ratio of clippees to clippers the more > the gain). Again there is some info on fmepedia about how to set this > up and how to make sure your Clipper features do actually arrive first. > > Oh yes - command files. They do help the performance because they > don't require FME to be stopped and started each time (2600 times in > your case). So that's good. But because it's all part of the same > process I think you need to keep an eye on memory usage. Check the > log. Does the translation start out fast then slow down? That might be > a low-memory indication. Of course it depends on how large these 2600 > files are. > > Hope this helps > Regards > > Mark > > Mark Ireland, Senior Product Specialist > Safe Software Inc. Surrey, BC, CANADA > [EMAIL PROTECTED] http://www.safe.com > Solutions for Spatial Data Translation, Distribution and Access > > > --- In [email protected], "kcheng888_007" <kcheng888_007@> wrote: > > > > Hi All, > > > > Here is the scenario; > > > > I have a 2600 ASCII files containing X,Y,Z coorindates and a single > > shapefile containing polygons for the entire area. > > > > I am removing the xyz points from the ASCII file within the polygons > > from all 2600 ASCII files. > > > > SETUP: > > > > I am using the batch deploy option to clip one ASCII file at time. > > > > However this process is taking a lot of time. > > > > I've created a command file (csvclip.txt) as such > > > > csvcl.fme --Sourcedataset c:\1.csv --DestDataset c:\out\1_o.csv > > csvcl.fme --Sourcedataset c:\2.csv --DestDataset c:\out\2_o.csv > > csvcl.fme --Sourcedataset c:\3.csv --DestDataset c:\out\3_o.csv > > etc .....to file 2600.cvs > > > > I run the command file as such > > > > fme.exe COMMAND_FILE csvclip.txt > > > > And process does start. > > > > Now my question; > > 1) Does FME execute each command line at a time or does it load > > everything into memory first? > > 2) The clipping is taking a lot of time and I was expecting to see > > one output file at a time in the output directory? which I don't... > > 3) Is there a way to clip one file at time and get one output file > > at a time? My current process doesn't do this. > > > For insights into what's up at Safe Software and what's on the development horizon, visit Safe's blog at spatial-etl.blogspot.com. Safe Software has also made slides available that outline enhancements planned for FME 2007. The slides are from the "Road Ahead" presentation given on Day 2 of the FME Worldwide Users Conference. To view these slides, visit www.safe.com/2006uc. Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/fme/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/fme/join (Yahoo! ID required) <*> To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
