Hi Mark, Batch deploy appears to be reading all the files into memory first then sending to the output.
Shouldn't it be one file in --> one file out? --- In [email protected], "kcheng888_007" <[EMAIL PROTECTED]> wrote: > > Hi Mark, > > any reason you are using mapping files? > > Well, I was think running the process thru the dos prompt would be > faster than actually opening up FME workbench. > > There are 2600 ASCII files and each ASCII file contains 80,000 to > 200,000 records. The clipping file is only one file with 5000+ > polygons. > > I did not use dataset fanout to avoid loading all 2600 files into > memory before clipping. > > I'm running on Windows XP 3.0+Mhz with 3.5gigs of RAM. Yes, that's > 3.5 GIGs for RAM. I also invoked the /3GB switch. > > I am using the clipper transformer and I'll try setting to the first > setting. > > I'll try your suggestions and get back to you. > > > > > > > > > --- In [email protected], "mark2atsafe" <mark.ireland@> wrote: > > > > Hi, > > First off, is there any reason you are using mapping files? It > would > > be a much simpler task to do this using the Batch Deploy in > Workbench. > > > > One issue I'm seeing is that CSV output is folder based, not file > > based, so what you should do is supply a folder name instead of a > > filename. Of course doing this would lead to each input file > > over-writing the same output file (and it might be doing this > anyway > > in your command file, hence the reason for a lack of output). It's > > another reason I'd suggest using Batch Deploy. > > > > An alternative is to use fanouts - but that would mean reading all > the > > data at once, which could cause problems in a Clipping process > where > > you intend to have files processed entirely separately. So I don't > > recommend it here. > > > > So, how many features are you processing, and how long is it > taking? > > What sort of machine spec do you have? I've noticed that with > spatial > > feature processing more memory provides a better result than a > faster > > processor. Also there are a number of tips that might help, for > > example pointing the temporary folder to a faster disk. See > > performance tuning FME on fmepedia.com > > > > Finally - are you actually using the Clipper transformer? If so > make > > sure the Clippers arrive first and that you have the Clippers First > > setting turned on. I believe this can make a very drastic > difference > > in performance (the higher the ratio of clippees to clippers the > more > > the gain). Again there is some info on fmepedia about how to set > this > > up and how to make sure your Clipper features do actually arrive > first. > > > > Oh yes - command files. They do help the performance because they > > don't require FME to be stopped and started each time (2600 times > in > > your case). So that's good. But because it's all part of the same > > process I think you need to keep an eye on memory usage. Check the > > log. Does the translation start out fast then slow down? That > might be > > a low-memory indication. Of course it depends on how large these > 2600 > > files are. > > > > Hope this helps > > Regards > > > > Mark > > > > Mark Ireland, Senior Product Specialist > > Safe Software Inc. Surrey, BC, CANADA > > support@ http://www.safe.com > > Solutions for Spatial Data Translation, Distribution and Access > > > > > > --- In [email protected], "kcheng888_007" <kcheng888_007@> wrote: > > > > > > Hi All, > > > > > > Here is the scenario; > > > > > > I have a 2600 ASCII files containing X,Y,Z coorindates and a > single > > > shapefile containing polygons for the entire area. > > > > > > I am removing the xyz points from the ASCII file within the > polygons > > > from all 2600 ASCII files. > > > > > > SETUP: > > > > > > I am using the batch deploy option to clip one ASCII file at > time. > > > > > > However this process is taking a lot of time. > > > > > > I've created a command file (csvclip.txt) as such > > > > > > csvcl.fme --Sourcedataset c:\1.csv --DestDataset c:\out\1_o.csv > > > csvcl.fme --Sourcedataset c:\2.csv --DestDataset c:\out\2_o.csv > > > csvcl.fme --Sourcedataset c:\3.csv --DestDataset c:\out\3_o.csv > > > etc .....to file 2600.cvs > > > > > > I run the command file as such > > > > > > fme.exe COMMAND_FILE csvclip.txt > > > > > > And process does start. > > > > > > Now my question; > > > 1) Does FME execute each command line at a time or does it load > > > everything into memory first? > > > 2) The clipping is taking a lot of time and I was expecting to > see > > > one output file at a time in the output directory? which I > don't... > > > 3) Is there a way to clip one file at time and get one output > file > > > at a time? My current process doesn't do this. > > > > > > For insights into what's up at Safe Software and what's on the development horizon, visit Safe's blog at spatial-etl.blogspot.com. Safe Software has also made slides available that outline enhancements planned for FME 2007. The slides are from the "Road Ahead" presentation given on Day 2 of the FME Worldwide Users Conference. To view these slides, visit www.safe.com/2006uc. Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/fme/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/fme/join (Yahoo! ID required) <*> To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
