Re: [gentoo-user] copy a bunch of files...
On Tue, Feb 8, 2011 at 8:27 PM, Mark Knecht markkne...@gmail.com wrote: Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark Another way to do it is with find: find /home/mark/Builder -type f -iname '*csv' -exec cp {} ~mark/CorrelationTests \; If catching the *csv is not enough, you can use -ipath instead of -iname and do something like this: -ipath '*Builder*csv' this will match all the path. Regards, Kfir
Re: [gentoo-user] copy a bunch of files...
On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote: Another way to do it is with find: find /home/mark/Builder -type f -iname '*csv' -exec cp {} ~mark/CorrelationTests \; Replace \; with + for a faster process, as Mark said there are hundreds of these files. Or, if you use zsh instead of bash, it can be as simple as cp Builder/**/*.csv CorrelationTests -- Neil Bothwick Learn from your parents' mistakes - use birth control! signature.asc Description: PGP signature
Re: [gentoo-user] copy a bunch of files...
On 02/09/2011 12:56:09 PM, Neil Bothwick wrote: On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote: Another way to do it is with find: find /home/mark/Builder -type f -iname '*csv' -exec cp {} ~mark/CorrelationTests \; Replace \; with + for a faster process, as Mark said there are hundreds of these files. Or, if you use zsh instead of bash, it can be as simple as cp Builder/**/*.csv CorrelationTests There is a problem with this approach, though. It can easily give command line too long. Helmut.
Re: [gentoo-user] copy a bunch of files...
On Wed, Feb 9, 2011 at 4:09 AM, Helmut Jarausch jarau...@igpm.rwth-aachen.de wrote: On 02/09/2011 12:56:09 PM, Neil Bothwick wrote: On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote: Another way to do it is with find: find /home/mark/Builder -type f -iname '*csv' -exec cp {} ~mark/CorrelationTests \; Replace \; with + for a faster process, as Mark said there are hundreds of these files. Or, if you use zsh instead of bash, it can be as simple as cp Builder/**/*.csv CorrelationTests There is a problem with this approach, though. It can easily give command line too long. Helmut. Lots of interesting ideas. I use apps so often I've never become very strong at the command line and yet people built this whole Linux empire using it. It's very powerful. One thing I didn't make clear in my original post - it didn't seem important to confuse my real question which was the copy itself and not locating the files - but which likely changes how well some of these commands would work in my specific case was that the Builder directory actually has _many_ CSV files ut specifically I needed only the ones in the Correlation directories. Additionally, being that this is stock futures trading data, generally at a given time I need the CSV files for a specific symbol, for instance in the original post: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I knew I wanted the Correlation directory but it turned out I had other directories with that name on the system, so I added the grep Builder to get me into the right tree and CSV to find only the CSV files. However at that point I only had Russell futures data (TF.D) so I didn't have to go further. Now, however, as I bring in Dow futures (YM, YM.D) , SP 500 futures (ES, ES.D), and NASDAQ futures (NQ, ND.D) I just an an extra grep in and I'm there in terms of finding the files I need for a certain test. Additionally I have test results for other date ranges that will show up soon, (2001-2005, or 2003-2011, etc. Additional greps are an easy way to just winnow it down to a point where I'm finding what I need to find. This thread has given me a lot of new commands to go look at. Thanks, Mark
Re: [gentoo-user] copy a bunch of files...
On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote: One thing I didn't make clear in my original post - it didn't seem important to confuse my real question which was the copy itself and not locating the files - but which likely changes how well some of these commands would work in my specific case was that the Builder directory actually has _many_ CSV files ut specifically I needed only the ones in the Correlation directories. Bear in mind that locate only returns files in its database, not any created since the last time its cron job was run. Find seems a more appropriate tool for this task. signature.asc Description: PGP signature
Re: [gentoo-user] copy a bunch of files...
On Wed, Feb 9, 2011 at 12:10 PM, Neil Bothwick n...@digimed.co.uk wrote: On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote: One thing I didn't make clear in my original post - it didn't seem important to confuse my real question which was the copy itself and not locating the files - but which likely changes how well some of these commands would work in my specific case was that the Builder directory actually has _many_ CSV files ut specifically I needed only the ones in the Correlation directories. Bear in mind that locate only returns files in its database, not any created since the last time its cron job was run. Find seems a more appropriate tool for this task. Good point. As I bring these files down from a bunch of Windows VMs I typically run updatedb before doing this, but find would probably be safer. Thanks, Mark
[gentoo-user] copy a bunch of files...
Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark
Re: [gentoo-user] copy a bunch of files...
Am 08.02.2011 19:27, schrieb Mark Knecht: Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG ~mark/CorrelationTests -IARG tells xargs to replace the occurrence of ARG in the parameters with the actual parameters read from stdin. or locate Correlation | grep Builder | grep csv | while read file; do cp $file ~mark/CorrelationTests; done BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate directory Builder, ending on .csv)? Even easier: locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests Warning: I've not tested every line. Use with caution. Hope this helps, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] copy a bunch of files...
On Tue, Feb 8, 2011 at 11:02 AM, Florian Philipp li...@binarywings.net wrote: Am 08.02.2011 19:27, schrieb Mark Knecht: Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG ~mark/CorrelationTests -IARG tells xargs to replace the occurrence of ARG in the parameters with the actual parameters read from stdin. Thanks! This worked nicely and is relatively easy to remember. or locate Correlation | grep Builder | grep csv | while read file; do cp $file ~mark/CorrelationTests; done This is what I was trying to do but was unsuccessful. BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate directory Builder, ending on .csv)? Yes, that does seem to work. I guess that's grepping for the path of a file starting with Builder and ending with CSV? Good one. Even easier: locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests Warning: I've not tested every line. Use with caution. Hope this helps, Florian Philipp It did very much. Thanks! Now to work on modifying the files, again with a loop for all the files in ~mark/CorrelationTests - Mark