Re: [gentoo-user] copy a bunch of files...
On Tue, Feb 8, 2011 at 8:27 PM, Mark Knecht markkne...@gmail.com wrote:
Hi,
Looking for a simple way to do a big copy at the command line. I
have a bunch of files (maybe 100 right now, but it will grow) that I
can find with locate and grep:
c2stable ~ # locate Correlation | grep Builder | grep csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
SNIP
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
c2stable ~ #
I need to copy these files to a new directory
(~mark/CorrelationTests) where I will modify what's in them before
running correlation tests on the contents.
How do I feed the output of the command above to cp at the command
line to get this done?
I've been playing with things like while read but I can't get it
right.
Thanks,
Mark
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {}
~mark/CorrelationTests \;
If catching the *csv is not enough, you can use -ipath instead of -iname and
do something like this:
-ipath '*Builder*csv'
this will match all the path.
Regards,
Kfir
Re: [gentoo-user] copy a bunch of files...
On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {}
~mark/CorrelationTests \;
Replace \; with + for a faster process, as Mark said there are hundreds
of these files.
Or, if you use zsh instead of bash, it can be as simple as
cp Builder/**/*.csv CorrelationTests
--
Neil Bothwick
Learn from your parents' mistakes - use birth control!
signature.asc
Description: PGP signature
Re: [gentoo-user] copy a bunch of files...
On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {}
~mark/CorrelationTests \;
Replace \; with + for a faster process, as Mark said there are
hundreds
of these files.
Or, if you use zsh instead of bash, it can be as simple as
cp Builder/**/*.csv CorrelationTests
There is a problem with this approach, though.
It can easily give command line too long.
Helmut.
Re: [gentoo-user] copy a bunch of files...
On Wed, Feb 9, 2011 at 4:09 AM, Helmut Jarausch
jarau...@igpm.rwth-aachen.de wrote:
On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {}
~mark/CorrelationTests \;
Replace \; with + for a faster process, as Mark said there are
hundreds
of these files.
Or, if you use zsh instead of bash, it can be as simple as
cp Builder/**/*.csv CorrelationTests
There is a problem with this approach, though.
It can easily give command line too long.
Helmut.
Lots of interesting ideas. I use apps so often I've never become very
strong at the command line and yet people built this whole Linux
empire using it. It's very powerful.
One thing I didn't make clear in my original post - it didn't seem
important to confuse my real question which was the copy itself and
not locating the files - but which likely changes how well some of
these commands would work in my specific case was that the Builder
directory actually has _many_ CSV files ut specifically I needed only
the ones in the Correlation directories. Additionally, being that this
is stock futures trading data, generally at a given time I need the
CSV files for a specific symbol, for instance in the original post:
c2stable ~ # locate Correlation | grep Builder | grep csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
SNIP
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
c2stable ~ #
I knew I wanted the Correlation directory but it turned out I had
other directories with that name on the system, so I added the grep
Builder to get me into the right tree and CSV to find only the CSV
files. However at that point I only had Russell futures data (TF.D) so
I didn't have to go further. Now, however, as I bring in Dow futures
(YM, YM.D) , SP 500 futures (ES, ES.D), and NASDAQ futures (NQ, ND.D)
I just an an extra grep in and I'm there in terms of finding the files
I need for a certain test.
Additionally I have test results for other date ranges that will show
up soon, (2001-2005, or 2003-2011, etc. Additional greps are an easy
way to just winnow it down to a point where I'm finding what I need to
find.
This thread has given me a lot of new commands to go look at.
Thanks,
Mark
Re: [gentoo-user] copy a bunch of files...
On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote: One thing I didn't make clear in my original post - it didn't seem important to confuse my real question which was the copy itself and not locating the files - but which likely changes how well some of these commands would work in my specific case was that the Builder directory actually has _many_ CSV files ut specifically I needed only the ones in the Correlation directories. Bear in mind that locate only returns files in its database, not any created since the last time its cron job was run. Find seems a more appropriate tool for this task. signature.asc Description: PGP signature
Re: [gentoo-user] copy a bunch of files...
On Wed, Feb 9, 2011 at 12:10 PM, Neil Bothwick n...@digimed.co.uk wrote: On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote: One thing I didn't make clear in my original post - it didn't seem important to confuse my real question which was the copy itself and not locating the files - but which likely changes how well some of these commands would work in my specific case was that the Builder directory actually has _many_ CSV files ut specifically I needed only the ones in the Correlation directories. Bear in mind that locate only returns files in its database, not any created since the last time its cron job was run. Find seems a more appropriate tool for this task. Good point. As I bring these files down from a bunch of Windows VMs I typically run updatedb before doing this, but find would probably be safer. Thanks, Mark
[gentoo-user] copy a bunch of files...
Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark
Re: [gentoo-user] copy a bunch of files...
Am 08.02.2011 19:27, schrieb Mark Knecht: Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG ~mark/CorrelationTests -IARG tells xargs to replace the occurrence of ARG in the parameters with the actual parameters read from stdin. or locate Correlation | grep Builder | grep csv | while read file; do cp $file ~mark/CorrelationTests; done BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate directory Builder, ending on .csv)? Even easier: locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests Warning: I've not tested every line. Use with caution. Hope this helps, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] copy a bunch of files...
On Tue, Feb 8, 2011 at 11:02 AM, Florian Philipp li...@binarywings.net wrote: Am 08.02.2011 19:27, schrieb Mark Knecht: Hi, Looking for a simple way to do a big copy at the command line. I have a bunch of files (maybe 100 right now, but it will grow) that I can find with locate and grep: c2stable ~ # locate Correlation | grep Builder | grep csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv SNIP /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv c2stable ~ # I need to copy these files to a new directory (~mark/CorrelationTests) where I will modify what's in them before running correlation tests on the contents. How do I feed the output of the command above to cp at the command line to get this done? I've been playing with things like while read but I can't get it right. Thanks, Mark locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG ~mark/CorrelationTests -IARG tells xargs to replace the occurrence of ARG in the parameters with the actual parameters read from stdin. Thanks! This worked nicely and is relatively easy to remember. or locate Correlation | grep Builder | grep csv | while read file; do cp $file ~mark/CorrelationTests; done This is what I was trying to do but was unsuccessful. BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate directory Builder, ending on .csv)? Yes, that does seem to work. I guess that's grepping for the path of a file starting with Builder and ending with CSV? Good one. Even easier: locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests Warning: I've not tested every line. Use with caution. Hope this helps, Florian Philipp It did very much. Thanks! Now to work on modifying the files, again with a loop for all the files in ~mark/CorrelationTests - Mark
