Re: Merge CSV Data Files - Ideas on how to do this.

[Mumia W.]

On 04/04/2007 02:56 PM, Bill wrote:
Hi all.  I have bunch of CSV files that have the same data.  This data is sent to clients.  Then the data is returned back with the Active/Remove field filled.  This field is only filled by 1 client.  All the files are returned back. Once they are returned back I need a script that will go through each of the CSV merge the data.  For example, below files are idential accept the active/remove field is filled in some files and not in other.  I had done this through putting all the files into a hash and looking at that way.  But it consumes alot of memory, which limits me with what i can do with a hash.  I just wanted to run this by the group and find out a more efficient way to do this. The program runs on Win32 system.  

Below are the files
[...]

Go through the files twice. The first time, only remember which users 
are either active or removed. The second time you go through the files, 
incorporate the information about active and removed users before you 
send the files.

Here is an example that uses your sample data files (not included):

#!/usr/bin/perl
use strict;
use warnings;
use Fcntl;
use DB_File;
use Text::CSV_XS;
use File::Spec::Functions;
use IO::File;
use Data::Dumper;

use constant SCAN => 0;
use constant WRITE => 1;

my $filesdir = 'dfiles';
my @files = glob(catdir($filesdir, '*.txt'));
my %users;
my $csv = Text::CSV_XS->new();

load_file($_, SCAN) for @files;
load_file($_, WRITE) for @files;


undef $csv;
exit;

#########################################

sub load_file {
    my ($filename, $mode) = @_;
    my $fh = IO::File->new($filename,'r') or die("open failure: $!");
    my $header = 0;
    local $_;

    if ($mode eq SCAN) {
        while ($_ = $fh->getline) {
            next unless $header++;
            next unless $csv->parse($_);
            my @fields = $csv->fields;
            next unless @fields > 2;
            my $uname = "$fields[0],$fields[1]";

            $users{$uname} = $fields[3]
                if ($fields[3] && !$users{$uname});
        }
    }

    if ($mode eq WRITE) {
        print "FILENAME: $filename\n";
        while ($_ = $fh->getline) {
            next unless $csv->parse($_);
            my @fields = $csv->fields;
            next unless @fields > 2;
            my $uname = "$fields[0],$fields[1]";

            $fields[3] = $users{$uname} if ($users{$uname});
            s/^\s+//, s/\s+$// for @fields;
            $csv->print(\*STDOUT, [EMAIL PROTECTED]);
            print "\n";
        }
    }

    print "\n";
    $fh->close or die ("close failure: $!");
}


__HTH__


--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/