Replying to myself, since that seems the reasonable thing to do here.
I've tested the following against a smaller directory that I know has
some duplicates, and am getting progress. Here is what I have so far
(work with the line wraps!):
Get-ChildItem S:\ -File -Recurse | select fullname, length |
Export-CSV -NoTypeInformation c:\temp\files.csv
Import-CSV c:\temp\files.csv | Select-Object -Property
@{Name="MD5";Expression={(Get-Filehash -algorithm md5
$_.FullName).MD5}},Length,FullName | Export-CSV -NoTypeInformation
c:\temp\filehash.csv
Import-CSV C:\temp\checker\fileMD5.csv | Sort-Object
@{Expression={$_.Length -as [int]}} | Export-CSV -NoTypeInformation
c:\temp\checker\FileMD5Sorted.csv
The above generates a file of 315286 lines (not including header) - of
course, that's the number of files in the directory tree. I get output
that looks like this (work with the line wraps again):
"MD5","Length","FullName"
"6467C3875955DF4514395F0AFCAAA62A","3182604288","S:\Infrastructure\Microsoft\OSes\Win7EntSP1_64bit\SW_DVD5_SA_Win_Ent_7w_SP1_64BIT_English_-2_MLF_X17-58882.ISO"
I noticed two oddities, however:
o- zero-length files generate a hash, and of course the hash is the
same for all of them. I probably should have expected that, but it
surprised me.
o- I find a handful of files (22 of them) at the top of the csv file
after sorting that don't seem to obey the sorting on the hash that the
other files followed. It's very strange. They're not duplicates of any
other files; their hashes and file sizes are out of sort order from
all of the rest, AFAICT. I'm not sure what to make of that.
But, ignoring those two things, I'd like to proceed a bit further:
o- Writing to another file only those lines that are duplicate files,
which I can do by selecting selecting the lines that have matching
hashes (and possibly also matching sizes)
o- Possibly adding another column, which would contain an integer that
would increment for each set of matched files, which would probably
lead to...
o- Among other things, calculating the amount of duplicated space (sum
of n-1 file sizes for each set of dupes), identifying duplicate
directories that can be eliminated in toto, etc.
But, I'm stymied on the execution of the logic. I'm such an
inexperienced programmer that I'm flailing on the first of these
steps. I believe I need to make a stepwise comparison of the MD5
column, which I think would look something like this:
$dupe = 1
read infile.line1 into variable1
read infile.line2 into variable2
if {
variable1.MD5 -eq variable2.MD5
prefix variable1 with dupe counter
write variable1 to the new csv file
while not eof
set variable1 to the contents of variable2
read line next into variable2
compare variable1.MD5 to variable2.MD5
if match
prefix variable1 with $dupe
append variable1 as new line of new csv file
else
increment dupe counter
endwhile }
else {
while not eof
set variable1 to the contents of variable2
read line next into variable2
compare variable1.MD5 to variable2.MD5
if match
prefix variable1 with $dupe
append variable1 as new line of new csv file
else
increment dupe counter
endwhile
I realize I could be way off base on the algorithm here, but that's
what I've been able to dream up.
Anyone care to critique and offer syntax suggestions - my googlefu is
about exhausted.
Kurt
On Thu, Jul 30, 2015 at 12:45 PM, Kurt Buff <[email protected]> wrote:
> I'm putting together what should be a simple little script, and failing.
>
> I am ultimately looking to run this against a directory, then sort the
> output on the hash field and then parse for duplicates. There are two
> conditions that concern me: 1) there are over 3m files in the target
> directory, and 2) many of the files are quite large, over 1g.
>
> I'm more concerned about the effects of the script on memory than on
> processor - the data is fairly static, and I intend to run it once a
> month or even less, but I did choose MD5 as the hash algorithm for
> speed, rather than accept the default of SHA256.
>
> This is pretty simple stuff, I'm sure, but I'm using this as a
> learning exercise more than anything, as there are duplicate file
> finders out in the world already.
>
> There are several problems with what I have put together so far, which
> this this:
>
> Get-ChildItem c:\stuff -Recurse | select length, fullname |
> export-csv -NoTypeInformation c:\temp\files.csv
> Import-CSV C:\temp\files.csv | ForEach-Object { (get-filehash
> -algorithm md5 $_.FullName) }; Length | Sort hash
>
> Using Length (or $_.Length) anywhere in the foreach statement gives an
> error, or gives weird output.
>
> Sample Output when not using Length, and therefore getting reasonable
> output (extra spaces and hyphen delimiters elided):
> Algorithm Hash
> Path
> MD5 592BE1AD0ED83C36D5E68CA7A014A510
> C:\stuff\Tools\SomeFile.DOC
>
> What I'd like to see instead
> Hash Length
> Path
> 592BE1AD0ED83C36D5E68CA7A014A510 79872 C:\stuff\Tools\SomeFile.DOC
>
> If anyone can offer some instruction, I'd appreciate it.
>
> Kurt
================================================
Did you know you can also post and find answers on PowerShell in the forums?
http://www.myitforum.com/forums/default.asp?catApp=1
