How in the heck did you create filenames longer than MAX_PATH? I've tried
everything I can think of, short of writing a C++ program that directly
connects to the NTFS FS.
Anyway, this does much of what you want, with that exception. It takes a LONG
time for large numbers of files. That is, a million or more.
[ CmdletBinding() ]
Param(
[string] $item = 'C:\'
)
function timer
{
( Get-Date ).ToLongTimeString()
}
function wv
{
Param(
[string] $str
)
Write-Verbose "$( timer ) $str"
}
## we only need to create this once
$md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
function GetMyHash
{
Param(
[object] $fileObject
)
$function = 'GetMyHash:'
wv "$function begin, fileobject name '$( $fileObject.FullName )'"
if( $fileObject.NameLength -ge 256 )
{
## here you should write a helper function!!!
Write-Error "$function, filename too long '$(
$fileObject.FullName )'"
}
else
{
# you can compress this to a single line, but if you do that,
you
# can't clean up resources. if you are doing hundreds of
thousands
# or millions of files -- you gotta clean up
# $md5 is declared at the script level
try
{
[System.IO.FileStream] $file = [System.IO.File]::Open(
$fileObject.FullName, 'Open', 'Read' )
$fileObject.Hash = [BitConverter]::ToString(
$md5.ComputeHash( $file ) )
$file.Close()
$file.Dispose()
$file = $null
}
catch
{
}
# $md5.Clear() --- don't do this
}
wv "$function exit"
}
function GetFileList
{
Param(
[string] $item
)
$function = 'GetFileList:'
wv "$function begin, item '$item'"
$params = New-Object System.Collections.Arraylist
$params.AddRange( @(
'/L',
'/S', # include subdirectories, but not empty ones
'/NJH', # no job header (no logo)
'/BYTES', # show filesizes in bytes
'/FP', # include full filename of file in path output
'/NC', # don't log file classes
'/NDL', # don't log directory names
'/TS', # include source filename timestamps
'/XJ', # exclude junction points
'/R:0', # number of retries on failed I/O
'/W:0' ) ) # how long to wait between retries
$countPattern = '^\s{3}Files\s:\s+(?<Count>\d+).*'
$sizePattern = '^\s{3}Bytes\s:\s+(?<Size>\d+(?:\.?\d+)\s[a-z]?).*'
wv "$function initiating robocopy"
$roboResults = robocopy.exe $item Nothing $params
wv "$function robocopy complete"
wv "$function result from robocopy has $( $roboResults.Count ) results"
[int] $count = 0
foreach( $result in $roboResults )
{
$count++
if( ( $count % 1000 ) -eq 0 )
{
wv "$function count $count of $( $roboResults.Count )"
}
if( $result -match
'(?<Size>\d+)\s(?<Date>\S+\s\S+)\s+(?<FullName>.*)' )
{
try
{
New-Object PSObject -Property @{
FullName = $matches.FullName
NameLength = $matches.FullName.Length
Size = $matches.Size
Date = [Datetime]$matches.Date
Hash = ''
}
}
catch
{
wv "fault: line = '$result'"
wv "fault: fullname = '$( $matches.FullName )'"
wv "fault: namelen = '$(
$matches.FullName.Length )'"
wv "fault: size = '$(
$matches.Size.ToString() )'"
wv "fault: date = '$( $matches.Date )'"
}
}
else
{
wv "$function nomatch: $result $( if( $result ) {
$result.Length } else { 'null' } )"
}
}
wv "$function exit, count = $count"
}
##
## Main
##
$global:fsiresult = GetFileList $item
wv "Gonna calculate hashes"
$global:fsiresult |% { GetMyHash $_ }
wv "Done"
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Kurt Buff
Sent: Wednesday, December 30, 2015 5:20 PM
To: [email protected]
Subject: Re: [powershell] Long file names - again...
Ultimate goal is still the same - I want to get fullname, length and a hash of
each file (md5 or sha1, for duplicate detection only), so AFAICT PowerShell is
still the way to go.
Unless I can use md5sum from unxtools or gnuwin32 toolsets, or something like
that.
Not worried about junctions at this point. I probably should look at
New-PsDrive for grins.
I'll also take a look at .NET CORE - though I doubt I can take advantage of it
if it hasn't been made easily available to powershell.
Kurt
On Wed, Dec 30, 2015 at 1:47 PM, Michael B. Smith <[email protected]> wrote:
> Actually, in this case, I think it's the .NET MAX_PATH (260).
>
> .NET itself, in its PathHelper class, doesn't accept the '\\?\' syntax and
> explicitly throws an exception if the path exceeds 259 (ANSI) or 129
> (Unicode). This class is used all over .NET.
>
> Now, .NET CORE has been enhanced to support long paths. But that isn't
> today... for most people.
>
> There are a few libraries (third party) you can use to add this support to
> PowerShell. They basically wrap the low-level APIs using P/Invoke (which you
> can do yourself, with Add-Type). They are much lower impact than installing
> Cygwin, but still... I guess my question is this: what are you really trying
> to do?
>
> Cmd.exe and the subst command are still my "go to solution" for long paths.
> You can also fake this in PowerShell with New-PsDrive (or "net share...") if
> you absolutely must stay in PowerShell, but it gets a little nasty if you
> need to worry about junctions.
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Kurt Buff
> Sent: Wednesday, December 30, 2015 4:13 PM
> To: [email protected]
> Subject: [powershell] Long file names - again...
>
> All,
>
> If anyone can help, I'd much appreciate it.
>
> I'm picking up where I left off some time ago in auditing our file server.
>
> I used robocopy to generate a list of files for each drive on our file server
> - all told over 10.3m lines, massaged the output (with findstr) to break it
> up by drive letter and to remove directories and things like $recycle.bin and
> 'system volume', then further massaged the output to remove the extraneous
> robocopy markings. I had to break it into smaller files by partition because
> processing the file in powershell overran RAM on a 16g machine.
>
> I then took each line (which looked like, e.g.
> i:\somedirectory\otherdirectory\file), then prepended '\\?\' to each
> line), because some number of the files have path lengths greater than
> 260 characters, and I'm hoping that using this specification will allow
> access to those files without adding funky 3rd party tools.
>
> So, I've ended up with a set of text files that have many lines that look
> like this:
> \\?\i:\somedirectory\otherdirectory\file
>
> What I'm trying to do is illustrated by the following, but I'm getting no
> output from it - it just returns without any output after a few moments.
>
> $files = get-content c:\batchfiles\file-i.txt
> foreach ( $file in $files )
> {
> get-childitem $file | select length, fullname
> }
>
>
> However, if I strip the '\\?\' from each line, it does what I want - but of
> course the script fails as soon as it encounters a file that has a
> name/directory specification that exceeds the Win32 API limit.
>
> I've tried surrounding the string with both double and single quotes, and
> still no joy.
>
> A simpler example tells the tale:
>
> This works, except for long file names:
> gci i:\somedirectory\otherdirectory\file
>
> These fail silently:
> gci \\?\i:\somedirectory\otherdirectory\file
> gci "\\?\i:\somedirectory\otherdirectory\file"
> gci '\\?\i:\somedirectory\otherdirectory\file'
>
> These fail with an error:
> gci "\\\\?\\i:\somedirectory\otherdirectory\file"
> gci '\\\\?\\i:\somedirectory\otherdirectory\file'
>
> The error is:
> Get-ChildItem : Cannot retrieve the dynamic parameters for the cmdlet.
> Cannot process argument because the value of argument "path" is not valid.
> Change the value of the "path" argument and run the operation again.
> At line:1 char:1
> + gci "\\\\?\\i:\CFRemoteImages\Air Canada Montreal STOC.vhd" | select
> length, ful ...
> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> + CategoryInfo : InvalidArgument: (:) [Get-ChildItem],
> ParameterBindingException
> + FullyQualifiedErrorId :
> GetDynamicParametersException,Microsoft.PowerShell.Commands.GetChildIt
> emCommand
>
>
> ================================================
> Did you know you can also post and find answers on PowerShell in the forums?
> http://www.myitforum.com/forums/default.asp?catApp=1
>
>
> ================================================
> Did you know you can also post and find answers on PowerShell in the forums?
> http://www.myitforum.com/forums/default.asp?catApp=1
================================================
Did you know you can also post and find answers on PowerShell in the forums?
http://www.myitforum.com/forums/default.asp?catApp=1
================================================
Did you know you can also post and find answers on PowerShell in the forums?
http://www.myitforum.com/forums/default.asp?catApp=1
