Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/19/16, Jürgen Hestermann  wrote:

>  > If I understood you correctly you would want to have
>  > function FileXXXUtf8(Fn) do:
>  > begin
>  > if (Length(Fn) > MaxPath) and (not StartsWith('\\?\',Fn) then
>  >   FileXXXUtf8('\\?\' + Fn)
>  > else
>  >   OriginalImplementationOfFileXXXUtf8()
>  > end;
>  > (With the exception of ExpandFilenameUtf8)
>
> Yes, more or less.
> Slashes should be converted to backslashes too.
>
> Also, expanding (relative or incomplete) paths can be done
> by Lazarus internally so that this would even work with long paths.

Again you fail to see the problem.
There is no way of telling wether e.g.
C:\longpath\..\..\..\..\longpath/foo/bar is meant as a extended long
path or not.
Only the programmer of that particular program may know.
If we guess wrong, the program will do something it is not supposed to do.

But now I'm only repeating arguments, so I'll stop contributing to this thread.

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-17 um 18:12 schrieb Bart:
> On 1/17/16, Jürgen Hestermann  wrote:
>> 2. without forward slashes and
> But then the function would not act on \\?\C:\maxpathchars\foo/bar if
> you omitted the \\?\, which seems to be the intention of your
> proposal?

I don't understand.
If the file path is already prepended by \\?\ then
forward slashes would not be resolved by the Win API anyway.
From 
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx

"Note:  File I/O functions in the Windows API convert "/" to "\" as part of 
converting the
name to an NT-style name, except when using the "\\?\" prefix as detailed in the 
following sections."

Nevertheless, we can also simply replace all / by \ before handing it over
to the Win API function.


> If I understood you correctly you would want to have
> function FileXXXUtf8(Fn) do:
> begin
> if (Length(Fn) > MaxPath) and (not StartsWith('\\?\',Fn) then
>   FileXXXUtf8('\\?\' + Fn)
> else
>   OriginalImplementationOfFileXXXUtf8()
> end;
> (With the exception of ExpandFilenameUtf8)

Yes, more or less.
Slashes should be converted to backslashes too.

Also, expanding (relative or incomplete) paths can be done
by Lazarus internally so that this would even work with long paths.

> To me, the better solution would be that a program that must be able
> to handle this should prepend the prefix itself (and taking care of
> all relevant issues like expanding, allowing/replacing '/' with '\'
> etc. beforehand) and then call the FileXXXUtf8 functions on the
> resulting string.

IMO this is just the wrong approach.
If Lazarus offers functions like FileExists then they should be rock solid.
Otherwise (if I have to find out the glory details anyway) I would
write my own FileExist (which I did now).
I have abandanded already many of these LCL functions because in
the end they did not help me avoiding any work.
There were always side effects that I had to overcome
with self written functions anyway.
So of what use are they?
Not only am I forced to know about all the OS specific details
I also need to find out what and how it has been implemented
in the Lazarus functions.
If I already know everything I can write my own
function that fits my needs much better.


--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-17 um 12:42 schrieb Bart:
> On 1/16/16, Jürgen Hestermann  wrote:
>> Well, at least when the path is too long (length(FileName>MAXPATH))
>> it can prepend \\?\ because it will fail without it anyway.
> That won't do for the same reasons as explained above.
> Say you have a directory: c:\maxpathchars\1\2\3\
> Now cehcking for ..\..\foobar will still fail if
> c:\maxpathchars\1\foorbar exists, unless you have a folder in
> c:\maxpathchars\1\2\3\ that actually has the name  '..' etc.
> I.o.w expanding the filename cannot be done anymore.

Yes, that's true.
Additional checks (i.e. for dots) are needed
but I think it should be possible to find out
when a path is:

1. not relative and
2. without forward slashes and
3. too long to be resolved without prepended \\?\

so that adding \\?\ would make the request successful
when it otherwise would fail.


> Note that even Windows itself (at least windows explorer) cannot handle this.

Windows (the API) *can* handle this, only explorer can't.
I have written a file manager that has no problem to copy/move/rename etc.
files with paths of arbitrary length.
Of course, many special things (like deleting to recycle bin and setting
current directory) do not work with such long paths,
but at least those things that work should be made functional.


> I once unzipped a file with such a filename (produced on linux of course).
> It unzipped well and FileexistsUtf8('\\?\realfilename) worked well.
> (Testing this was the reason I did this.)
> After that windows was unable to remove the file in question.
> I tried various tools for that, icluding rm - R *.
> No such luck.

It seems that the unzip tool you used *was* able to handle long paths.
In such cases you can use more appropiate file managers like Total Commander or
Ztree that have larger limits than 255 (but still Ztree has a limit of 400 and
Total commander of 1000 character which I don't understand).
When using the W functions the limit is near 32000 so why don't they all 
support this?
Strange.


> To me it looks like a rather odd design desicion by MS have a MAXPATH
> but then have obscure ways to circumvent that.

Yes, again it is very bad design from MS that all (Windows users) have to live 
with.
But we should not make it worse than it already is.
So if the W functions support long paths we should support this too.


> AFACS it cannot be resolved/fixed in a manner that will not break
> backwards compatibility.

I think it is possible (see above).
You only need to find out the exact limitations and detect them before adding 
\\?\


> Further more: this is not something that belongs to Lazarus, but to fpc.

I am not sure. I thought FileExistsUTF8 is part of Lazarus, not Free Pascal.


>> I did not find any comments that mentions long paths
> You need new glasses then ;-)
> === code ==
>   //if Filename uses ExtendedLengthPath scheme then it cannot be expanded
>   //AND it should not be altered by ForcePathDelims or ResolveDots
>   //See: 
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
>   if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim) and
>  (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use
> AllowDirectorySeparators here!
>  then Exit;
> === end code ===

I have read this but how is this related to \\?\ and long paths?
What is ExtendedLengthPath scheme?
I don't see any information about W functions and \\?\ prefix.



--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-17 um 14:23 schrieb Jürgen Hestermann:
> Additional checks (i.e. for dots) are needed
> but I think it should be possible to find out
> when a path is:
> 1. not relative and
> 2. without forward slashes and
> 3. too long to be resolved without prepended \\?\

Even better would be to do the expansion of relative paths
internally before handing it over to the Windows API
so that they even work when Windows would fail.
The same applies to forward slashes.


--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Sven Barth]

Am 17.01.2016 14:23 schrieb "Jürgen Hestermann" :
> >> I did not find any comments that mentions long paths
> > You need new glasses then ;-)
> > === code ==
> >   //if Filename uses ExtendedLengthPath scheme then it cannot be
expanded
> >   //AND it should not be altered by ForcePathDelims or ResolveDots
> >   //See:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
> >   if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim)
and
> >  (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use
> > AllowDirectorySeparators here!
> >  then Exit;
> > === end code ===
>
> I have read this but how is this related to \\?\ and long paths?
> What is ExtendedLengthPath scheme?
> I don't see any information about W functions and \\?\ prefix.

Did you look at the if-clause following the comment? What do you think
PathDelim + PathDelim + '?' + PathDelim evaluates to?

Regards,
Sven
--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Sven Barth]

Am 17.01.2016 12:42 schrieb "Bart" :
> In the end I used a bootable linux dvd (puppy linux), mounted the
> partition in question and used the filemanger from that distro to
> remove the folder and file.

You know that you could just have written a program that uses the *W API to
delete files with a \\?\ prefixed path? ;)

Regards,
Sven
--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/16/16, Jürgen Hestermann  wrote:

> Well, at least when the path is too long (length(FileName>MAXPATH))
> it can prepend \\?\ because it will fail without it anyway.

That won't do for the same reasons as explained above.
Say you have a directory: c:\maxpathchars\1\2\3\
Now cehcking for ..\..\foobar will still fail if
c:\maxpathchars\1\foorbar exists, unless you have a folder in
c:\maxpathchars\1\2\3\ that actually has the name  '..' etc.
I.o.w expanding the filename cannot be done anymore.

Note that even Windows itself (at least windows explorer) cannot handle this.
I once unzipped a file with such a filename (produced on linux of course).
It unzipped well and FileexistsUtf8('\\?\realfilename) worked well.
(Testing this was the reason I did this.)

After that windows was unable to remove the file in question.
I tried various tools for that, icluding rm - R *.
No such luck.

In the end I used a bootable linux dvd (puppy linux), mounted the
partition in question and used the filemanger from that distro to
remove the folder and file.

To me it looks like a rather odd design desicion by MS have a MAXPATH
but then have obscure ways to circumvent that.

AFACS it cannot be resolved/fixed in a manner that will not break
backwards compatibility.
Further more: this is not something that belongs to Lazarus, but to fpc.

> I did not find any comments that mentions long paths

You need new glasses then ;-)

=== code ==
  //if Filename uses ExtendedLengthPath scheme then it cannot be expanded
  //AND it should not be altered by ForcePathDelims or ResolveDots
  //See: 
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
  if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim) and
 (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use
AllowDirectorySeparators here!
 then Exit;
=== end code ===

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Sven Barth  wrote:

>
> You know that you could just have written a program that uses the *W API to
> delete files with a \\?\ prefixed path? ;)

I did that, and id did not work.
Probably because I did something wrong.

To be honest, I then removed the file using a linux live dvd, at a
time when I booted that up for an entriley different reason.

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Jürgen Hestermann  wrote:

>  > AFACS it cannot be resolved/fixed in a manner that will not break
>  > backwards compatibility.
>
> I think it is possible (see above).
> You only need to find out the exact limitations and detect them before
> adding \\?\

Only if we decide to _always_ try that (under specified circumstances).
If not it will be an optional parameter and the function then will
have a different signature, which will be incompatible with existing
programs using it as a function variable.

>> Further more: this is not something that belongs to Lazarus, but to fpc.

> I am not sure. I thought FileExistsUTF8 is part of Lazarus, not Free Pascal.

The only difference between File* and File*UTF8 functions
(fpc<->Lazarus) is that the UTF8 versions were/are able to handle any
unicodecharacter inside the given path/filename (regardless of you
current Windows codepage).
Otherwise it does not behave different compared to the fpc File*
functions, nor should it.

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Jürgen Hestermann  wrote:

>  >> I did not find any comments that mentions long paths
>  > You need new glasses then ;-)
>  > === code ==
>  >   //if Filename uses ExtendedLengthPath scheme then it cannot be expanded
>  >   //AND it should not be altered by ForcePathDelims or ResolveDots
>  >   //See:
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
>  >   if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim) and
>  >  (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use
>  > AllowDirectorySeparators here!
>  >  then Exit;
>  > === end code ===
>
> I have read this but how is this related to \\?\ and long paths?
> What is ExtendedLengthPath scheme?
> I don't see any information about W functions and \\?\ prefix.
>

Quoted from the link:
"To specify an extended-length path, use the "\\?\" prefix. For
example, "\\?\D:\very long path"."

Maybe you need new eyes, not new glasses ;-)

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-17 um 15:25 schrieb Bart:
> Quoted from the link:
> "To specify an extended-length path, use the "\\?\" prefix. For
> example, "\\?\D:\very long path"."
> Maybe you need new eyes, not new glasses ;-)

I know the Windows API specification very well.

But you seem to forget the original thread.
I asked:

> Why not prepend "\\?\" at this last Windows step so it works for long paths

to which you answered:

> This will break existing programs, including the ide.
> Currently you can use both / and \ in filenames on Wndows filename
> functions, because the API does not care.
> That is unlues you prepend \\?\, then character in the given string
> MUST be in the filename.
> Also this will then return false on '..\foo.bar', sincenow te APi
> expects that the two dots are part of the filename.
> See the comments in ExpandFilenameUTF8 (I think I put a reference there).

How does the comment bring any new facts to the discussion?
It just repeats the text you wrote before.
I expected more information in the comment which I did not find.


--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Jürgen Hestermann  wrote:

> How does the comment bring any new facts to the discussion?
> It just repeats the text you wrote before.
> I expected more information in the comment which I did not find.

We seem to misunderstand eachther, or at least I do so.

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-17 um 15:39 schrieb Bart:
>>  > AFACS it cannot be resolved/fixed in a manner that will not break
>>  > backwards compatibility.
>> I think it is possible (see above).
>> You only need to find out the exact limitations and detect them before
>> adding \\?\
> Only if we decide to _always_ try that (under specified circumstances).
> If not it will be an optional parameter and the function then will
> have a different signature, which will be incompatible with existing
> programs using it as a function variable.

If you check for \\?\ (which is already done),
detect (or even expand) relative paths and
detect slashes (or even convert them to backslashes)
then no backward compatibility would be broken.
No drawback but the advantage of no failure for long paths.


>>> Further more: this is not something that belongs to Lazarus, but to fpc.
>> I am not sure. I thought FileExistsUTF8 is part of Lazarus, not Free Pascal.
> The only difference between File* and File*UTF8 functions
> (fpc<->Lazarus) is that the UTF8 versions were/are able to handle any
> unicodecharacter inside the given path/filename (regardless of you
> current Windows codepage).
> Otherwise it does not behave different compared to the fpc File*
> functions, nor should it.

Still the original function I mentioned was from Lazarus.
Therefore I wrote this to the Lazarus forum.
Otherwise I would have been blamed for not using the right forum. ;-)


BTW:
The following code in ExpandFileNameUtf8 seems to be not quite correct:

 if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim) and
 (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use 
AllowDirectorySeparators here!
then Exit;

When PathDelim='/' it would make no sense.
Only Windows has the '\\?\' prefix and as far as I know it cannot be '//?/'.
For Linux this maybe a valid file/directory name (but I don't know).


--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Jürgen Hestermann  wrote:

> BTW:
> The following code in ExpandFileNameUtf8 seems to be not quite correct:
>
>   if (Length(Fn) > 3) and (Fn[1] = PathDelim) and (Fn[2] = PathDelim) and
>   (Fn[3] = '?') and (Fn[4] = PathDelim) //Do NOT use
> AllowDirectorySeparators here!
>  then Exit;
>
> When PathDelim='/' it would make no sense.

You are wrong.
The code in question is in a includefile that is only included under
Windows, where PathDelim = DirectorySeparator = '\', always (it's a
constant).

The filename should have been a clue here (as is the include direcory
of it's main unit file):
($lazarus)\components\lazutils\winlazfileutils.inc.
I just preferred writing PathDelim when I wrote that piece of code.

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Bart  wrote:
> include direcory

Read: "include directive"

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/17/16, Jürgen Hestermann  wrote:

> Yes, that's true.
> Additional checks (i.e. for dots) are needed
> but I think it should be possible to find out
> when a path is:
>
> 1. not relative and

Yes.

> 2. without forward slashes and

But then the function would not act on \\?\C:\maxpathchars\foo/bar if
you omitted the \\?\, which seems to be the intention of your
proposal?
Mind you that the \\?\ prefix is not only meant for the MaxPath situation.

If I understood you correctly you would want to have
function FileXXXUtf8(Fn) do:

begin
if (Length(Fn) > MaxPath) and (not StartsWith('\\?\',Fn) then
  FileXXXUtf8('\\?\' + Fn)
else
  OriginalImplementationOfFileXXXUtf8()
end;

(With the exception of ExpandFilenameUtf8)

> 3. too long to be resolved without prepended \\?\
>
> so that adding \\?\ would make the request successful
> when it otherwise would fail.

See above.
At least it would be ambigous.

To me, the better solution would be that a program that must be able
to handle this should prepend the prefix itself (and taking care of
all relevant issues like expanding, allowing/replacing '/' with '\'
etc. beforehand) and then call the FileXXXUtf8 functions on the
resulting string.
You, as the programmer, should know wether or not the input in your
program can rely on '/' meaning "current system pathdelimiter" or not.

E.g. Lazarus itself treats '/' as pathdelimiter (at least in may
places), and therefore automatic use of \\?\ prefix may be dangerous
(because failing would be a better option than succeding with a
"strange" filename).
(Or it may not be, I simply cannot tell without studying all the sources.)

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Jürgen Hestermann]

Am 2016-01-16 um 00:11 schrieb Bart:
> On 1/15/16, Jürgen Hestermann  wrote:
>> Why not prepend "\\?\" at this last Windows step so it works for long paths
> This will break existing programs, including the ide.
> Currently you can use both / and \ in filenames on Wndows filename
> functions, because the API does not care.
> That is unlues you prepend \\?\, then character in the given string
> MUST be in the filename.
> Also this will then return false on '..\foo.bar', sincenow te APi
> expects that the two dots are part of the filename.

Well, at least when the path is too long (length(FileName>MAXPATH))
it can prepend \\?\ because it will fail without it anyway.

I have found this issue the hard way as I checked for file existence in
my program and it reported false although the file existed.
So I overwrote it. IMO this is a bug.
A function that claims to check for file existence should be as reliable
as possible and not have some obscure side effects.


> See the comments in ExpandFilenameUTF8 (I think I put a reference there).

I did not find any comments that mentions long paths

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Ondrej Pokorny]

On 16.01.2016 14:01, Jürgen Hestermann wrote:

I have found this issue the hard way as I checked for file existence in
my program and it reported false although the file existed.
So I overwrote it. IMO this is a bug.
A function that claims to check for file existence should be as reliable
as possible and not have some obscure side effects.


Yes, it can bite every one of us at some point.
It has to be solved in a backwards-compatible way, though.

Please report to mantis.

Ondrej

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Re: [Lazarus] FileExistsUTF8 and long paths

[Bart]

On 1/15/16, Jürgen Hestermann  wrote:

> Why not prepend "\\?\" at this last Windows step so it works for long paths

This will break existing programs, including the ide.

Currently you can use both / and \ in filenames on Wndows filename
functions, because the API does not care.
That is unlues you prepend \\?\, then character in the given string
MUST be in the filename.
Also this will then return false on '..\foo.bar', sincenow te APi
expects that the two dots are part of the filename.

See the comments in ExpandFilenameUTF8 (I think I put a reference there).

Bart

--
___
Lazarus mailing list
Lazarus@lists.lazarus.freepascal.org
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus