subject:"phobo's std.file is completely broke\!"

On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
Panteleev wrote:
One point of view is that the expected behavior is that the 
functions succeed. Another point of view is that Phobos should 
not allow programs to create files and directories with 
invalid paths. Consider, e.g. that a user writes a program 
that creates a large tree of deeply nested filesystem objects. 
When they are done and wish to delete them, their file manager 
fails and displays an error. The user's conclusion? D sucks 
because it corrupts the filesystem and creates objects they 
can't operate with.


You don't even need to use crazy third-party software.

Try this program:

mkdir(`\\?\C:\ a \`);
write(`\\?\C:\ a \a.txt`, "Hello");

Then, try doing the following:

- Double-click the created text file.

- Try deleting the directory from Explorer (by sending it to 
the recycle bin).


- Try permanently deleting it (Shift+Delete).

- Try renaming it.

All of these fail for me. Deleting the directory doesn't even 
show an error - nothing at all happens.


When the OS itself fails to properly deal with such files, I 
don't think D has any business in *facilitating* their creation 
by default.


*Windows Explorer* prevents you from creating a folder or file 
whose name STARTS with spaces. It trims them automatically, 
whether you want it or not.


So it's NOT a surprise that *Windows Explorer* (!) has problems 
if you use it on such files which were created manually.


But obviously, *Windows* OS doesn't prevent you to create them 
through scripts and applications...

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

On Thursday, 20 September 2018 at 02:48:06 UTC, Nick Sabalausky 
(Abscissa) wrote:

On 09/19/2018 02:33 AM, Jonathan Marler wrote:


What drives me mad is when you have library writers who try to 
"protect" you from the underlying system by translating 
everything you do into what they "think" you're trying to do.


What drives me mad is when allegedly cross-platform tools 
deliberately propagate non-cross-platform quirks that could 
easily be abstracted away and pretend that's somehow "helping" 
me instead of making a complete wreck of the whole point of 
cross-platform. Bonus points if they're doing it mainly to help 
with my C++-standard premature optimizations.


If I actually want to deal with platform-specific quirks, then 
I'll use the platform's API directly. (And then I'll beat 
myself with a brick, just for fun.)


+1

A cross-platform library has to be designed to operate in the 
same way on each supported platform, even if this means that it's 
harder to implement on some platform, or that some platforms will 
need more complicated implementations.


That's the whole point of this "HAL" approach.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky 
(Abscissa) wrote:

(Not on a Win box at the moment.)


I added the output of my test program to the gist:
https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986#file-output-txt


assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );


Actually it's crazier than that. The concatenation of the current 
directory plus the relative path must be < MAX_PATH (approx.). 
Meaning, if you are 50 directories deep, a relative path starting 
with 50 `..\` still won't allow you to access C:\file.txt.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Neia Neutuladh via Digitalmars-d

On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir 
Panteleev wrote:
When the OS itself fails to properly deal with such files, I 
don't think D has any business in *facilitating* their creation 
by default.


Dear lord Windows is terrible. Can we just deprecate it?

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky 
(Abscissa) wrote:

I'm not sure I'm quite following you. Is this what you mean?:

string dir = ...; // Such that...
assert( dir.isRelativePath );
assert( dir.length < MAX_LENGTH-12 );
assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );

// ??? This *currently* goes BOOM on Windows
// ??? installations with MAX_LENGTH restriction active?
rmdir(path);

(Not on a Win box at the moment.)


Correct.

Re: phobo's std.file is completely broke!


On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote:


rmdir(path);


Obviously meant "rmdir(dir);" here. Editing mishap.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Thursday, 20 September 2018 at 03:25:05 UTC, Nick Sabalausky 
(Abscissa) wrote:

On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote:


rmdir(path);


Obviously meant "rmdir(dir);" here. Editing mishap.


and MAX_PATH instead of MAX_LENGTH, and absolutePath instead of 
toAbsolutePath, and !isAbsolute instead of isRelativePath, but I 
understood what you meant :)

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/19/2018 07:04 AM, Vladimir Panteleev wrote:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5].


This is not a good criteria: relative paths whose pointing to objects 
whose absolute path exceeds MAX_PATH will fail, too. So, it looks like 
Phobos would need to expand relative paths unconditionally.




I'm not sure I'm quite following you. Is this what you mean?:

string dir = ...; // Such that...
assert( dir.isRelativePath );
assert( dir.length < MAX_LENGTH-12 );
assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );

// ??? This *currently* goes BOOM on Windows
// ??? installations with MAX_LENGTH restriction active?
rmdir(path);

(Not on a Win box at the moment.)

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
Panteleev wrote:
One point of view is that the expected behavior is that the 
functions succeed. Another point of view is that Phobos should 
not allow programs to create files and directories with invalid 
paths. Consider, e.g. that a user writes a program that creates 
a large tree of deeply nested filesystem objects. When they are 
done and wish to delete them, their file manager fails and 
displays an error. The user's conclusion? D sucks because it 
corrupts the filesystem and creates objects they can't operate 
with.


You don't even need to use crazy third-party software.

Try this program:

mkdir(`\\?\C:\ a \`);
write(`\\?\C:\ a \a.txt`, "Hello");

Then, try doing the following:

- Double-click the created text file.

- Try deleting the directory from Explorer (by sending it to the 
recycle bin).


- Try permanently deleting it (Shift+Delete).

- Try renaming it.

All of these fail for me. Deleting the directory doesn't even 
show an error - nothing at all happens.


When the OS itself fails to properly deal with such files, I 
don't think D has any business in *facilitating* their creation 
by default.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/19/2018 01:49 PM, Neia Neutuladh wrote:

On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir Panteleev wrote:

BTW, something follows from the above:

write(`C:\` ~ (short path) ~  `con`) will fail

but:

write(`C:\` ~ (long path) ~ `con`) will succeed.

This is just one issue I've noticed... there's probably more lurking.


Also, according to the internet:

write(chainPath(shortDirectory, "A "), "Win32 API strips trailing space");
readText(chainPath(shortDirectory, "A")); // Win32 API strips trailing 
space


But:
write(chainPath(longDirectory, "A "), "Win32 API strips trailing space");
readText(chainPath(longDirectory, "A")); // File not found

write(chainPath(shortDirectory, "A."));  // fails
write(chainPath(longDirectory, "A."));  // succeeds



Ok, excellent, now we're getting somewhere! :) Based on these actual 
concrete examples, yes, I can now grant that converting paths that are 
past the limit would indeed cause subtly different behaviour.


So clearly this needs to be an all-or-nothing deal...

This is why we should use the exact same behavior in all cases. Always 
use `\\?\` or never use it.


Yup.

Since Windows path handling is weird by default, I'd prefer always using 
`\\?\`. It's overhead, but not a huge amount of additional overhead 
compared to filesystem manipulation.


My thoughts, exactly.

I'll also point out that this applies to user-code, too. Ie, with Phobos 
as it currently stands, user code has three options:


1. Never use `\\?\` and any time they compile their code on Windows, 
they must always be acutely aware of, and plan for, all of the fun 
little quirks of the Win33 file APIs.


2. Be *aware* of `\\?\` and always use it any time they compile their 
code on Windows (and always remember use longPath() or some such for all 
file operations when working on a cross-platform codebase).


3. Sometimes use `\\?\` and sometimes don't, and then deal with the fun 
little inconsistencies that creates.


I think ALL those options look really bad, but #2 seems the least 
terrible. But gee, wouldn't it be nice if we could just relieve that 
stupid burden on everybody of selecting and following #2? In the rare 
case where "doing the safe/correct thing by default" isn't good enough, 
there'd be nothing stopping anyone from calling the platform API 
directly if they really, really need to.


Opt-out is for safe, reliable, consistent things.
Opt-in is for difficult, tricky, dangerous things.
Not the other way around.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/19/2018 02:33 AM, Jonathan Marler wrote:


What drives me mad is when you have library writers 
who try to "protect" you from the underlying system by translating 
everything you do into what they "think" you're trying to do.


What drives me mad is when allegedly cross-platform tools deliberately 
propagate non-cross-platform quirks that could easily be abstracted away 
and pretend that's somehow "helping" me instead of making a complete 
wreck of the whole point of cross-platform. Bonus points if they're 
doing it mainly to help with my C++-standard premature optimizations.


If I actually want to deal with platform-specific quirks, then I'll use 
the platform's API directly. (And then I'll beat myself with a brick, 
just for fun.)

Re: phobo's std.file is completely broke!

2018-09-19 Thread Neia Neutuladh via Digitalmars-d

On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir 
Panteleev wrote:

BTW, something follows from the above:

write(`C:\` ~ (short path) ~  `con`) will fail

but:

write(`C:\` ~ (long path) ~ `con`) will succeed.

This is just one issue I've noticed... there's probably more 
lurking.


Also, according to the internet:

write(chainPath(shortDirectory, "A "), "Win32 API strips trailing 
space");
readText(chainPath(shortDirectory, "A")); // Win32 API strips 
trailing space


But:
write(chainPath(longDirectory, "A "), "Win32 API strips trailing 
space");

readText(chainPath(longDirectory, "A")); // File not found

write(chainPath(shortDirectory, "A."));  // fails
write(chainPath(longDirectory, "A."));  // succeeds


This is why I think the whole idea is bankrupt.


This is why we should use the exact same behavior in all cases. 
Always use `\\?\` or never use it.


Since Windows path handling is weird by default, I'd prefer 
always using `\\?\`. It's overhead, but not a huge amount of 
additional overhead compared to filesystem manipulation.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:13:31 UTC, Ecstatic Coder 
wrote:
On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic 
Coder wrote:
None would ever be, considering you obviously have decided to 
ignore such a simple solution to the 260 character limit...


Add "ad hominem" to your pile of fallacies, I guess.


Now I will, thanks :)

Once again, this forum proves to be very effective at removing 
any motivation from D users to get involved and contribute to 
the D language.


Sorry, I didn't intend to make you feel unwelcome.

But, you did say I was ignoring something, after I've addressed 
it twice in the same thread. That wasn't nice of you either.


In any case, the solution in question won't work:
https://forum.dlang.org/post/riuqfngyzfumehsmo...@forum.dlang.org

Re: phobo's std.file is completely broke!

2018-09-19 Thread Kagamin via Digitalmars-d

On Wednesday, 19 September 2018 at 09:58:00 UTC, Vladimir 
Panteleev wrote:

On my Windows VM, I get:

C:\(long path here): The filename or extension is too long. 
(error 206)


This seems like a completely reasonable error message to me, so 
I think we're good there already.


It can be a long file name. Each individual component in the path 
still should not exceed MAX_PATH even when you use unparsed path.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Kagamin via Digitalmars-d

On Wednesday, 19 September 2018 at 10:29:11 UTC, Vladimir 
Panteleev wrote:
- GetFullPathName is documented as also having the MAX_PATH 
limit, but the framework seems to use it for normalization 
BEFORE prepending the prefix.


  
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea


That part is erroneous :)
Recent additions to msdn are too often incompetent, e.g. long 
path registry switch in windows 10 doesn't enable the feature, 
it's only the first of two gates, manifest is not optional. Don't 
ask me who does it. They can't even get backslashes right.
Passing unparsed path to GetFullPathName makes no sense because 
such unparsed path must be already full normalized path, and it 
doesn't have MAX_PATH limit, probably because it doesn't talk to 
file system.


- GetFullPathName has a big warning on it about how you 
shouldn't use it in multithreaded programs.


I wonder about that too, shouldn't the system do the same to 
resolve the absolute path? Theoretically it can keep an open 
handle to the current directory and get its path from that.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 11:04:13 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick 
Sabalausky (Abscissa) wrote:
2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 
bytes[5].


This is not a good criteria: relative paths whose pointing to 
objects whose absolute path exceeds MAX_PATH will fail, too. 
So, it looks like Phobos would need to expand relative paths 
unconditionally.


Here is my test program:
https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:
2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 
bytes[5].


This is not a good criteria: relative paths whose pointing to 
objects whose absolute path exceeds MAX_PATH will fail, too. So, 
it looks like Phobos would need to expand relative paths 
unconditionally.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 09:27:29 UTC, Vladimir 
Panteleev wrote:
This might be a change which we won't be able to back out of if 
it turns out to be a bad idea, because then we break other 
classes of programs that depend on this change. See 
https://forum.dlang.org/post/eepblrtjmqzbtopyl...@forum.dlang.org for an example.


Case in point:

https://msdn.microsoft.com/ru-ru/office/mt762842%28v=vs.90%29?f=255=-2147217396

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 09:58:30 UTC, Kagamin wrote:
On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir
Panteleev wrote:
Someone mentioned in this thread that .NET runtime does do the
long-path workaround automatically.

AFAIK, CoreFX does, but .net doesn't. .net did its own path
normalization and length check, which can be turned off since
4.6.2.

Thanks. I had a quick look. It looks pretty involved. Here are
some of the relevant parts:

https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L16-L43
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L68-L95
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/Path.Windows.cs#L37-L63
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L82-L99
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L118-L141

And for example directory deletion:

https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/System.IO.FileSystem/src/System/IO/Directory.cs#L296-L300
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/Interop/Windows/kernel32/Interop.RemoveDirectory.cs#L19-L23

Some things stood out to me:

- GetFullPathName is documented as also having the MAX_PATH
limit, but the framework seems to use it for normalization BEFORE
prepending the prefix.

https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea

- GetFullPathName has a big warning on it about how you shouldn't
use it in multithreaded programs.

- The code seems to compare the length against 260 characters,
but in my tests, the limit is actually about 12 characters
shorter. The same file defines MaxShortDirectoryPath = 248, but
that constant isn't used anywhere in the code.

Maybe we shouldn't use this as a reference after all...

Re: phobo's std.file is completely broke!

2018-09-19 Thread Kagamin via Digitalmars-d

On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir 
Panteleev wrote:
Someone mentioned in this thread that .NET runtime does do the 
long-path workaround automatically.


AFAIK, CoreFX does, but .net doesn't. .net did its own path 
normalization and length check, which can be turned off since 
4.6.2.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky 
(Abscissa) wrote:
Instead, what it really means is that our APIs should be 
designed to *REJECT* long paths with an appropriately 
meaningful error message


On my Windows VM, I get:

C:\(long path here): The filename or extension is too long. 
(error 206)


This seems like a completely reasonable error message to me, so I 
think we're good there already.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 09:16:30 UTC, Nick Sabalausky 
(Abscissa) wrote:
Essentially they boil down to "it is impossible to prove the 
algorithm is correct" (for both detecting when the path fix is 
needed, and fixing the path).


If you're referring to the inability to deterministically 
reason about just what in the h*ll MS's API's actually do, then 
I agree. But the problem is, it's equally true of all Win APIs. 
Only way to fix that is to omit Win support entirely.


It's not our job to fix it. Just provide a D interface to it, 
which already we do well.


Otherwise, I disagree. I think it is not only provable, but 
also unnecessary to prove simply because such proof has never 
been necessary for Phobos, and there is nothing inherent to 
this problem which is inherently more complicated than anything 
already existing in Phobos (you can even omit the questionable 
modules like std.xml, it all still holds).


No, we are mucking with data on the way between the user's 
program and the OS, because we think we can fix it. Not only 
should we not be doing that in the first place, but even if we 
get it right, it might still not be what the user wants.


Otherwise, present counterexamples demonstrating the inherent 
ambiguity/non-provability.


I don't understand what you mean here.


Forcing the path transformation can introduce regressions,


All phobos/compiler changes have the potential for regressions, 
plus we have unittests. Unless you can demonstrate how this 
necessarily goes above and beyond the risk from any other 
change in a way that cannot be sufficiently mitigated by tests, 
then the concern is irrelevant.


This might be a change which we won't be able to back out of if 
it turns out to be a bad idea, because then we break other 
classes of programs that depend on this change. See 
https://forum.dlang.org/post/eepblrtjmqzbtopyl...@forum.dlang.org 
for an example.



or make the situation worse on systems where it's not needed.


Provide an example where the situation is made worse.


1. A user is happily using D on a system where the workaround is 
not needed.
2. A new D version comes out, with the workaround forcibly 
enabled.

3. The user's program is now broken.

If you provide a specific implementation for the workaround 
you're envisioning, I could try to come up with more specific 
situations where it would fail. There's been lots of reasons 
mentioned in this thread where things can go wrong, and surely 
there will be more that we can't think of ahead of time.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/19/2018 04:41 AM, Vladimir Panteleev wrote:
On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky 
(Abscissa) wrote:

What's the other issue(s)?


Essentially they boil down to "it is impossible to prove the algorithm 
is correct" (for both detecting when the path fix is needed, and fixing 
the path).


If you're referring to the inability to deterministically reason about 
just what in the h*ll MS's API's actually do, then I agree. But the 
problem is, it's equally true of all Win APIs. Only way to fix that is 
to omit Win support entirely.


Otherwise, I disagree. I think it is not only provable, but also 
unnecessary to prove simply because such proof has never been necessary 
for Phobos, and there is nothing inherent to this problem which is 
inherently more complicated than anything already existing in Phobos 
(you can even omit the questionable modules like std.xml, it all still 
holds). Otherwise, present counterexamples demonstrating the inherent 
ambiguity/non-provability.



Forcing the path transformation can introduce regressions,


All phobos/compiler changes have the potential for regressions, plus we 
have unittests. Unless you can demonstrate how this necessarily goes 
above and beyond the risk from any other change in a way that cannot be 
sufficiently mitigated by tests, then the concern is irrelevant.


or 
make the situation worse on systems where it's not needed.




Provide an example where the situation is made worse.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:46:13 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir 
Panteleev wrote:
If you're referring to NUL, COM1, COM2, etc, then this is 
completely orthogonal.


Yes. How so? It is the same issue: paths with certain 
properties are valid on all platforms except on Windows. 
Phobos errors out when attempting to access/create them. A 
simple workaround is available: expand/normalize the path, 
prepend the UNC prefix, and use Unicode APIs.


I just remembered, there is a third class of paths with these 
properties: paths containing directory components that begin or 
end with spaces.


There are probably more... I think some special characters are 
also valid only in UNC paths.


BTW, something follows from the above:

write(`C:\` ~ (short path) ~  `con`) will fail

but:

write(`C:\` ~ (long path) ~ `con`) will succeed.

This is just one issue I've noticed... there's probably more 
lurking. This is why I think the whole idea is bankrupt.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir 
Panteleev wrote:
If you're referring to NUL, COM1, COM2, etc, then this is 
completely orthogonal.


Yes. How so? It is the same issue: paths with certain 
properties are valid on all platforms except on Windows. Phobos 
errors out when attempting to access/create them. A simple 
workaround is available: expand/normalize the path, prepend the 
UNC prefix, and use Unicode APIs.


I just remembered, there is a third class of paths with these 
properties: paths containing directory components that begin or 
end with spaces.


There are probably more... I think some special characters are 
also valid only in UNC paths.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky 
(Abscissa) wrote:

What's the other issue(s)?


Essentially they boil down to "it is impossible to prove the 
algorithm is correct" (for both detecting when the path fix is 
needed, and fixing the path). Forcing the path transformation can 
introduce regressions, or make the situation worse on systems 
where it's not needed.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/19/2018 02:55 AM, Vladimir Panteleev wrote:
On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky 
(Abscissa) wrote:

- Does it actually, necessarily perform those additional OS calls?


We need to expand relative paths to absolute ones, for which we need to 
fetch the current directory.




So in other words, NO, it does NOT necessarily perform additional OS 
calls. It ONLY performs an additional call if the given path is relative 
*AND* if we've decided to not simply reject too-long relative paths 
outright (which I'd be fine with as a compromise. At least it would be 
well-defined and enforced with a meaningful message.)



- Is it really?


Is what really what? If you mean the memory allocation, we do need a 
buffer to store the current directory. We also need to canonicalize away 
things like \..\, though we may be able to get away with it without 
allocating.


So, in many cases, it's NOT really "a good deal of extra logic that 
performs additional OS calls and generates additional GC garbage".


- If it actually does, are those additional, necessarily OS calls 
prohibitively expensive?


They are certainly going to be less expensive that actual filesystem 
operations that hit the physical disk,


Sounds like QED to me. Especially if the alternative is silently 
incorrect behaviour on an entirely realistic subset of cases. (Realistic 
enough that both the thread's OP and the bug report's OP each ran into 
purely by accident.)


but it will still be an unwanted 
overhead in 99.9% of cases.




That's an extremely exaggerated figure, and we've already established 
that the overhead is minor and often able to be elided. Weighted against 
the cost of incorrect behaviour on a subset of non-rejected inputs, I'd 
say that's a very clear "Yes, please!"


The extreme minority of currently-hypothetical cases which require 
minimal overhead for individual file I/O operations are free to 
low-level optimize themselves as-needed. Correct behaviour should never 
be sacrificed for minor performance tweaks when the minor performance 
tweak can still be obtained through other means if absolutely necessary.



In any case, the overhead is only one issue.



What's the other issue(s)?

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky 
(Abscissa) wrote:
Someone mentioned in this thread that .NET runtime does do the 
long-path workaround automatically. One thing we could do is 
copy EXACTLY what C# is doing.




This is a complete textbook example of the "appeal to 
authority" fallacy.


If an approach is valid, then it stands on its own merits 
regardless of whether or not Microsoft implemented it.


If an approach in invalid, then it fails on its own demerits 
regardless of whether or not Microsoft implemented it.


What MS has or hasn't implemented and released is completely 
irrelevant WRT validity and correctness.


What it *might* be useful for is as a starting point for 
further exploration. But that is all.


No, absolutely not.

Microsoft is in charge of the implementation. We can't know that 
any deviation from Microsoft's algorithm will work in all 
situations, or all past/future implementations of the API. There 
is the considerable possibility that there are situations which 
we cannot foresee from our limited knowledge of the problem and 
Windows API implementation; on the other hand, Microsoft not only 
has complete knowledge of the implementation, but also controls 
its future. They have an incentive to keep the .NET algorithm 
working.


If we deviate from the .NET algorithm and D breaks (but not C#), 
it is our fault.


If we implement the .NET algorithm, then we are as good as C#. If 
it breaks, it's Microsoft's fault.


You cannot evaluate any intrinsic merit here because the result 
is beyond your control.


If one extra OS API call + allocation per std.file API call is 
unacceptable, then explain how it is unacceptable. I disagree 
that it is significant enough to be unacceptable.


It is not unacceptable, but it is a drawback.

If a user needs to optimize their 
already-working-for-all-accepted-inputs application, then they 
are free to do so. I argue that building this into the standard 
library's default behaviour amounts to mandatory premature 
optimization, prioritizing premature optimization over 
correctness. Prove me wrong.


You could extend this argument to any severity of workarounds. 
Where do you draw the line?


- Using paths longer than MAX_PATH is an exceptional 
situation. Putting the workaround in the main code path 
penalizes 99.9% of use cases.


I have many filepaths on my system right now which exceed 
MAX_PATH in total length. I submit that this "penalty" you 
speak of is nothing more than a trivial performance boost at 
the expense of correctness. Furthermore, I submit that long 
paths which need extra optimization are MORE exceptional than 
long paths which do NOT need extra optimization.


Optimization is the least concern.

- The registry switch in newer Windows versions removes the 
need for this workaround, so systems with it enabled are 
penalized as well.


Using the Phobos-based workaround on a system WITH the longpath 
setting supported and enabled results in slightly reduced 
performance (which can be overridden and optimized when 
necessary).


I'm more concerned about differences in behavior.

OTOH, NOT using the Phobos-based workaround on a system where 
the longpath setting is NOT supported *OR* NOT enabled results 
in erroneous behavior.


I disagree that failure on paths exceeding MAX_PATH is 
necessarily erroneous behavior. The API reports an error given 
the user's path, so should Phobos.


The superior default is clear: Use the workaround except where 
the workaround in known to be safe to omit.


We don't even have an algorithm for determining for sure when the 
workaround is needed.



- There is still the matter regarding special filenames,


If you're referring to NUL, COM1, COM2, etc, then this is 
completely orthogonal.


Yes. How so? It is the same issue: paths with certain properties 
are valid on all platforms except on Windows. Phobos errors out 
when attempting to access/create them. A simple workaround is 
available: expand/normalize the path, prepend the UNC prefix, and 
use Unicode APIs.


as well as whet her the expected behavior is really to succeed 
and create paths inaccessible to most software, instead of 
failing.


Ok, suppose we decide "Sure, we have reason to believe there 
may be a significant amount of software on Windows which fails 
to handle long paths and we want to ensure maximum 
compatibility with those admittedly broken programs." That's 
fine. I can get behind that. HOWEVER, that does NOT mean we 
should leave our APIs as they are, because currently, our APIs 
fail at that goal. Instead, what it really means is that our 
APIs should be designed to *REJECT* long paths with an 
appropriately meaningful error message - and a reasonable 
workaround - and NOT to blindly just pass them along as they 
currently do.


This is not possible, because you need to precisely know how the 
implementation will handle the path. Considering the 
implementation's behavior can be configured by

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

They are certainly going to be less expensive that actual 
filesystem operations that hit the physical disk, but it will 
still be an unwanted overhead in 99.9% of cases.


In any case, the overhead is only one issue.


Seriously, checking the file path string *length* is above 260 
characters to see if it needs to be fixed is not what I call an 
overhead.


And IF the path is indeed too long, IN THOSE CASES personally I'd 
prefer that the D standard library fixes the path in order to 
make the disk/file operation succeed, than having my application 
crash, because I didn't know I had to put a "version ( Windows )" 
fix somewhere in my code.


But hey, I may be wrong, software robustness and stability is 
often much overrated... ;)

Re: phobo's std.file is completely broke!


On 09/19/2018 02:26 AM, Vladimir Panteleev wrote:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

[...]


Someone mentioned in this thread that .NET runtime does do the long-path 
workaround automatically. One thing we could do is copy EXACTLY what C# 
is doing.




This is a complete textbook example of the "appeal to authority" fallacy.

If an approach is valid, then it stands on its own merits regardless of 
whether or not Microsoft implemented it.


If an approach in invalid, then it fails on its own demerits regardless 
of whether or not Microsoft implemented it.


What MS has or hasn't implemented and released is completely irrelevant 
WRT validity and correctness.


What it *might* be useful for is as a starting point for further 
exploration. But that is all.



However, there are still drawbacks to this:

- There is still the matter of overhead (one OS API call 
(GetCurrentDirectory) and at least one GC allocation (for the current 
directory buffer)).


If one extra OS API call + allocation per std.file API call is 
unacceptable, then explain how it is unacceptable. I disagree that it is 
significant enough to be unacceptable.


If a user needs to optimize their 
already-working-for-all-accepted-inputs application, then they are free 
to do so. I argue that building this into the standard library's default 
behaviour amounts to mandatory premature optimization, prioritizing 
premature optimization over correctness. Prove me wrong.


- Using paths longer than MAX_PATH is an exceptional situation. Putting 
the workaround in the main code path penalizes 99.9% of use cases.


I have many filepaths on my system right now which exceed MAX_PATH in 
total length. I submit that this "penalty" you speak of is nothing more 
than a trivial performance boost at the expense of correctness. 
Furthermore, I submit that long paths which need extra optimization are 
MORE exceptional than long paths which do NOT need extra optimization.


- The registry switch in newer Windows versions removes the need for 
this workaround, so systems with it enabled are penalized as well.


Using the Phobos-based workaround on a system WITH the longpath setting 
supported and enabled results in slightly reduced performance (which can 
be overridden and optimized when necessary). Note that it is possible 
(and very simple) to detect this situation and handle it optimally by 
skipping the workaround.


OTOH, NOT using the Phobos-based workaround on a system where the 
longpath setting is NOT supported *OR* NOT enabled results in erroneous 
behavior. Not something as trivial as slightly-degraded performance on 
IO access.


The superior default is clear: Use the workaround except where the 
workaround in known to be safe to omit.



- There is still the matter regarding special filenames,


If you're referring to NUL, COM1, COM2, etc, then this is completely 
orthogonal.


as well as 
whet her the expected behavior is really to succeed and create paths 
inaccessible to most software, instead of failing.


Ok, suppose we decide "Sure, we have reason to believe there may be a 
significant amount of software on Windows which fails to handle long 
paths and we want to ensure maximum compatibility with those admittedly 
broken programs." That's fine. I can get behind that. HOWEVER, that does 
NOT mean we should leave our APIs as they are, because currently, our 
APIs fail at that goal. Instead, what it really means is that our APIs 
should be designed to *REJECT* long paths with an appropriately 
meaningful error message - and a reasonable workaround - and NOT to 
blindly just pass them along as they currently do.


Either way, Phobos needs changed:

Do you believe D should prevent its own software from being broken on 
long paths? Then Phobos should be modified to detect and fix long paths.


Do you believe D should permit breakage on long paths and encourage its 
programs to play nicely with other non-D Windows software that is *also* 
broken on long paths? Then Phobos should be modified to detect and 
*reject* long paths.


Either way, the current Phobos behavior is clearly the worst of both 
worlds and needs modification.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder 
wrote:
None would ever be, considering you obviously have decided to 
ignore such a simple solution to the 260 character limit...


Add "ad hominem" to your pile of fallacies, I guess.


Now I will, thanks :)

Once again, this forum proves to be very effective at removing 
any motivation from D users to get involved and contribute to the 
D language.


That's probably one of the keys of its success...

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky 
(Abscissa) wrote:
- Does it actually, necessarily perform those additional OS 
calls?


We need to expand relative paths to absolute ones, for which we 
need to fetch the current directory.



- Is it really?


Is what really what? If you mean the memory allocation, we do 
need a buffer to store the current directory. We also need to 
canonicalize away things like \..\, though we may be able to get 
away with it without allocating.


- If it actually does, are those additional, necessarily OS 
calls prohibitively expensive?


They are certainly going to be less expensive that actual 
filesystem operations that hit the physical disk, but it will 
still be an unwanted overhead in 99.9% of cases.


In any case, the overhead is only one issue.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Jonathan Marler via Digitalmars-d

On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
Panteleev wrote:
On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:

[...]


One more thing:

There is the argument that the expected behavior of Phobos 
functions creating filesystems objects with long paths is to 
succeed and create those files. However, this results in 
filesystem objects that most software will fail to access 
(everyone needs to also use the long paths workaround).


One point of view is that the expected behavior is that the 
functions succeed. Another point of view is that Phobos should 
not allow programs to create files and directories with invalid 
paths. Consider, e.g. that a user writes a program that creates 
a large tree of deeply nested filesystem objects. When they are 
done and wish to delete them, their file manager fails and 
displays an error. The user's conclusion? D sucks because it 
corrupts the filesystem and creates objects they can't operate 
with.


I was wanting to reply with something similar:)

My 2 cents..whatever it's worth.  Vladimir has expressed most if 
not all the points I would have brought up.  Abscissa did bring 
up a good idea to help users support long filenames, but I agree 
with Vladimir that this should be "opt-in".  Provide a function 
in phobos for it, plus, it lets them cache the result AND 
infinitely better, the developer knows what's going on.  What 
drives me mad is when you have library writers who try to 
"protect" you from the underlying system by translating 
everything you do into what they "think" you're trying to do.  
This will inevitably result in large complex adaptation layers as 
both the underlying system and the front-facing API change over 
time with unwieldy maintenance burden.  An opt-in solution 
doesn't have this problem because you've kept each solution 
orthogonal rather than developing a translation layer that needs 
to be able to determine what the underlying system does or does 
not support.  This is a fundamental example of encapsulation, the 
filesystem library should be it's own component with the windows 
filesystem workaround being an optional "add-on" that the 
filesystem library doesn't need to know about.  This workaround 
could look like an extra function in phobos...or you could even 
write a module that wraps std.file and does the translation on a 
per-call basis.

Re: phobo's std.file is completely broke!


On 09/19/2018 12:04 AM, Vladimir Panteleev wrote:
On Wednesday, 19 September 2018 at 01:50:54 UTC, Nick Sabalausky 
(Abscissa) wrote:
And at least for me, moving from Windows to Linux would have been a 
LOT harder if it weren't for the OS abstractions that are already in 
Phobos.


It's one thing to call unlink on POSIX and RemoveFileW on Windows. 


Granted.

Another is adding a good deal of extra logic that performs additional OS 
calls and generates additional GC garbage to work around API problems 
even on systems that don't need it.


- Is it really?

- Does it actually, necessarily perform those additional OS calls?

- If it actually does, are those additional, necessarily OS calls 
prohibitively expensive? (Note that this is being compared to the 
theoretical minimum of successfully performing the same desired 
operation on the same data via the WinAPI, and not compared to the 
software which fails to perform appropriate checks for invalid input.)


- How have you determined that the "additional GC garbage" required to 
"work around API problems" is still significant "even on systems that 
don't need it"?

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 06:16:21 UTC, Paolo Invernizzi 
wrote:
On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:


Operating on paths longer than MAX_PATH is not a typical 
situation.


https://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/1499/

The worst situation was node.js on Windows,  anyway...


Not sure that's actually MAX_PATH related... cmd has a very low 
limit on command line length. Ran into this myself:


https://github.com/VerySleepy/tests/blob/721e52fcb14d8134394264586a4fe92e73574059/scripts/toolchains_download.cmd#L8

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

[...]


Someone mentioned in this thread that .NET runtime does do the 
long-path workaround automatically. One thing we could do is copy 
EXACTLY what C# is doing.


The rationale being that:
- .NET is made by Microsoft
- The Windows API's filesystem implementation is made by Microsoft
- Given that these two are made by the same party, it's 
reasonable to assume that the .NET authors authoritatively "knew 
what they were doing" when implementing the workaround.
- The algorithm used by .NET is very likely to be supported by 
the API (even future implementations), as well as third-party 
implementations of the API.


However, there are still drawbacks to this:

- There is still the matter of overhead (one OS API call 
(GetCurrentDirectory) and at least one GC allocation (for the 
current directory buffer)).
- Using paths longer than MAX_PATH is an exceptional situation. 
Putting the workaround in the main code path penalizes 99.9% of 
use cases.
- The registry switch in newer Windows versions removes the need 
for this workaround, so systems with it enabled are penalized as 
well.
- There is still the matter regarding special filenames, as well 
as whether the expected behavior is really to succeed and create 
paths inaccessible to most software, instead of failing.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Paolo Invernizzi via Digitalmars-d

On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:


Operating on paths longer than MAX_PATH is not a typical 
situation.


https://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/1499/

The worst situation was node.js on Windows,  anyway...

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:

[...]


One more thing:

There is the argument that the expected behavior of Phobos 
functions creating filesystems objects with long paths is to 
succeed and create those files. However, this results in 
filesystem objects that most software will fail to access 
(everyone needs to also use the long paths workaround).


One point of view is that the expected behavior is that the 
functions succeed. Another point of view is that Phobos should 
not allow programs to create files and directories with invalid 
paths. Consider, e.g. that a user writes a program that creates a 
large tree of deeply nested filesystem objects. When they are 
done and wish to delete them, their file manager fails and 
displays an error. The user's conclusion? D sucks because it 
corrupts the filesystem and creates objects they can't operate 
with.

Re: phobo's std.file is completely broke!

2018-09-19 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:
This actually leads to an interesting point. Let's change gears 
for a moment to "API Design Theory"...


Suppose you implement API function X which takes Y as an 
argument. Question: Should X accept values (or types) of Y for 
which X fails? Answer: Ideally, no. At least to the extent 
realistically possible.


So what should a well-designed function X do when faced with 
known-to-be unsupported input Y?[1] Here are the 
possibilities[2]:


([1] "Unsupported" is defined here as "for the given input, the 
function does not correctly and successfully perform its 
intended goal".)


I don't think that accurately describes the situation. 
Considering how there is now a toggle which changes the API 
implementation's behavior, the limitation is no longer really a 
part of the API, but a part of the implementation.


Consider, also, other implementations of the Win32 API (Wine and 
ReactOS), as well as possible future versions of Windows (which 
may do away with this limitation entirely).



1. Add support for input Y to function X.
2. Reject input Y (via either: abort, throw or static compile 
error)
3. Accept input Y and allow one of the following to occur[3]: 
Silently fail to perform the expected task, silently do the 
wrong thing, or trigger an unclear error 
(assert/exception/compile) from deeper in the callstack. ([3] 
Note that which one of these three possibilities occurs is 
dependent on function X's exact implementation details.)


Of these three possibilities for a function X faced with 
unsupported input Y, the first two options are (in theory) 
acceptable[4]. The third possibility is absolutely not 
acceptable [to the extent realistically possible.]


This is, again, flawed logic. It is unreasonable to expect what 
the implementation of the underlying API will do, because it is 
outside of our control. Consider that this discussion would have 
occurred just before Microsoft added the registry switch to allow 
long path names. The solution would have been "obvious"; the 
reality is that it is an implementation we do not control and can 
change in the future at whim. At best, we can treat the API at 
face value and don't attempt to work around implementation quirks.


Consider also the case of special filenames like "con" or "prn". 
Creating these using the standard API is not allowed, but this 
can also be bypassed with UNC paths. All the arguments in favor 
of making Phobos support paths longer than MAX_PATH (using the 
UNC prefix) seem to also favor detecting and supporting these 
reserved file names. But is doing so really Phobos' burden?


We simply cannot define our API in terms of what we think the 
underlying implementation supports or doesn't.



No, that's completely nuts!
A library, especially a standard library, should not introduce 
new limitations, but pampering over the limitations of the 
platform is not the right thing to do.


This is debatable. Why, exactly, is pampering over 
pre-Win10-v1607's maximum non-\\?\ filepath length a bad thing? 
Exactly what problems does it cause?


https://forum.dlang.org/post/bqsjebjxuljlqusao...@forum.dlang.org

If the platforms API is piling POS, there's nothing a sane 
library can do about.


That is patently untrue. It might be true in specific 
circumstances, but it is not generally true. If you believe it 
to be true in this specific case, then please explain 
*how*/*why* there is nothing the library can do about it.


It is not Phobos' job to work around quirks in implementations 
beyond our control which can change at any moment.


The general rule of thumb is: "Typical situations should work 
as expected, atypical situations should be possible."


Operating on paths longer than MAX_PATH is not a typical 
situation.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/17/2018 11:27 AM, Patrick Schluter wrote:

On Monday, 17 September 2018 at 12:37:13 UTC, Temtaime wrote:


It's problem with phobos.
It should be able handle all the paths whatever length they have, on 
all the platforms without noising the user.


Even with performance penalty, but it should.


This actually leads to an interesting point. Let's change gears for a 
moment to "API Design Theory"...


Suppose you implement API function X which takes Y as an argument. 
Question: Should X accept values (or types) of Y for which X fails? 
Answer: Ideally, no. At least to the extent realistically possible.


So what should a well-designed function X do when faced with known-to-be 
unsupported input Y?[1] Here are the possibilities[2]:


([1] "Unsupported" is defined here as "for the given input, the function 
does not correctly and successfully perform its intended goal".)


([2] Again, we're assuming "to the extent realistically possible" is 
still in play here.)


1. Add support for input Y to function X.
2. Reject input Y (via either: abort, throw or static compile error)
3. Accept input Y and allow one of the following to occur[3]: Silently 
fail to perform the expected task, silently do the wrong thing, or 
trigger an unclear error (assert/exception/compile) from deeper in the 
callstack. ([3] Note that which one of these three possibilities occurs 
is dependent on function X's exact implementation details.)


Of these three possibilities for a function X faced with unsupported 
input Y, the first two options are (in theory) acceptable[4]. The third 
possibility is absolutely not acceptable [to the extent realistically 
possible.]


([4] The library containing function X may impose additional 
restrictions on what is/isn't acceptable.)


So, what does this mean in our specific situation?

If good API design is followed, then any filesystem-based API, when 
running on Windows and faced with a path exceeding the local system's 
size limit, must do one of the following:


1. Modify Phobos to support long filepaths even on Windows versions 
below Win10 v1607.

2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5].
3. Provide a very good justification why possibilities #1 and #2 are 
sufficiently unrealistic/problematic.


([5] Note that the following still technically satisfy possibility #2 
since they do not involve passing unsupported input to the given 
function, but they could still be optionally determined unacceptable if 
desired: A. Rejecting all paths longer than MAX_PATH-12 bytes, even if 
\\?\-based. B. Rejecting too-long paths even on Win10 v1607+ with 
LongPathsEnabled set in the registry.)




No, that's completely nuts!
A library, especially a standard library, should not introduce new 
limitations, but pampering over the limitations of the platform is not 
the right thing to do.


This is debatable. Why, exactly, is pampering over pre-Win10-v1607's 
maximum non-\\?\ filepath length a bad thing? Exactly what problems does 
it cause?


If the platforms API is piling POS, there's 
nothing a sane library can do about.


That is patently untrue. It might be true in specific circumstances, but 
it is not generally true. If you believe it to be true in this specific 
case, then please explain *how*/*why* there is nothing the library can 
do about it.


If your app writes to a FAT12 formatted floppy disk you don't expect the 
library to implement code to alleviate its limitation, like 8+3 
filenames or fixed number of files in the root directory.


The general rule of thumb is: "Typical situations should work as 
expected, atypical situations should be possible." Therefore, please 
explain *ANY* one of the following:


1. How writing to a FAT12 formatted floppy disk qualifies as a typical 
situation.


2. How abstracting over the MAX_PATH limitation makes writing to a FAT12 
formatted floppy impossible.


3. How my claim of "Typical situations should work as expected, atypical 
situations should be possible" is fundamentally wrong for D.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder 
wrote:
None would ever be, considering you obviously have decided to 
ignore such a simple solution to the 260 character limit...


Add "ad hominem" to your pile of fallacies, I guess. I've 
addressed it twice in this thread already - it is problematic for 
technical reasons. It seems you are the one ignoring the problems 
with it...

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

Do the PS2, GameCube and Xbox filesystems all have identical 
file path limits?


Guess ;)

And, did any of the paths in your game exceed 260 characters in 
length?


No. But the suggested GetPhysicalPath() solution would also work 
equally well in this case.



These comparisons are not helpful.


None would ever be, considering you obviously have decided to 
ignore such a simple solution to the 260 character limit...

Re: phobo's std.file is completely broke!

2018-09-18 Thread Vladimir Panteleev via Digitalmars-d

On Tuesday, 18 September 2018 at 18:04:19 UTC, Ecstatic Coder 
wrote:
There will always be inherent differences between platforms, 
because they are wildly different.


Right.

Technically the PS2 console, the GameCube and the Xbox console 
were very different from each other, so I had no choice but to 
implement low-level abstraction function (GetPhysicalPath() 
etc) to make the file system classes work similarly across all 
four systems.


Do the PS2, GameCube and Xbox filesystems all have identical file 
path limits?


And, did any of the paths in your game exceed 260 characters in 
length?


These comparisons are not helpful.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 01:50:54 UTC, Nick Sabalausky 
(Abscissa) wrote:
And at least for me, moving from Windows to Linux would have 
been a LOT harder if it weren't for the OS abstractions that 
are already in Phobos.


It's one thing to call unlink on POSIX and RemoveFileW on 
Windows. Another is adding a good deal of extra logic that 
performs additional OS calls and generates additional GC garbage 
to work around API problems even on systems that don't need it.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Vladimir Panteleev via Digitalmars-d

On Wednesday, 19 September 2018 at 02:20:45 UTC, Nick Sabalausky 
(Abscissa) wrote:
3. Building on what Vladimir and Jay have said in the bug 
report, I propose we do this:


This has been proposed before in this thread. I don't think it's 
a good idea:


https://forum.dlang.org/post/bqsjebjxuljlqusao...@forum.dlang.org

I proposed something a little different:

https://forum.dlang.org/post/dnlbtkmdcjucqetkw...@forum.dlang.org

Re: phobo's std.file is completely broke!


On 09/18/2018 09:46 PM, Jonathan M Davis wrote:

On Tuesday, September 18, 2018 7:28:43 PM MDT Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:


It's worth noting that the discussion made it very clear that Walter's
viewpoint on the matter was based on his own misunderstanding (ie,
mistakenly failed to notice that `\\?\` != `\\.\`) He never actually
made any comment after that was pointed out.


If a clean, simple solution can be found that allows long paths on Windows
to work seemlessly without subtle side effects, then I don't see why it
can't be implemented, but it needs to be something that's not going to cause
problems. Otherwise, it needs to be left up to the caller to do whatever
happens to be correct for their particular circumstances. We want Phobos to
work seemlessly across platforms where reasonably possible, but
unfortunately, it's not always reasonable. Either way, Microsoft has clearly
made a mess of this. So, I don't know how reasonable it is to work around
it. Regardless, we either need to figure out a sane way to work around the
problem (without causing new problems in the process) or document it so that
the situation is clear.



Exactly. And this is precisely why I'm irritated by just how much of 
this entire thread has been meaningless high-level philosophical 
hand-wringing, with barely any attention paid to the technical details 
of the problem itself.


Though I admit, I've allowed myself to get dragged into it, too. So 
allow me to rectify that:


1. For reference, here is the relevant MS documentation:
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#file-and-directory-names

2. For reference, here is the existing bug report discussion:
https://issues.dlang.org/show_bug.cgi?id=8967

3. Building on what Vladimir and Jay have said in the bug report, I 
propose we do this:


- We add a public function to Phobos which takes a a UTF-8 path and does 
the following:

- No-op outside of Windows
- No-op if the path is less than MAX_PATH-12 bytes. (Worrying about 
the case where the real limit is MAX_PATH-1 would be inconsequential and 
thus needless.)
- No-op if the path begins with "\\" (and is therefore either a 
network share path, a "\\.\" device path, is already a "\\?\" path, or 
is just some undetectable malformed path that happens to look like one 
of the above)

- Otherwise, returns:
toUTF16z(buildNormalizedPath("\\?", path.toAbsolute))
- By "no-op" above, I really mean: toUTF16z(path)
- In all cases where Phobos passes a path to WinAPI, the path is first 
passed through this function (except for any specific cases where it can 
be shown that the path should NOT be passed through this function).


4. What technical problems does this proposal have?

5. For each technical problem with the proposal: How can the proposal be 
adjusted to compensate? Or, why can the technical problem NOT be 
reasonably solved? Or in general: How should this proposal be modified, 
and why?

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

On 09/15/2018 06:40 AM, Vladimir Panteleev wrote:

On Saturday, 15 September 2018 at 10:05:26 UTC, Josphe Brigmo wrote:

Also, windows 10 does not have this problem

What do you mean by "windows 10"? Do you mean Explorer, the default file
manager?

According to MS docs:

"Starting in Windows 10, version 1607, MAX_PATH limitations have been
removed from common Win32 file and directory functions. However, you
must opt-in to the new behavior."

-- From:
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

It goes on to explain the registry setting for that.

If it did I wouldn't have had this problem and wasted a day of my life
trying to figure out what is going on(I didn't design my program
around having to hack such things, I just assumed they would work,
because, after all, they should, right?).

I, too, spent a LOT of time fighting the Windows filesystem APIs. See
e.g. *
https://dump.thecybershadow.net/d78d9911adc16ec749914b6923759454/longpathdelete.d
(that also sets ownership/ACLs via external processes, as the C API is
unreasonably complicated).

The problem is 100% due to Windows.

It was one of the big reasons why I moved to Linux for software
development. Such problems do not exist there.

Agreed, but this sounds to me like a perfect reason to abstract away as
much as that garbage as we reasonably can. D's in the business of
improving the user-experience of programming, not the promotion of one
OS over another.

Bear in mind, the very point of plenty of things in Phobos, or any std
lib for that matter, is to abstract away the need to waste everyone's
time making them fiddle about with pointless little OS differences.
Sure, there are going to be things that can't be reasonably abstracted
away, but that's FAR from justifying not doing what we can. Again:
normal expectations should be the default and just work, exceptional
needs should still be possible.

And at least for me, moving from Windows to Linux would have been a LOT
harder if it weren't for the OS abstractions that are already in Phobos.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Jonathan M Davis via Digitalmars-d

On Tuesday, September 18, 2018 7:28:43 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d wrote:
> On 09/15/2018 08:14 PM, Jonathan M Davis wrote:
> > The issue was reported in bugzilla quite some time ago.
> >
> > https://issues.dlang.org/show_bug.cgi?id=8967
> >
> > However, while Walter's response on it basically indicates that we
> > should
> > just close it as "won't fix," we never actually did
>
> It's worth noting that the discussion made it very clear that Walter's
> viewpoint on the matter was based on his own misunderstanding (ie,
> mistakenly failed to notice that `\\?\` != `\\.\`) He never actually
> made any comment after that was pointed out.

If a clean, simple solution can be found that allows long paths on Windows
to work seemlessly without subtle side effects, then I don't see why it
can't be implemented, but it needs to be something that's not going to cause
problems. Otherwise, it needs to be left up to the caller to do whatever
happens to be correct for their particular circumstances. We want Phobos to
work seemlessly across platforms where reasonably possible, but
unfortunately, it's not always reasonable. Either way, Microsoft has clearly
made a mess of this. So, I don't know how reasonable it is to work around
it. Regardless, we either need to figure out a sane way to work around the
problem (without causing new problems in the process) or document it so that
the situation is clear.

- Jonathan M Davis

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/15/2018 08:14 PM, Jonathan M Davis wrote:


The issue was reported in bugzilla quite some time ago.

https://issues.dlang.org/show_bug.cgi?id=8967

However, while Walter's response on it basically indicates that we should
just close it as "won't fix," we never actually did


It's worth noting that the discussion made it very clear that Walter's 
viewpoint on the matter was based on his own misunderstanding (ie, 
mistakenly failed to notice that `\\?\` != `\\.\`) He never actually 
made any comment after that was pointed out.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/18/2018 05:25 AM, Vladimir Panteleev wrote:

On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder wrote:


I expect that calling the function F on system X will work the same as 
calling that same function on system Y.


You ask for the impossible.


I think it's safe to assume a "...to the extent reasonably possible." 
suffixed to his statement.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/18/2018 06:14 PM, Bastiaan Veelo wrote:
On Tuesday, 18 September 2018 at 19:57:09 UTC, Nick Sabalausky 
(Abscissa) wrote:
Yes, the OP needs to file a bug report (and if he's already done so, 
then please post a link here for our reference).


It’s an old issue, and the OP posted the link a bit further up in this 
thread: https://forum.dlang.org/post/gyaswfyzwoofaozki...@forum.dlang.org




Ah, don't know how I managed to miss that. Thanks.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Bastiaan Veelo via Digitalmars-d

On Tuesday, 18 September 2018 at 19:57:09 UTC, Nick Sabalausky 
(Abscissa) wrote:
Yes, the OP needs to file a bug report (and if he's already 
done so, then please post a link here for our reference).


It’s an old issue, and the OP posted the link a bit further up in 
this thread: 
https://forum.dlang.org/post/gyaswfyzwoofaozki...@forum.dlang.org

Re: phobo's std.file is completely broke!

2018-09-18 Thread bachmeier via Digitalmars-d

On Tuesday, 18 September 2018 at 19:33:00 UTC, Nick Sabalausky 
(Abscissa) wrote:

On 09/15/2018 09:54 AM, tide wrote:

On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:
On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
wrote:
For very long file names it is broke and every command 
fails. These paths are not all that long but over 256 limit. 
(For windows)


Please file a bug report with reproducible examples if you 
believe it's a bug.


I feel people need to stop saying this. It feels like people 
are just being told to say this if there is a bug. There is a 
larger issue, Bugzilla doesn't and isn't working. Someone will 
probably throw up some stats about how many bugs are filed and 
how many are resolved. Those exist because someone working on 
Dlang comes across a bug that affects them, creates a patch 
for it first, then goes and creates a bugzilla entry and marks 
it resolved. Issues are rarely resolved by anyone other than 
the person that created the bug report to begin with. Or 
issues created by a team member is resolved by another team 
member.


While that's admittedly all-too-true, filing a proper bug 
report is still an essential step.


Like you, I'm all for bringing attention to important issues on 
the newsgroup. However, it is CRUCIAL for this to be IN 
ADDITION to filing a bug report, and NOT INSTEAD of filing a 
bug report.


Correct. There's no point in having a lengthy discussion on the 
topic without a bug report because it'll just waste their time 
and then they'll complain that their bug is being ignored. Unless 
there is a change in the operation of this project, all bugs 
reported in the forum need to be addressed by telling them to 
file a bug report.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/15/2018 08:09 PM, Vladimir Panteleev wrote:

On Saturday, 15 September 2018 at 23:50:43 UTC, Josphe Brigmo wrote:

[...]


D is generally described as a system programming language. There is 
value in favoring a simple and obvious implementation ("do what I say") 
over going out of one's way to make usage simpler ("do what I mean"). 
The tradeoff is performance and complexity. Performance is generally an 
important factor for users of system programming languages, and 
complexity is a source of unforeseen problems in non-trivial use cases.


Consider, for example, how integers are treated in D and Python. D's 
integers are fixed-length and roll over on overflow. Python integers are 
bigints, which makes them slower, but can handle numbers of any size.


 From your posts, it sounds like you're looking for a programming 
language closer to Python than to D.




D's philosophy is (or at least is supposed to be) "Whenever possible, 
the right thing should be default, alternatives should still be 
possible." (And if I'm mistaken on that, then I've very much chosen the 
wrong language to use.)


Note that this does not contradict D's integer behaviour, because 
changing that would be *prohibitively* expensive and cause the need for 
workarounds to be common yet very difficult (if even possible). 
Therefore, it is a justifiable exception to the rule.


By contrast, I find it difficult to believe that a fix to the OP's 
problem would be prohibitively expensive. Of course, maybe I'm wrong. 
And maybe, as some have vaguely suggested, the WinAPI makes a correct 
fix impossible. **BUT**, unfortunately, I (as well as the OP) have no 
way to know because there appears to be a distressingly blatant LACK of 
technical discussion on the specific OP problem itself. Instead, all I'm 
seeing is a bunch of high-level design-philosophy bickering left 
completely divorced from the specifics of the original problem in question.


Yes, the OP needs to file a bug report (and if he's already done so, 
then please post a link here for our reference). But sheesh, people, 
it's *no wonder* he's gotten upset about it. I would too, and so would 
most of the rest of you (I've seen it happen).

Re: phobo's std.file is completely broke!

2018-09-18 Thread Nick Sabalausky (Abscissa) via Digitalmars-d


On 09/15/2018 09:54 AM, tide wrote:

On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
For very long file names it is broke and every command fails. These 
paths are not all that long but over 256 limit. (For windows)


Please file a bug report with reproducible examples if you believe 
it's a bug.


I feel people need to stop saying this. It feels like people are just 
being told to say this if there is a bug. There is a larger issue, 
Bugzilla doesn't and isn't working. Someone will probably throw up some 
stats about how many bugs are filed and how many are resolved. Those 
exist because someone working on Dlang comes across a bug that affects 
them, creates a patch for it first, then goes and creates a bugzilla 
entry and marks it resolved. Issues are rarely resolved by anyone other 
than the person that created the bug report to begin with. Or issues 
created by a team member is resolved by another team member.


While that's admittedly all-too-true, filing a proper bug report is 
still an essential step.


Like you, I'm all for bringing attention to important issues on the 
newsgroup. However, it is CRUCIAL for this to be IN ADDITION to filing a 
bug report, and NOT INSTEAD of filing a bug report.

Re: phobo's std.file is completely broke!

There will always be inherent differences between platforms, 
because they are wildly different.


Right.

Technically the PS2 console, the GameCube and the Xbox console 
were very different from each other, so I had no choice but to 
implement low-level abstraction function (GetPhysicalPath() etc) 
to make the file system classes work similarly across all four 
systems.


That wasn't an easy task, but it made the life so much easier for 
the game programmers that it was obvious this was "the right 
thing" to do.


The fact that D's standard library has already bitten me several 
time with its platform specific problem clearly shows that you 
have chosen another path.


That's your right, but don't expect those who develop 
cross-platform tools in D to be happy to HAVE to put ugly 
"version ( ... )" stuff in their code when their software 
suddenly break on some platforms for unknown (= undocumented) 
reasons...

Re: phobo's std.file is completely broke!

2018-09-18 Thread Vladimir Panteleev via Digitalmars-d

On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder 
wrote:
This attitude is unfortunately the cause of a lot frustration 
among cross-platform developers like me.


I chose D for my file scripting needs because it's a 
cross-platform language.


I expect that calling the function F on system X will work the 
same as calling that same function on system Y.


You ask for the impossible.

How do you expect the following to "work the same" across all 
platforms:


import std.stdio;
import std.file;
auto fn = "a.txt";
auto f = File(fn, "w");
remove(fn);

or this:

import std.file;
write("a", "a");
write("A", "A");
assert(readText("a") == "a");
assert(readText("A") == "A");

There will always be inherent differences between platforms, 
because they are wildly different. Using this sentiment as an 
argument for Phobos to smooth over this one particular difference 
that you care about, and for which incidentally an apparent 
solution appears to be available, is fallacious.

Re: phobo's std.file is completely broke!

2018-09-18 Thread Kagamin via Digitalmars-d

On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder 
wrote:
I expect that calling the function F on system X will work the 
same as calling that same function on system Y.


That's the contract in cross-platform programming.


Heh, I remember working around a filesystem that doesn't support 
unicode, I used bullet • as utf8 prefix and heuristically 
detected that names starting with that prefix are utf8 encoded 
and decoded from that.

Re: phobo's std.file is completely broke!


On Monday, 17 September 2018 at 22:58:46 UTC, tide wrote:
On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
Panteleev wrote:

On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:
Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Funny the other guy was saying to create a bugzilla issue.


Do that *first*.


That's already been done.

The path needs to be normalized, which means that \.\ and 
\..\ fragments need to be removed away first. Depending on 
your interpretation of symlinks/junctions/etc., 
"foo/bar/../" might mean something else than "foo/" if "bar" 
is a reparse point.


All these issues yet for some reason this function was 
included in the lot: 
https://dlang.org/phobos/std_path.html#absolutePath

[...]
This issue exists anyways, you'd only expand the path when it 
need to be used. If the file changes within milliseconds, I 
don't see that happening often and if it does there's a flaw 
in your design that'd happen even if the path didn't have to 
be constructed first.


You've missed the point. Complexity breeds bugs and unexpected 
behavior. The expectation is that D's function to delete a 
file should do little else than call the OS function.


If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


version(Windows)
{
if(path.length >= MAX_PATH)
{
// throw Exception(...) // effectively what happens now

// do workaround for
}
}

The complexity would only exist for those that need it. It'd be 
the difference between their code not working and code working. 
I'm sure people would rather their code work than not work in 
this case.


So you pass a valid path (selected by a user through a UI) to 
rmDir, and it doesn't remove the directory. You think this is 
acceptable behavior?


It is absolutely not acceptable behavior. Complain to 
Microsoft. The OS should not allow users to create or select 
paths that programs cannot operate on without jumping through 
crazy hoops.


Not that crazy, you can get the actual absolutePath with one of 
the OS functions. It isn't that difficult of a workaround.


"Workaround" ;)

That's the problem actually.

As suggested previously, the std.file functions should call a 
GetPhysicalPath function which just returns the path unchanged on 
Linux and MacOS, and on Windows simply checks if the file path is 
smaller or not than the 256 character limit, and if needed makes 
it absolute and prefixes it.


This has no performance impact, and brings a consistent behavior 
across platforms.


THAT would be a nice solution for the cross-platform developers 
who erroneously think that the standard library is already Doing 
The Right Thing (TM) so that their code doesn't need 
platform-specific "workarounds"...

Re: phobo's std.file is completely broke!

2018-09-17 Thread Nick Sabalausky (Abscissa) via Digitalmars-d

On Saturday, 15 September 2018 at 23:06:57 UTC, Jonathan M Davis 
wrote:
On Saturday, September 15, 2018 6:54:50 AM MDT Josphe Brigmo 
via Digitalmars-d wrote:

On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe

wrote:
> On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo
>
> wrote:
>> Phobos *NEEDS* to be modified to work with these newer OS's.
>
> You need to look at the source code before posting. The code 
> for remove is literally

>
> DeleteFileW(name);
>
> it is a one-line wrapper, and obviously uses the unicode 
> version.

>
> https://github.com/dlang/phobos/blob/master/std/file.d#L1047

It doesn't matter, the fact is that something in phobos is 
broke. Do you really expect me to do all the work? The fact 
that using executeShell or "\\?\" solves 90% of the 
problems(maybe all of them) proves that phobos is not up to 
par.

Using std.file should be on par with using the Windows API from 
C or C++. It doesn't try to fix the arguably broken behavior of 
the Windows API with regards to long paths but requires that 
the programmer deal with them just like they would in C/C++. 
The main differences are that the std.file functions in 
question use D strings rather than C strings, and they 
translate them to the UTF-16 C strings for you rather than 
requiring you to do it. But they don't do anything like add 
"\\?\" for you any more than the Windows API itself does that.

If you consider that to be broken, then sorry. For better or 
worse, it was decided that it was better to let the programmer 
deal with those intricacies rather than trying to tweak the 
input to make it work based on the idea that that could have 
undesirable consequences in some circumstances. On some level, 
that does suck, but the Windows API does not make it easy to 
make this work like it would on a *nix system without 
introducing subtle bugs.

If you find that the std.file functions don't work whereas 
using the same input to the Windows API functions in C/C++ 
would have, then there's a bug in the D code, and it needs to 
be fixed, but if it acts the same as the C/C++ code, then it's 
working as intended.

- Jonathan M Davis

This attitude is unfortunately the cause of a lot frustration 
among cross-platform developers like me.

I chose D for my file scripting needs because it's a 
cross-platform language.

I expect that calling the function F on system X will work the 
same as calling that same function on system Y.

That's the contract in cross-platform programming.

Unfortunately D fails at being consistent.

I recently learned this lesson with my Resync tool.

No everybody wants the cross-platform to behave inconsistently.

For example, in the past I've implemented a proprietary 
cross-platform C++ game engine for Windows, PS2, Xbox and 
GameCube.

The games needed some tuning for the graphics, etc.

But code-wise, the engine made the games behave consistently 
across the different platforms.

This was all about making each method of each class behaving the 
same. As simple as that.

Indeed, on some platforms, the game engine also provided extra 
classes and/or methods to add some functionalities specific to 
these platforms.

But the common trunc was implemented (!) to behave the same. That 
was what our game developers expected...

Re: phobo's std.file is completely broke!

2018-09-17 Thread Vladimir Panteleev via Digitalmars-d


On Monday, 17 September 2018 at 22:58:46 UTC, tide wrote:

version(Windows)
{
if(path.length >= MAX_PATH)
{
// throw Exception(...) // effectively what happens now

// do workaround for
}
}

The complexity would only exist for those that need it. It'd be 
the difference between their code not working and code working. 
I'm sure people would rather their code work than not work in 
this case.


No good:

1. When hitting the situation where the extra logic does make a 
difference, and the program is operating on paths with some being 
under the the limit and some over, this will make it behave 
inconsistently depending on the data it's operating on.


2. When the registry key you mentioned is set, the workaround is 
unnecessary, and the extra logic can introduce unwanted behavior.


Not that crazy, you can get the actual absolutePath with one of 
the OS functions. It isn't that difficult of a workaround.


Which OS function is that, for the record?

Re: phobo's std.file is completely broke!

2018-09-17 Thread tide via Digitalmars-d

On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir Panteleev 
wrote:

On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:
Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Funny the other guy was saying to create a bugzilla issue.


Do that *first*.


That's already been done.

The path needs to be normalized, which means that \.\ and 
\..\ fragments need to be removed away first. Depending on 
your interpretation of symlinks/junctions/etc., "foo/bar/../" 
might mean something else than "foo/" if "bar" is a reparse 
point.


All these issues yet for some reason this function was 
included in the lot: 
https://dlang.org/phobos/std_path.html#absolutePath

[...]
This issue exists anyways, you'd only expand the path when it 
need to be used. If the file changes within milliseconds, I 
don't see that happening often and if it does there's a flaw 
in your design that'd happen even if the path didn't have to 
be constructed first.


You've missed the point. Complexity breeds bugs and unexpected 
behavior. The expectation is that D's function to delete a file 
should do little else than call the OS function.


If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


version(Windows)
{
if(path.length >= MAX_PATH)
{
// throw Exception(...) // effectively what happens now

// do workaround for
}
}

The complexity would only exist for those that need it. It'd be 
the difference between their code not working and code working. 
I'm sure people would rather their code work than not work in 
this case.


So you pass a valid path (selected by a user through a UI) to 
rmDir, and it doesn't remove the directory. You think this is 
acceptable behavior?


It is absolutely not acceptable behavior. Complain to 
Microsoft. The OS should not allow users to create or select 
paths that programs cannot operate on without jumping through 
crazy hoops.


Not that crazy, you can get the actual absolutePath with one of 
the OS functions. It isn't that difficult of a workaround.

Re: phobo's std.file is completely broke!


On 09/15/2018 06:57 AM, Josphe Brigmo wrote:


You are missing the point, MAX_PATH is more than just phobos. It's built 
in to the windows design. Windows enforces it.


All ansi api calls are limited by MAX_PATH.

The way to fix it is to use the wide api calls which are not limited or 
to use other tricks.


Phobos *NEEDS* to be modified to work with these newer OS's.

You wouldn't like it if phobos limit something that it didn't need that 
you would use, would you?


File operations are so common that this stuff is relevant to all that 
use windows(and some that don't).




As has already been said, *file a bug report*.

When told this before, instead of submitting a proper bug report, you 
simply moved your goalposts and began complaining about very vague, 
general things instead.


You appear to be less interested in getting this, or any other specific 
issue fixed than you are in simply ranting, griping and berating.

Re: phobo's std.file is completely broke!

2018-09-17 Thread Patrick Schluter via Digitalmars-d


On Monday, 17 September 2018 at 12:37:13 UTC, Temtaime wrote:
On Sunday, 16 September 2018 at 22:49:26 UTC, Vladimir 
Panteleev wrote:

To elaborate:

On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
Panteleev wrote:
If *YOU* are OK with the consequences of complexity, 
implement this in YOUR code, but do not enforce it upon 
others.


This is much better done in user code anyway, because you only 
need to expand / normalize the path and prepend the prefix 
only once (root of the directory tree you're operating on), 
instead of once per directory call.


We could add a `string longPath(string)` function in Phobos 
(no-op on POSIX, expands and prepends prefix on Windows). I 
believe I suggested the same in the bug report years ago when 
we discussed it.


It is absolutely not acceptable behavior. Complain to 
Microsoft. The OS should not allow users to create or select 
paths that programs cannot operate on without jumping through 
crazy hoops.


Microsoft could have solved this easily enough:

extern(System) void AllowLongPaths();

Programs (or programming language runtimes) which can handle 
paths longer than MAX_PATH could call that function. It can 
also be used as a hint to the OS that file/directory selection 
dialogs, as you mentioned, are allowed to select paths longer 
than MAX_PATH.


It's problem with phobos.
It should be able handle all the paths whatever length they 
have, on all the platforms without noising the user.


Even with performance penalty, but it should.


No, that's completely nuts!
A library, especially a standard library, should not introduce 
new limitations, but pampering over the limitations of the 
platform is not the right thing to do. If the platforms API is 
piling POS, there's nothing a sane library can do about.
If your app writes to a FAT12 formatted floppy disk you don't 
expect the library to implement code to alleviate its limitation, 
like 8+3 filenames or fixed number of files in the root directory.

Re: phobo's std.file is completely broke!

2018-09-17 Thread Temtaime via Digitalmars-d

On Sunday, 16 September 2018 at 22:49:26 UTC, Vladimir Panteleev 
wrote:

To elaborate:

On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
Panteleev wrote:
If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


This is much better done in user code anyway, because you only 
need to expand / normalize the path and prepend the prefix only 
once (root of the directory tree you're operating on), instead 
of once per directory call.


We could add a `string longPath(string)` function in Phobos 
(no-op on POSIX, expands and prepends prefix on Windows). I 
believe I suggested the same in the bug report years ago when 
we discussed it.


It is absolutely not acceptable behavior. Complain to 
Microsoft. The OS should not allow users to create or select 
paths that programs cannot operate on without jumping through 
crazy hoops.


Microsoft could have solved this easily enough:

extern(System) void AllowLongPaths();

Programs (or programming language runtimes) which can handle 
paths longer than MAX_PATH could call that function. It can 
also be used as a hint to the OS that file/directory selection 
dialogs, as you mentioned, are allowed to select paths longer 
than MAX_PATH.


It's problem with phobos.
It should be able handle all the paths whatever length they have, 
on all the platforms without noising the user.


Even with performance penalty, but it should.

Re: phobo's std.file is completely broke!

2018-09-16 Thread Vladimir Panteleev via Digitalmars-d


To elaborate:

On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir Panteleev 
wrote:
If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


This is much better done in user code anyway, because you only 
need to expand / normalize the path and prepend the prefix only 
once (root of the directory tree you're operating on), instead of 
once per directory call.


We could add a `string longPath(string)` function in Phobos 
(no-op on POSIX, expands and prepends prefix on Windows). I 
believe I suggested the same in the bug report years ago when we 
discussed it.


It is absolutely not acceptable behavior. Complain to 
Microsoft. The OS should not allow users to create or select 
paths that programs cannot operate on without jumping through 
crazy hoops.


Microsoft could have solved this easily enough:

extern(System) void AllowLongPaths();

Programs (or programming language runtimes) which can handle 
paths longer than MAX_PATH could call that function. It can also 
be used as a hint to the OS that file/directory selection 
dialogs, as you mentioned, are allowed to select paths longer 
than MAX_PATH.

Re: phobo's std.file is completely broke!

2018-09-16 Thread Vladimir Panteleev via Digitalmars-d


On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:
Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Funny the other guy was saying to create a bugzilla issue.


Do that *first*.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.


All these issues yet for some reason this function was included 
in the lot: https://dlang.org/phobos/std_path.html#absolutePath

[...]
This issue exists anyways, you'd only expand the path when it 
need to be used. If the file changes within milliseconds, I 
don't see that happening often and if it does there's a flaw in 
your design that'd happen even if the path didn't have to be 
constructed first.


You've missed the point. Complexity breeds bugs and unexpected 
behavior. The expectation is that D's function to delete a file 
should do little else than call the OS function.


If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


So you pass a valid path (selected by a user through a UI) to 
rmDir, and it doesn't remove the directory. You think this is 
acceptable behavior?


It is absolutely not acceptable behavior. Complain to Microsoft. 
The OS should not allow users to create or select paths that 
programs cannot operate on without jumping through crazy hoops.

Re: phobo's std.file is completely broke!

2018-09-16 Thread tide via Digitalmars-d

On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:
There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they 
get so old. From the comments it is not clear that a pull 
request wouldn't be accepted to fix the issue. Personally I 
think phobos should not exception for long file names.


Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Walters concern is that the path will change unexpected for 
the user. Where does that matter for something like rmDir ? 
The user passes a long path, and rmDir swallows it, never to 
be seen again by the user. What does it matter if the path 
gets corrected if it is too long?


It's more than that.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.


The path also needs to be absolute, so it has to be expanded to 
a full path before it can be prefixed with the UNC prefix. 
Given how the current directory is state tied to the process 
(not thread), you get bonus race condition issues. There's also 
issues like the current directory object being renamed/moved; 
then a relative path will no longer correspond to what the 
program thinks the absolute paths is.


All things considered, this goes well into the territory of 
"not D's problem". My personal recommendation: if you care 
about long path names, use an operating system which implements 
them right.


Well my mistake, seems absolutePath is just named incorrectly and 
should be more accurately named concatenatePath.

Re: phobo's std.file is completely broke!

2018-09-16 Thread tide via Digitalmars-d

On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:
There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they 
get so old. From the comments it is not clear that a pull 
request wouldn't be accepted to fix the issue. Personally I 
think phobos should not exception for long file names.


Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Walters concern is that the path will change unexpected for 
the user. Where does that matter for something like rmDir ? 
The user passes a long path, and rmDir swallows it, never to 
be seen again by the user. What does it matter if the path 
gets corrected if it is too long?


It's more than that.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.


The path also needs to be absolute, so it has to be expanded to 
a full path before it can be prefixed with the UNC prefix. 
Given how the current directory is state tied to the process 
(not thread), you get bonus race condition issues. There's also 
issues like the current directory object being renamed/moved; 
then a relative path will no longer correspond to what the 
program thinks the absolute paths is.


All things considered, this goes well into the territory of 
"not D's problem". My personal recommendation: if you care 
about long path names, use an operating system which implements 
them right.


I'd agree with you that it isn't **Phobos** problem, but since 
most of the functions there aren't @system nor @nogc, I do 
believe it is. And if you want @system and @nogc with no safety 
you can go look into core.stdc for that.

Re: phobo's std.file is completely broke!

2018-09-16 Thread tide via Digitalmars-d

On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:
There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they 
get so old. From the comments it is not clear that a pull 
request wouldn't be accepted to fix the issue. Personally I 
think phobos should not exception for long file names.


Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum 
thread, or email Walter/Andrei to ask for a resolution.


Funny the other guy was saying to create a bugzilla issue.

Walters concern is that the path will change unexpected for 
the user. Where does that matter for something like rmDir ? 
The user passes a long path, and rmDir swallows it, never to 
be seen again by the user. What does it matter if the path 
gets corrected if it is too long?


It's more than that.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.


All these issues yet for some reason this function was included 
in the lot: https://dlang.org/phobos/std_path.html#absolutePath


The path also needs to be absolute, so it has to be expanded to 
a full path before it can be prefixed with the UNC prefix. 
Given how the current directory is state tied to the process 
(not thread), you get bonus race condition issues. There's also 
issues like the current directory object being renamed/moved; 
then a relative path will no longer correspond to what the 
program thinks the absolute paths is.


This issue exists anyways, you'd only expand the path when it 
need to be used. If the file changes within milliseconds, I don't 
see that happening often and if it does there's a flaw in your 
design that'd happen even if the path didn't have to be 
constructed first.


All things considered, this goes well into the territory of 
"not D's problem". My personal recommendation: if you care 
about long path names, use an operating system which implements 
them right.


So you pass a valid path (selected by a user through a UI) to 
rmDir, and it doesn't remove the directory. You think this is 
acceptable behavior?

Re: phobo's std.file is completely broke!

2018-09-15 Thread Vladimir Panteleev via Digitalmars-d


On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:
There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they get 
so old. From the comments it is not clear that a pull request 
wouldn't be accepted to fix the issue. Personally I think 
phobos should not exception for long file names.


Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum thread, 
or email Walter/Andrei to ask for a resolution.


Walters concern is that the path will change unexpected for the 
user. Where does that matter for something like rmDir ? The 
user passes a long path, and rmDir swallows it, never to be 
seen again by the user. What does it matter if the path gets 
corrected if it is too long?


It's more than that.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.


The path also needs to be absolute, so it has to be expanded to a 
full path before it can be prefixed with the UNC prefix. Given 
how the current directory is state tied to the process (not 
thread), you get bonus race condition issues. There's also issues 
like the current directory object being renamed/moved; then a 
relative path will no longer correspond to what the program 
thinks the absolute paths is.


All things considered, this goes well into the territory of "not 
D's problem". My personal recommendation: if you care about long 
path names, use an operating system which implements them right.

Re: phobo's std.file is completely broke!

On Sunday, 16 September 2018 at 01:33:52 UTC, Vladimir Panteleev 
wrote:

On Sunday, 16 September 2018 at 01:19:46 UTC, tide wrote:
I guess that's why Bugzilla is a complete disaster. No one, at 
all, is maintaining it. As there are only 2 people that can 
really maintain it, and I don't see either of them commenting 
on bugs to provide direction, at least very often.


Well, I think that's looking at the situation from the wrong 
angle.


Most of D's code was written by volunteer contributors, and 
usually the code's author ends up maintaining that code, at 
least for a while. So, when you find a bug in some part of D 
and can't fix it yourself, looking at who wrote or last 
maintained the relevant code and pinging them would be the 
first step.


There are some things we can improve, like upgrading the 
platform or improving the categorization so people can receive 
notifications when someone files a bug in a Phobos module they 
care about. I've been slowly working on that front 
(https://github.com/CyberShadow/bugzilla-meta), but it doesn't 
change the underlying facts that bugs are most likely to be 
fixed by people who work on the code, not Andrei or Walter or 
any one person "in charge" of Bugzilla.


There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they get 
so old. From the comments it is not clear that a pull request 
wouldn't be accepted to fix the issue. Personally I think phobos 
should not exception for long file names.


Walters concern is that the path will change unexpected for the 
user. Where does that matter for something like rmDir ? The user 
passes a long path, and rmDir swallows it, never to be seen again 
by the user. What does it matter if the path gets corrected if it 
is too long?


As for any stored path, they can remain the same, as in DirEntry. 
The length of the path is what determines if it needs to use the 
special syntax or not. The user won't see any difference at all. 
From what I saw C# supports long names after a certain .net 
version, you might be able to see how they implemented it there. 
Parts of it are open source iirc.


Anyways there's only so many issues one person can chase to hell 
for someone as stubborn as ./.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Vladimir Panteleev via Digitalmars-d


On Sunday, 16 September 2018 at 01:19:46 UTC, tide wrote:
I guess that's why Bugzilla is a complete disaster. No one, at 
all, is maintaining it. As there are only 2 people that can 
really maintain it, and I don't see either of them commenting 
on bugs to provide direction, at least very often.


Well, I think that's looking at the situation from the wrong 
angle.


Most of D's code was written by volunteer contributors, and 
usually the code's author ends up maintaining that code, at least 
for a while. So, when you find a bug in some part of D and can't 
fix it yourself, looking at who wrote or last maintained the 
relevant code and pinging them would be the first step.


There are some things we can improve, like upgrading the platform 
or improving the categorization so people can receive 
notifications when someone files a bug in a Phobos module they 
care about. I've been slowly working on that front 
(https://github.com/CyberShadow/bugzilla-meta), but it doesn't 
change the underlying facts that bugs are most likely to be fixed 
by people who work on the code, not Andrei or Walter or any one 
person "in charge" of Bugzilla.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Jonathan M Davis via Digitalmars-d

On Saturday, September 15, 2018 7:19:46 PM MDT tide via Digitalmars-d wrote:
> On Sunday, 16 September 2018 at 00:53:45 UTC, Jonathan M Davis
>
> wrote:
> > On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir
> >
> > Panteleev via Digitalmars-d wrote:
> >> On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis
> >>
> >> wrote:
> >> > As for figuring out who is "officially" part of the dlang
> >> > org (or at least has the rights to merge PRs from at least
> >> > one dlang repo), you can look here
> >> >
> >> > https://github.com/orgs/dlang/people
> >> >
> >> > though it's possible to hide your membership, so while I see
> >> > 59 members when I look at it while signed in, if I look at
> >> > it while not signed in, I see only 40.
> >>
> >> I think it's worth clarifying: Only Walter and Andrei have the
> >> final say on things. Everyone else is a contributor (with a
> >> small few financially rewarded for their work). So, the above
> >> list of people isn't a good metric for much other than knowing
> >> who can merge your PR.
> >
> > Yeah, the list is mostly of folks who have contributed
> > significantly enough to have been given merge permissions at
> > some point. Some of us may have some influence based on how
> > long we've been with the project and the level of knowledge and
> > expertise that we've shown in the process, but as far as
> > deciding the direction of the project or anything like that,
> > it's all Walter and Andrei. The primary influence that any of
> > us have is simply by the work we contribute via PRs and the
> > feedback we give when reviewing PRs. It's not like there's a
> > hierarchy of authority or anything like that. For the most
> > part, the only authority that Walter and Andrei have given
> > others is the authority to merge PRs.
>
> I guess that's why Bugzilla is a complete disaster. No one, at
> all, is maintaining it. As there are only 2 people that can
> really maintain it, and I don't see either of them commenting on
> bugs to provide direction, at least very often.

Pretty much everything done around here is done by volunteers. There are
contributors who sometimes go through bugzilla to update or close bugs, and
there are contributors who go looking through bugzilla for bugs to fix, but
there is no one whose job it is to manage the list of bugs. I don't know of
any open source project that works that way. Usually, at most, you end up
with bugs being automatically assigned to people based on what they're for.

- Jonathan M Davis

Re: phobo's std.file is completely broke!

On Sunday, 16 September 2018 at 00:53:45 UTC, Jonathan M Davis 
wrote:
On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir 
Panteleev via Digitalmars-d wrote:

On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis

wrote:
> As for figuring out who is "officially" part of the dlang 
> org (or at least has the rights to merge PRs from at least 
> one dlang repo), you can look here

>
> https://github.com/orgs/dlang/people
>
> though it's possible to hide your membership, so while I see 
> 59 members when I look at it while signed in, if I look at 
> it while not signed in, I see only 40.

I think it's worth clarifying: Only Walter and Andrei have the 
final say on things. Everyone else is a contributor (with a 
small few financially rewarded for their work). So, the above 
list of people isn't a good metric for much other than knowing 
who can merge your PR.

Yeah, the list is mostly of folks who have contributed 
significantly enough to have been given merge permissions at 
some point. Some of us may have some influence based on how 
long we've been with the project and the level of knowledge and 
expertise that we've shown in the process, but as far as 
deciding the direction of the project or anything like that, 
it's all Walter and Andrei. The primary influence that any of 
us have is simply by the work we contribute via PRs and the 
feedback we give when reviewing PRs. It's not like there's a 
hierarchy of authority or anything like that. For the most 
part, the only authority that Walter and Andrei have given 
others is the authority to merge PRs.

- Jonathan M Davis

I guess that's why Bugzilla is a complete disaster. No one, at 
all, is maintaining it. As there are only 2 people that can 
really maintain it, and I don't see either of them commenting on 
bugs to provide direction, at least very often.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Jonathan M Davis via Digitalmars-d

On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir Panteleev via 
Digitalmars-d wrote:
> On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis
>
> wrote:
> > As for figuring out who is "officially" part of the dlang org
> > (or at least has the rights to merge PRs from at least one
> > dlang repo), you can look here
> >
> > https://github.com/orgs/dlang/people
> >
> > though it's possible to hide your membership, so while I see 59
> > members when I look at it while signed in, if I look at it
> > while not signed in, I see only 40.
>
> I think it's worth clarifying: Only Walter and Andrei have the
> final say on things. Everyone else is a contributor (with a small
> few financially rewarded for their work). So, the above list of
> people isn't a good metric for much other than knowing who can
> merge your PR.

Yeah, the list is mostly of folks who have contributed significantly enough
to have been given merge permissions at some point. Some of us may have some
influence based on how long we've been with the project and the level of
knowledge and expertise that we've shown in the process, but as far as
deciding the direction of the project or anything like that, it's all Walter
and Andrei. The primary influence that any of us have is simply by the work
we contribute via PRs and the feedback we give when reviewing PRs. It's not
like there's a hierarchy of authority or anything like that. For the most
part, the only authority that Walter and Andrei have given others is the
authority to merge PRs.

- Jonathan M Davis

Re: phobo's std.file is completely broke!

2018-09-15 Thread Vladimir Panteleev via Digitalmars-d

On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis 
wrote:
As for figuring out who is "officially" part of the dlang org 
(or at least has the rights to merge PRs from at least one 
dlang repo), you can look here


https://github.com/orgs/dlang/people

though it's possible to hide your membership, so while I see 59 
members when I look at it while signed in, if I look at it 
while not signed in, I see only 40.


I think it's worth clarifying: Only Walter and Andrei have the 
final say on things. Everyone else is a contributor (with a small 
few financially rewarded for their work). So, the above list of 
people isn't a good metric for much other than knowing who can 
merge your PR.

Re: phobo's std.file is completely broke!

On Saturday, 15 September 2018 at 18:21:43 UTC, Josphe Brigmo 
wrote:
On Saturday, 15 September 2018 at 13:37:29 UTC, Vladimir 
Panteleev wrote:
Can you list some programming languages that achieve this task 
in a way you approve of?


Plenty, pick just about any one. C#, Haskell, javascript, lua, 
python, perl, C++(yes, c++, we are not talking about language 
features but usability). The simple fact is that C++ can be 
used to do anything almost 100% correct while D can fail. D is 
only a better language, not a better compiler(except it's 
speed).


See you are just talking out of your ass right now. I just tried 
C++, it doesn't work. You can't use std::fstream with a path that 
is larger than 260. I also moved an executable to that large 
path. And guess what I couldn't even get Windows to run it. Not 
through powershell nor explorer. I can't even run applications 
like "ls" with powershell. Let alone "cd" into the folder. oddly 
enough the only thing that came close was git bash, which gave me 
an error message. While powerhshell just said couldn't find path.


```
Error: Current working directory has a path longer than allowed 
for a

Win32 working directory.
Can't start native Windows application from here.
```

I don't know how you can complain about D when Windows is so 
fundamentally broken for large files path, that even it's set of 
tools don't support it.


Another one, Python: 
https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation


It does not support long paths, you have to use \\?\ or enable it 
with the registry (only on newer Windows 10 versions).


If you are going to make claims -- Do Your Research.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Jonathan M Davis via Digitalmars-d

On Saturday, September 15, 2018 5:47:08 PM MDT tide via Digitalmars-d wrote:
> On Saturday, 15 September 2018 at 18:33:52 UTC, bachmeier wrote:
> > On Saturday, 15 September 2018 at 13:54:45 UTC, tide wrote:
> >> On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:
> >>> On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo
> >>>
> >>> wrote:
>  For very long file names it is broke and every command
>  fails. These paths are not all that long but over 256 limit.
>  (For windows)
> >>>
> >>> Please file a bug report with reproducible examples if you
> >>> believe it's a bug.
> >>
> >> I feel people need to stop saying this.
> >
> > That's how things are done in this project (and most projects).
> > Anyone not liking that is out of luck. I don't use any open
> > source projects that use vague posts to a forum to report bugs.
>
> That's how they are, but if you are going to say silly things. At
> least take the initiative on yourself to go see if the bug
> already exists. Odds are it has probably already been reported,
> someone making a forums post about it is just able the next step
> after the bug report has been sitting in the queue for 4+ years.
>
> The problem isn't reporting the bug, it's that for D, no one is
> managing bugzilla. It seems to be the conclusion was reached that
> this is not a bug and won't be fixed. That could have been done a
> long time ago and closed the bug. Or at least I think Jonathan is
> part of the Dlang organization. Not sure, hard to tell who is and
> isn't on these boards.

The issue was reported in bugzilla quite some time ago.

https://issues.dlang.org/show_bug.cgi?id=8967

However, while Walter's response on it basically indicates that we should
just close it as "won't fix," we never actually did - which is something
that probably happens too often. There was further discussion in there that
never went anywhere. What we should probably do is ensure that the std.file
documentation is clear about the issue and count that as the fix. Either
way, I think that it's pretty clear that the documentation needs to be
improved.

As for figuring out who is "officially" part of the dlang org (or at least
has the rights to merge PRs from at least one dlang repo), you can look here

https://github.com/orgs/dlang/people

though it's possible to hide your membership, so while I see 59 members when
I look at it while signed in, if I look at it while not signed in, I see
only 40.

- Jonathan M Davis

Re: phobo's std.file is completely broke!

2018-09-15 Thread Vladimir Panteleev via Digitalmars-d

On Saturday, 15 September 2018 at 23:50:43 UTC, Josphe Brigmo 
wrote:

[...]


D is generally described as a system programming language. There 
is value in favoring a simple and obvious implementation ("do 
what I say") over going out of one's way to make usage simpler 
("do what I mean"). The tradeoff is performance and complexity. 
Performance is generally an important factor for users of system 
programming languages, and complexity is a source of unforeseen 
problems in non-trivial use cases.


Consider, for example, how integers are treated in D and Python. 
D's integers are fixed-length and roll over on overflow. Python 
integers are bigints, which makes them slower, but can handle 
numbers of any size.


From your posts, it sounds like you're looking for a programming 
language closer to Python than to D.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Vladimir Panteleev via Digitalmars-d

On Saturday, 15 September 2018 at 18:21:43 UTC, Josphe Brigmo 
wrote:
Can you list some programming languages that achieve this task 
in a way you approve of?


Plenty, pick just about any one. C#, Haskell, javascript, lua, 
python, perl, C++(yes, c++, we are not talking about language 
features but usability).


I had a quick look at the implementations of some of the 
languages you mentioned. They use the C API or use the Windows 
API without the UNC prefix.


You are lying.

Re: phobo's std.file is completely broke!

2018-09-15 Thread Josphe Brigmo via Digitalmars-d

On Saturday, 15 September 2018 at 23:06:57 UTC, Jonathan M Davis 
wrote:
On Saturday, September 15, 2018 6:54:50 AM MDT Josphe Brigmo 
via Digitalmars-d wrote:

On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe

wrote:
> On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo
>
> wrote:
>> Phobos *NEEDS* to be modified to work with these newer OS's.
>
> You need to look at the source code before posting. The code 
> for remove is literally

>
> DeleteFileW(name);
>
> it is a one-line wrapper, and obviously uses the unicode 
> version.

>
> https://github.com/dlang/phobos/blob/master/std/file.d#L1047

It doesn't matter, the fact is that something in phobos is 
broke. Do you really expect me to do all the work? The fact 
that using executeShell or "\\?\" solves 90% of the 
problems(maybe all of them) proves that phobos is not up to 
par.

Using std.file should be on par with using the Windows API from 
C or C++. It doesn't try to fix the arguably broken behavior of 
the Windows API with regards to long paths but requires that 
the programmer deal with them just like they would in C/C++. 
The main differences are that the std.file functions in 
question use D strings rather than C strings, and they 
translate them to the UTF-16 C strings for you rather than 
requiring you to do it. But they don't do anything like add 
"\\?\" for you any more than the Windows API itself does that.

If you consider that to be broken, then sorry. For better or 
worse, it was decided that it was better to let the programmer 
deal with those intricacies rather than trying to tweak the 
input to make it work based on the idea that that could have 
undesirable consequences in some circumstances. On some level, 
that does suck, but the Windows API does not make it easy to 
make this work like it would on a *nix system without 
introducing subtle bugs.

Do you not realize how moronic that is though?  You are expecting 
each user to know the correct behavior and to compensate for it. 
With that approach all you are doing is creating a whole mess of 
problems.

See, there is only one phobos but many users of phobos. You are 
expecting every single user to deal with it instead of dealing 
with it in one place.

Your reason is because "It's a problem with windows".

Both are "We'll just kick the can down the road cause the other 
guy kicked it to us".

Now, this is great if you want repeat customers and don't care 
about their sanity but terrible to make progress.

If you find that the std.file functions don't work whereas 
using the same input to the Windows API functions in C/C++ 
would have, then there's a bug in the D code, and it needs to 
be fixed, but if it acts the same as the C/C++ code, then it's 
working as intended.

And that is precisely the problem. For some reason you don't get 
that because it is broke in windows, and you following windows, 
you are just perpetuating the "brokeness".  This is why D sucks 
the way it does, because of these types of mentalities of "It's 
not our problem".

You basically expect the customers, before eating to wash the 
dishes, cook the meal, set the table, etc. All you do is provide 
the food, and not all that great food... and when they are done 
you expect them to wash the dishes against, clean up their mess, 
and pay the bill.

I'm sure you'll never realize how wrong your mentality is, but at 
least you could do everyone a favor and think about what you are 
actually saying. Your logic might have a valid reason, but it is 
not producing the results that make the world a better place.

D won't ever get any traction with the wrong mentalities and I 
don't even know how it has made it this far.

Trust me, if you expect the user to know everything and also 
solve all the problems over and over and over and over, you will 
have very few users.

But, keep on doing what your doing, it's worked so well, we will 
see how far it gets D.

If D is as perfect as you think it is, why isn't everyone jumping 
on board? I know, you have answers for that one too!

Re: phobo's std.file is completely broke!