Re: This is why I don't use D.
20.09.2018 07:16, Neia Neutuladh пишет: On Thursday, 20 September 2018 at 02:51:52 UTC, Neia Neutuladh wrote: On Monday, 10 September 2018 at 01:27:20 UTC, Neia Neutuladh wrote: Not on dlang.org anywhere, but I built a crude version of this. Results are available at http://ikeran.org/report/. A quick status update: And source code is available at https://git.ikeran.org/dhasenan/dubautotester Please don't judge me. Autotester should show build logs because for example `nanogui` package reported as failed although it builds on my machines successfully.
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir Panteleev wrote: One point of view is that the expected behavior is that the functions succeed. Another point of view is that Phobos should not allow programs to create files and directories with invalid paths. Consider, e.g. that a user writes a program that creates a large tree of deeply nested filesystem objects. When they are done and wish to delete them, their file manager fails and displays an error. The user's conclusion? D sucks because it corrupts the filesystem and creates objects they can't operate with. You don't even need to use crazy third-party software. Try this program: mkdir(`\\?\C:\ a \`); write(`\\?\C:\ a \a.txt`, "Hello"); Then, try doing the following: - Double-click the created text file. - Try deleting the directory from Explorer (by sending it to the recycle bin). - Try permanently deleting it (Shift+Delete). - Try renaming it. All of these fail for me. Deleting the directory doesn't even show an error - nothing at all happens. When the OS itself fails to properly deal with such files, I don't think D has any business in *facilitating* their creation by default. *Windows Explorer* prevents you from creating a folder or file whose name STARTS with spaces. It trims them automatically, whether you want it or not. So it's NOT a surprise that *Windows Explorer* (!) has problems if you use it on such files which were created manually. But obviously, *Windows* OS doesn't prevent you to create them through scripts and applications...
Re: Mobile is the new PC and AArch64 is the new x64
On Wednesday, 19 September 2018 at 18:14:47 UTC, Dave Jones wrote: Only if the new product meets all the use cases of the old product. Again this is what you dont understand. Why did the iPhone, and after that the smartphone industry as a whole, completely crush the classic cell phones when they have such a poor battery life? Smartphones don't have everything previous phones had. The pros simply outweighed the cons. You don't need a new product to do everything the previous product did for it to be successful enough to replace it. I used to be all about being able to replace a phone's battery manually. With basically any newer smartphone now, you can't anymore. Just because phones aren't doing everything PC's are doing doesn't mean they can't overtake their market. All they need is to have sufficient advantages over them, and even if you don't think this is the case right now, the average user could very well disagree. Just like with removable batteries (which wasn't even a really technical thing).
Re: Mobile is the new PC and AArch64 is the new x64
On Tuesday, 11 September 2018 at 06:42:26 UTC, Chris wrote:
On Monday, 10 September 2018 at 19:28:01 UTC, aberba wrote:
On Monday, 10 September 2018 at 16:09:41 UTC, rjframe wrote:
That's exactly whats happening in Africa. The continent is
leapfrogging from nothing to a smart phone thanks to China.
Many don'[t know how to even use a PC. Especially the young
and upcoming generation.
The smart phone market is really significant.
That's why I'm trying to draw attention to first class ARM
support by the D Foundation (not just some voluntary efforts,
much as they are appreciated). ARM in dmd wouldn't be a bad
idea perhaps, as Manu suggested. It's become more than obvious
over the last couple of years that mobile devices have become
very important and that people often use them instead of PCs or
laptops. Fewer and fewer developers can "escape" development
for mobile ("we want an app too"), and if a language doesn't
offer first class support for mobile, then devs won't bother
with it. A lot of other (new) languages are directing their
efforts towards mobile for a reason.
Embedded software is a kind of software that is built into
hardware systems and is designed to perform specific functions.
Modern devices which are stuffed with different functions are
full of embedded software solutions - that is why they are so
helpful. See https://www.ssla.co.uk/
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 02:48:06 UTC, Nick Sabalausky (Abscissa) wrote: On 09/19/2018 02:33 AM, Jonathan Marler wrote: What drives me mad is when you have library writers who try to "protect" you from the underlying system by translating everything you do into what they "think" you're trying to do. What drives me mad is when allegedly cross-platform tools deliberately propagate non-cross-platform quirks that could easily be abstracted away and pretend that's somehow "helping" me instead of making a complete wreck of the whole point of cross-platform. Bonus points if they're doing it mainly to help with my C++-standard premature optimizations. If I actually want to deal with platform-specific quirks, then I'll use the platform's API directly. (And then I'll beat myself with a brick, just for fun.) +1 A cross-platform library has to be designed to operate in the same way on each supported platform, even if this means that it's harder to implement on some platform, or that some platforms will need more complicated implementations. That's the whole point of this "HAL" approach.
Re: dub auto-tester
On Thursday, 20 September 2018 at 04:41:21 UTC, Joakim wrote: Nice, what will it take to get this integrated with the official dub website? I need to: * add JSON output to the auto-tester * get the dub registry to scrape the data (or, optionally, push the data to the registry, but that opens up trust issues that I'd rather not get into) * enable SSL on the report site (and probably get a new domain for it) * alter the dub UI to include the data That's all pretty straightforward. BTW, the gitea self-hosted github-workalike you're using looks nice, too bad it's written in Go. ;) Gitea has the benefit of existing, which is surprisingly important.
Re: This is why I don't use D.
On Thursday, 20 September 2018 at 04:16:41 UTC, Neia Neutuladh wrote: And source code is available at https://git.ikeran.org/dhasenan/dubautotester Please don't judge me. Nice work.
dub auto-tester
On Thursday, 20 September 2018 at 04:16:41 UTC, Neia Neutuladh wrote: On Thursday, 20 September 2018 at 02:51:52 UTC, Neia Neutuladh wrote: On Monday, 10 September 2018 at 01:27:20 UTC, Neia Neutuladh wrote: Not on dlang.org anywhere, but I built a crude version of this. Results are available at http://ikeran.org/report/. A quick status update: And source code is available at https://git.ikeran.org/dhasenan/dubautotester Please don't judge me. Nice, what will it take to get this integrated with the official dub website? BTW, the gitea self-hosted github-workalike you're using looks nice, too bad it's written in Go. ;)
Re: This is why I don't use D.
On Thursday, 20 September 2018 at 02:51:52 UTC, Neia Neutuladh wrote: On Monday, 10 September 2018 at 01:27:20 UTC, Neia Neutuladh wrote: Not on dlang.org anywhere, but I built a crude version of this. Results are available at http://ikeran.org/report/. A quick status update: And source code is available at https://git.ikeran.org/dhasenan/dubautotester Please don't judge me.
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky (Abscissa) wrote: (Not on a Win box at the moment.) I added the output of my test program to the gist: https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986#file-output-txt assert( dir.toAbsolutePath.length > MAX_LENGTH-12 ); Actually it's crazier than that. The concatenation of the current directory plus the relative path must be < MAX_PATH (approx.). Meaning, if you are 50 directories deep, a relative path starting with 50 `..\` still won't allow you to access C:\file.txt.
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir Panteleev wrote: When the OS itself fails to properly deal with such files, I don't think D has any business in *facilitating* their creation by default. Dear lord Windows is terrible. Can we just deprecate it?
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky (Abscissa) wrote: I'm not sure I'm quite following you. Is this what you mean?: string dir = ...; // Such that... assert( dir.isRelativePath ); assert( dir.length < MAX_LENGTH-12 ); assert( dir.toAbsolutePath.length > MAX_LENGTH-12 ); // ??? This *currently* goes BOOM on Windows // ??? installations with MAX_LENGTH restriction active? rmdir(path); (Not on a Win box at the moment.) Correct.
Re: phobo's std.file is completely broke!
On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote: rmdir(path); Obviously meant "rmdir(dir);" here. Editing mishap.
Re: phobo's std.file is completely broke!
On Thursday, 20 September 2018 at 03:25:05 UTC, Nick Sabalausky (Abscissa) wrote: On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote: rmdir(path); Obviously meant "rmdir(dir);" here. Editing mishap. and MAX_PATH instead of MAX_LENGTH, and absolutePath instead of toAbsolutePath, and !isAbsolute instead of isRelativePath, but I understood what you meant :)
Re: phobo's std.file is completely broke!
On 09/19/2018 07:04 AM, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky (Abscissa) wrote: 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5]. This is not a good criteria: relative paths whose pointing to objects whose absolute path exceeds MAX_PATH will fail, too. So, it looks like Phobos would need to expand relative paths unconditionally. I'm not sure I'm quite following you. Is this what you mean?: string dir = ...; // Such that... assert( dir.isRelativePath ); assert( dir.length < MAX_LENGTH-12 ); assert( dir.toAbsolutePath.length > MAX_LENGTH-12 ); // ??? This *currently* goes BOOM on Windows // ??? installations with MAX_LENGTH restriction active? rmdir(path); (Not on a Win box at the moment.)
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir Panteleev wrote: One point of view is that the expected behavior is that the functions succeed. Another point of view is that Phobos should not allow programs to create files and directories with invalid paths. Consider, e.g. that a user writes a program that creates a large tree of deeply nested filesystem objects. When they are done and wish to delete them, their file manager fails and displays an error. The user's conclusion? D sucks because it corrupts the filesystem and creates objects they can't operate with. You don't even need to use crazy third-party software. Try this program: mkdir(`\\?\C:\ a \`); write(`\\?\C:\ a \a.txt`, "Hello"); Then, try doing the following: - Double-click the created text file. - Try deleting the directory from Explorer (by sending it to the recycle bin). - Try permanently deleting it (Shift+Delete). - Try renaming it. All of these fail for me. Deleting the directory doesn't even show an error - nothing at all happens. When the OS itself fails to properly deal with such files, I don't think D has any business in *facilitating* their creation by default.
Re: phobo's std.file is completely broke!
On 09/19/2018 01:49 PM, Neia Neutuladh wrote: On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir Panteleev wrote: BTW, something follows from the above: write(`C:\` ~ (short path) ~ `con`) will fail but: write(`C:\` ~ (long path) ~ `con`) will succeed. This is just one issue I've noticed... there's probably more lurking. Also, according to the internet: write(chainPath(shortDirectory, "A "), "Win32 API strips trailing space"); readText(chainPath(shortDirectory, "A")); // Win32 API strips trailing space But: write(chainPath(longDirectory, "A "), "Win32 API strips trailing space"); readText(chainPath(longDirectory, "A")); // File not found write(chainPath(shortDirectory, "A.")); // fails write(chainPath(longDirectory, "A.")); // succeeds Ok, excellent, now we're getting somewhere! :) Based on these actual concrete examples, yes, I can now grant that converting paths that are past the limit would indeed cause subtly different behaviour. So clearly this needs to be an all-or-nothing deal... This is why we should use the exact same behavior in all cases. Always use `\\?\` or never use it. Yup. Since Windows path handling is weird by default, I'd prefer always using `\\?\`. It's overhead, but not a huge amount of additional overhead compared to filesystem manipulation. My thoughts, exactly. I'll also point out that this applies to user-code, too. Ie, with Phobos as it currently stands, user code has three options: 1. Never use `\\?\` and any time they compile their code on Windows, they must always be acutely aware of, and plan for, all of the fun little quirks of the Win33 file APIs. 2. Be *aware* of `\\?\` and always use it any time they compile their code on Windows (and always remember use longPath() or some such for all file operations when working on a cross-platform codebase). 3. Sometimes use `\\?\` and sometimes don't, and then deal with the fun little inconsistencies that creates. I think ALL those options look really bad, but #2 seems the least terrible. But gee, wouldn't it be nice if we could just relieve that stupid burden on everybody of selecting and following #2? In the rare case where "doing the safe/correct thing by default" isn't good enough, there'd be nothing stopping anyone from calling the platform API directly if they really, really need to. Opt-out is for safe, reliable, consistent things. Opt-in is for difficult, tricky, dangerous things. Not the other way around.
Re: Small @nogc experience report
On 19/09/18 22:53, Walter Bright wrote: On 9/19/2018 10:13 AM, Shachar Shemesh wrote: assert(condition, string); // string is useless without actual info about what went wrong. assert(condition, format(string, arg, arg)); // No good - format is not @nogc Another method: debug assert(condition, format(string, arg, arg)); else assert(condition, string); because @nogc is ignored in debug conditionals, just like purity is ignored in debug conditionals. That doesn't cut it on so many levels... First of all, no four lines solution that requires copy/paste (or worse, retyping) as a standard will actually get employed by programmers. The disincentive is too high. If we overcome this problem, we're still left with the fact that in all of the important runs the data I need in order to debug an assert violation is not going to be there when I need it. Let me put it this way: the Weka code base has over 13,000 asserts. Every single time an assert triggered on me that either did not have a message, or had only a message but not data, I needed more data than it had. It is, unfortunately, not true to say that this only ever happened in debug builds. In fact, we do release builds with asserts as part of our CI, so a relatively rare bug has a very high chance of throwing an assert in non-debug runs. Writing asserts should be easy, and making them useful in case they trigger should also be easy. What's more, soft mandating writing asserts with assert messages and parameters all but eliminates the common anti-pattern used by many novices of asserting that the compiler does what it's supposed to, e.g: if( a>13 ) return; assert(a<13); Shachar
Re: This is why I don't use D.
On Monday, 10 September 2018 at 01:27:20 UTC, Neia Neutuladh wrote: Not on dlang.org anywhere, but I built a crude version of this. Results are available at http://ikeran.org/report/. A quick status update: Per-package reports and build badges are now a thing. http://ikeran.org/report/package/accessors.html Dunno if anybody would want a build badge, but it's there. And a bit big. Oh well. Backfill has gotten all .0 releases from 2.069.0 forward. I'm cutting off the backfill for point releases at 2.080.1, about two days from now. 19 compilers rather than 43 makes this project a bit more sustainable. (It was previously taking six hours to build a single compiler release, but with some changes, it started taking 16.) After the backfill finishes, I'll start grabbing new dub packages, which will be trivial. The next tasks are auto-detecting new DMD versions (pretty straightforward) and making machine-readable output. As an aside, the current output is about 50MB of HTML and SVG, served as static files. I thought of using arsd's cgi.d, but there really isn't a point for now.
Re: phobo's std.file is completely broke!
On 09/19/2018 02:33 AM, Jonathan Marler wrote: What drives me mad is when you have library writers who try to "protect" you from the underlying system by translating everything you do into what they "think" you're trying to do. What drives me mad is when allegedly cross-platform tools deliberately propagate non-cross-platform quirks that could easily be abstracted away and pretend that's somehow "helping" me instead of making a complete wreck of the whole point of cross-platform. Bonus points if they're doing it mainly to help with my C++-standard premature optimizations. If I actually want to deal with platform-specific quirks, then I'll use the platform's API directly. (And then I'll beat myself with a brick, just for fun.)
Re: Truly @nogc Exceptions?
On 9/19/18 7:53 PM, Seb wrote: On Wednesday, 19 September 2018 at 21:28:56 UTC, Steven Schveighoffer wrote: On 9/19/18 5:16 PM, Steven Schveighoffer wrote: One further thing: I didn't make the sink version of message @nogc, but in actuality, it could be. We recently introduced support for output ranges in the formatting of Phobos: https://dlang.org/changelog/2.079.0.html#toString Output ranges have the advantage that they could be @nogc and because of the templatization also @safe. I don't think that will work here, as Throwable is a class. All I can think of is that you would have 2 versions, a @nogc one that takes a @nogc delegate, and one that is not. Of course, your exception type could define something completely separate, and you can deal with it in your own project as needed. If there was a way to say "this is @nogc if you give it a @nogc delegate, and not if you don't", that would be useful. The compiler could verify it at compile-time. -Steve
Re: Truly @nogc Exceptions?
On Wednesday, 19 September 2018 at 21:28:56 UTC, Steven Schveighoffer wrote: On 9/19/18 5:16 PM, Steven Schveighoffer wrote: One further thing: I didn't make the sink version of message @nogc, but in actuality, it could be. We recently introduced support for output ranges in the formatting of Phobos: https://dlang.org/changelog/2.079.0.html#toString Output ranges have the advantage that they could be @nogc and because of the templatization also @safe.
Re: extern(C++, ns) is wrong
On Wed, 19 Sep 2018 at 13:15, Walter Bright via Digitalmars-d wrote: > > On 9/18/2018 5:22 PM, Manu wrote: > > Thank you Walter for coming to the party! > > I suppose I should explain this. > > 1. The PR uses a different syntax, i.e. strings instead of identifiers. This > implies it is not creating a scope, and doesn't interfere with the scoping of > the existing syntax. The design of the feature is simple and easy to document > and understand, and doesn't break anything. Wait up, just to be clear... is this a statement about the PR using strings different than your perception of the conversation surrounding this? This PR implements *exactly* what I was calling for literally the same day that the first patch introducing support was merged however many years ago, and every few months thereafter... I'm just curious to know if this PR was somehow surprising to you. Because if so, it demonstrates a total and colossal failure to communicate the preferred design of this feature ;) It was always desired to be a string from day -1, and the bugzilla issue presented it that way too.
Re: Tuple DIP
On 19.09.2018 23:14, 12345swordy wrote: On Tuesday, 3 July 2018 at 16:11:05 UTC, 12345swordy wrote: On Thursday, 28 June 2018 at 13:24:11 UTC, Timon Gehr wrote: On 26.06.2018 11:55, Francesco Mecca wrote: On Friday, 12 January 2018 at 22:44:48 UTC, Timon Gehr wrote: As promised [1], I have started setting up a DIP to improve tuple ergonomics in D: [...] What is the status of the DIP? Is it ready to be proposed and dicussed? I still need to incorporate all the feedback from this thread. Also, I have started an implementation, and ideally I'd like to have it finished by the time the DIP is discussed. Unfortunately I am rather busy with work at the moment. Is there any way we can help on this? *Bump* I want this. So do I, but I need to get a quiet weekend or so to finish this. I am very tempted to start my own dip on this and finish it. Here's the current state of my implementation in DMD: https://github.com/dlang/dmd/compare/master...tgehr:tuple-syntax It has no tests yet, but basically, with those changes, you can write tuple literals `(1, 2.0, "3")`, you can unpack tuples using `auto (a, b) = t;` or `(int a, string b) = t;`, and tuples can be expanded using alias this on function calls, so you can now write things like `zip([1,2,3],[4,5,6]).map!((a,b)=>a+b)`. The implementation is still missing built-in syntax for tuple types, tuple assignments, and tuple unpacking within function argument lists and foreach loops.
Re: Truly @nogc Exceptions?
On 9/19/18 5:16 PM, Steven Schveighoffer wrote: One further thing: I didn't make the sink version of message @nogc, but in actuality, it could be. Notice how it allocates using the stack. Even if we needed some indeterminate amount of memory, it would be simple to use C malloc/free, or alloca. But traditionally, we don't put any attributes on these base functions. Would it make sense in this case? Aand, no we can't. Because the sink could actually allocate. -Steve
Truly @nogc Exceptions?
Given dip1008, we now can throw exceptions inside @nogc code! This is
really cool, and helps make code that uses exceptions or errors @nogc.
Except...
The mechanism to report what actually went wrong for an exception is a
string passed to the exception during *construction*. Given that you
likely want to make such an exception inside a @nogc function, you are
limited to passing a compile-time-generated string (either a literal or
one generated via CTFE).
To demonstrate what I mean, let me give you an example member function
inside a type containing 2 fields, x and y:
void foo(int[] arr)
{
auto x = arr[x .. y];
}
There are 2 ways this can throw a range error:
a) x > y
b) y > arr.length
But which is it? And what are x and y, or even the array length?
The error message we get is basic (module name and line number aren't
important here):
core.exception.RangeError@testerror.d(6): Range violation
Not good enough -- we have all the information present to give a more
detailed message. Why not:
Attempted slice with wrong ordered parameters, 5 .. 4
or
Slice parameter 6 is greater than length 5
All that information is available, yet we don't see anything like that.
Let's look at the base of all exception and error types to see why we
don't have such a thing. The part which prints this message is the
member function toString inside Throwable, repeated here for your
reading pleasure [1]:
void toString(scope void delegate(in char[]) sink) const
{
import core.internal.string : unsignedToTempString;
char[20] tmpBuff = void;
sink(typeid(this).name);
sink("@"); sink(file);
sink("("); sink(unsignedToTempString(line, tmpBuff, 10));
sink(")");
if (msg.length)
{
sink(": "); sink(msg);
}
if (info)
{
try
{
sink("\n");
foreach (t; info)
{
sink("\n"); sink(t);
}
}
catch (Throwable)
{
// ignore more errors
}
}
}
(Side Note: there is an overload for toString which takes no delegate
and returns a string. But since this overload is present, doing e.g.
writeln(myEx) will use it)
Note how this *doesn't* allocate anything.
But hang on, what about the part that actually prints the message:
sink(typeid(this).name);
sink("@"); sink(file);
sink("("); sink(unsignedToTempString(line, tmpBuff, 10));
sink(")");
if (msg.length)
{
sink(": "); sink(msg);
}
Hm... Note how the file name, and the line number are all *members* of
the exception, and there was no need to allocate a special string to
contain the message we saw. So it *is* possible to have a custom message
without allocation. It's just that the only interface for details is via
the `msg` string member field -- which is only set on construction.
We can do better.
I noticed that there is a @__future member function inside Throwable
called message. This function returns the message that the Throwable is
supposed to display (defaulting to return msg). I believe this was
inserted at Sociomantic's request, because they need to be able to have
a custom message rendered at *print* time, not *construction* time [2].
This makes sense -- why do we need to allocate some string that will
never be printed (in the case where an exception is caught and handled)?
This helps alleviate the problem a bit, as we could construct our
message at print-time when the @nogc requirement is no longer present.
But we can do even better.
What if we added ALSO a function:
void message(scope void delegate(in char[]) sink)
In essence, this does *exactly* what the const(char)[] returning form of
message does, but it doesn't require any allocation, nor storage of the
data to print inside the exception. We can print numbers (and other
things) and combine them together with strings just like the toString
function does.
We can then replace the code for printing the message inside toString
with this:
bool printedColon = false;
void subSink(in char[] data)
{
if(!printedColon && data.length > 0)
{
sink(": ");
printedColon = true;
}
sink(data);
}
message(&subSink);
In this case, we then have a MUCH better mechanism to implement our
desired output from the slice error:
class RangeSliceError : Throwable
{
size_t lower;
size_t upper;
size_t len;
...
override void message(scope void delegate(in char[]) sink)
{
import core.internal.string : unsignedToTempString;
char[20] tmpBuff = void;
if (lower > upper)
{
sink("Attempted slice with wrong ordered parameters ");
sink(unsignedToTempString(lower, tmpBuff, 10));
Re: Tuple DIP
On Tuesday, 3 July 2018 at 16:11:05 UTC, 12345swordy wrote: On Thursday, 28 June 2018 at 13:24:11 UTC, Timon Gehr wrote: On 26.06.2018 11:55, Francesco Mecca wrote: On Friday, 12 January 2018 at 22:44:48 UTC, Timon Gehr wrote: As promised [1], I have started setting up a DIP to improve tuple ergonomics in D: [...] What is the status of the DIP? Is it ready to be proposed and dicussed? I still need to incorporate all the feedback from this thread. Also, I have started an implementation, and ideally I'd like to have it finished by the time the DIP is discussed. Unfortunately I am rather busy with work at the moment. Is there any way we can help on this? *Bump* I want this. I am very tempted to start my own dip on this and finish it.
Re: extern(C++, ns) is wrong
On 9/18/2018 5:22 PM, Manu wrote: Thank you Walter for coming to the party! I suppose I should explain this. 1. The PR uses a different syntax, i.e. strings instead of identifiers. This implies it is not creating a scope, and doesn't interfere with the scoping of the existing syntax. The design of the feature is simple and easy to document and understand, and doesn't break anything. 2. Manu and Atila are at the forefront of interfacing with C++. They are both doing massive amounts of work making it work. I'm willing to give a lot of deference to people who know what they are doing and have earned it through their major contributions and longtime membership in our community. 3. The PR for it looks to be a solid piece of work, and even if it turns out otherwise, it's pretty low risk and fixable. Thanks to Manu, Atila and look-at-me.
Re: Small @nogc experience report
On 9/19/2018 11:35 AM, Steven Schveighoffer wrote: I'm running into this coincidentally right now, when trying to debug a PR. I found I'm getting a range error deep inside a phobos function. But because Phobos is trying to be pure @nogc nothrow @safe, I can do almost nothing to display what is wrong. That's why attribute enforcement is weakened in debug conditionals.
Re: Small @nogc experience report
On 9/19/2018 10:13 AM, Shachar Shemesh wrote: assert(condition, string); // string is useless without actual info about what went wrong. assert(condition, format(string, arg, arg)); // No good - format is not @nogc Another method: debug assert(condition, format(string, arg, arg)); else assert(condition, string); because @nogc is ignored in debug conditionals, just like purity is ignored in debug conditionals.
Re: Small @nogc experience report
On 9/7/2018 10:35 AM, Eugene Wissner wrote: fill() uses enforce() which allocates and throws. The addition of -dip1008 stopped using the gc for throwing exceptions, but it's opt-in at the moment.
Re: extern(C++, ns) is wrong
On Wednesday, September 19, 2018 7:26:07 AM MDT Steven Schveighoffer via Digitalmars-d wrote: > On 9/18/18 9:49 PM, Jonathan M Davis wrote: > > On Tuesday, September 18, 2018 6:22:55 PM MDT Manu via Digitalmars-d wrote: > >> https://github.com/dlang/dmd/pull/8667 > >> > >> O_O > >> > >> Thank you Walter for coming to the party! > > > > Oh, wow. I sure wasn't expecting that. I thought that he'd made it > > pretty > > clear that a DIP was needed, and even then, it didn't seem likely that > > it > > would be accepted. This is awesome. I guess that he finally came around. > > I think a big part is that the implementation was done. I think there's > a big difference between "I don't really love this, but crap, I'll have > to implement it all" and "I don't really love this, but the > implementation isn't too intrusive, and all I have to do is click the > merge button, OK." I sure wish I had the skills to hack dmd, there are > so many ideas I'd like to see implemented in the language :) > > Anyways, great to see this merged! Having an implementation available hasn't always done the trick, but it's certainly easier to try to convince Walter accept something that's already been done than to convince him that he should implement something - which really shouldn't be a surprise. I expect that pretty much all of us are that way. - Jonathan M Davis
Re: Small @nogc experience report
On 19/09/18 21:35, Steven Schveighoffer wrote:
On 9/19/18 1:13 PM, Shachar Shemesh wrote:
There is a catch, though. Writing Mecca with @nogc required
re-implementing quite a bit of druntime. Mecca uses its own exception
allocations (mkEx, just saw it's not yet documented, it's under
mecca.lib.exception). The same module also has "enforceNGC". We also
have our own asserts. This is partially to support our internal
logging facility, that needs a static list of formats, but it also
solves a very important problem with D's @nogc:
void func() @nogc {
assert(condition, string); // string is useless without actual info
about what went wrong.
assert(condition, format(string, arg, arg)); // No good - format is
not @nogc
ASSERT!"format"(condition, arg, arg); // @nogc and convenient
}
So, yes, we do use @nogc, but it took a *lot* of work to do it.
I'm running into this coincidentally right now, when trying to debug a
PR. I found I'm getting a range error deep inside a phobos function. But
because Phobos is trying to be pure @nogc nothrow @safe, I can do almost
nothing to display what is wrong.
What I ended up doing is making an extern(C) hook that had the "right"
attributes, even though it's not @nogc (let's face it, you are about to
crash anyway).
But it got me thinking, what a useless interface to display errors we
have! Inside Throwable, there is the function toString(someDelegate
sink) which prints out the exception trace.
Near the front there is this:
if (msg.length)
{
sink(": "); sink(msg);
}
My, wouldn't it be nice to be able to override this! And forget about
the whole msg BS. When an exception trace is printed, there are almost
no restrictions as to what can be done. We should delay the generation
of the message until then as well! Not to mention that if we can output
things piecemeal through the sink, we don't even have to allocate at all.
I'm going to write up a more detailed post on this, but it's annoying to
throw exceptions without any information EXCEPT what can be converted
into a string at runtime at the time of exception. All that is missing
is this hook to generate the message.
-Steve
Then by all means, have a look at ASSERT inside mecca.lib.exception.
Re: Small @nogc experience report
On 9/19/18 1:13 PM, Shachar Shemesh wrote:
There is a catch, though. Writing Mecca with @nogc required
re-implementing quite a bit of druntime. Mecca uses its own exception
allocations (mkEx, just saw it's not yet documented, it's under
mecca.lib.exception). The same module also has "enforceNGC". We also
have our own asserts. This is partially to support our internal logging
facility, that needs a static list of formats, but it also solves a very
important problem with D's @nogc:
void func() @nogc {
assert(condition, string); // string is useless without actual info
about what went wrong.
assert(condition, format(string, arg, arg)); // No good - format is
not @nogc
ASSERT!"format"(condition, arg, arg); // @nogc and convenient
}
So, yes, we do use @nogc, but it took a *lot* of work to do it.
I'm running into this coincidentally right now, when trying to debug a
PR. I found I'm getting a range error deep inside a phobos function. But
because Phobos is trying to be pure @nogc nothrow @safe, I can do almost
nothing to display what is wrong.
What I ended up doing is making an extern(C) hook that had the "right"
attributes, even though it's not @nogc (let's face it, you are about to
crash anyway).
But it got me thinking, what a useless interface to display errors we
have! Inside Throwable, there is the function toString(someDelegate
sink) which prints out the exception trace.
Near the front there is this:
if (msg.length)
{
sink(": "); sink(msg);
}
My, wouldn't it be nice to be able to override this! And forget about
the whole msg BS. When an exception trace is printed, there are almost
no restrictions as to what can be done. We should delay the generation
of the message until then as well! Not to mention that if we can output
things piecemeal through the sink, we don't even have to allocate at all.
I'm going to write up a more detailed post on this, but it's annoying to
throw exceptions without any information EXCEPT what can be converted
into a string at runtime at the time of exception. All that is missing
is this hook to generate the message.
-Steve
Re: Mobile is the new PC and AArch64 is the new x64
On Tuesday, 18 September 2018 at 07:53:31 UTC, Joakim wrote: On Monday, 17 September 2018 at 22:27:41 UTC, Neia Neutuladh wrote: You are making your arguments to fit your desires. I can't make head nor tails of this claim, you have a talent for vague non sequiturs. My arguments are based on data, the overwhelming sales numbers I linked. I have no idea what desires you think are involved, I suspect you don't either. :) Data and statistics are open to interpretation and you consistently interpret them to fit your desired direction. If oil production ever drops 30% because some workable substitute comes along, as has happened to PCs now, yes, there is no chance of stabilization. It will be a steady decline from there, as these trends have a kind of momentum. Only if the new product meets all the use cases of the old product. Again this is what you dont understand. So you're not averse to having some external hardware sat on your desk. Hmmm. My original post links to examples of using your smartphone connected to a keyboard and monitor or a laptop shell, so I'm not sure where you ever got the idea I was against "external hardware." If you're gonna have all that on your desk it's no stretch to think well i'll have computer on there too. Oh shit yeah I forgot we're only allowed one computer. Not if you're into high end gaming. The vast majority of PCs don't have cards capable of that either. For the few who want it, there will be specialized solutions, whether consoles or whatever. It's one of the many use cases that mobile doesnt meet. It's one of the areas where PC sales has been growing year on year. But of course that doesnt fit your wishful thinking scenario. Obviously people will ditch PCs for consoles, oh wait they said that 15 years ago, and now they are saying the opposite. There's no such thing as professional audio breakout box for android AFAIK. Up until a few years ago the problem was Android couldn't do low latency audio, I'm not sure if the situation has changed. If and when that becomes a market that actually matters, somebody will cater to it, just as google optimized the Android video stack for VR a couple years ago: It's not about it being a market that matters, it's a fact that the kind of processing power and hardware you need for digital audio workstation isnt met by mobile devices and likely never will be because it's like 3d rendering. People doing this stuff always want more cpu cycles, and more bandwidth and storage. It's not like browsing the web where it gets to a point where it's good enough, it's never good enough. http://authorearnings.com/wp-content/uploads/2017/01/Slide29.jpg And how does that contradict anything I said? It contradicts your statement... "print is pretty much dead" If you can make the argument that print is dead and ebooks booming when print still outsells ebooks in unit sales 2 to 1, and even more than that if you look at revenue then you need to see a shrink. :) Or I actually understand market dynamics, :) which it's now become clear that you don't. This print decline is never going to stop. And yet sales of print books have been up for each of the last three years. And you call that decline. ebooks have peaked while print keeps growing, whereas the article I linked and this guy's data show ebooks growing and print continuing to decline. You didnt read the article carefully enough. The growth is in "adult fiction", the market as a whole has fallen. It's funny how you keep making mistakes and then act as though I made them. The article notes that the indie ebook market has likely "taken off," more than making up for the slight drop in ebook sales from publishers. LOL, if i tried to say "PC sales are likely levelling out" you'd be all over it, oh wait you were. Hypocritical much. So the reason to look at the 4 to 1 ratio of ebook sales to print of adult fiction online is because it's a bellwether, as one of the largest market segments that's actually driven by consumers. You're cherry picking the one bit of data that fits your narrative and then making up some BS about it being a bellwether for the whole market. Tell you what you find me one reputable source that backs up your BS about adult fiction being a bellwether. It's not one or the other. I cant believe you dont understand that. Yes, yes, we all know how many people are still rocking VHS tapes, iPods, point-and-shoot cameras, etc., all the tech I've pointed out has basically died off. Because DVDs (and now streaming) did everything VHS did and more. Because phones can do everything an IPOD does and more. Because phones do everything P&S cameras did and more. Phones do not do everything a desktop PC can do, far from it. Phones cant do everything a DSLR can do, and likely wont. etc.. 15 years ago, around the time of the original xbox, people were saying the PC gaming market was d
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir Panteleev wrote: BTW, something follows from the above: write(`C:\` ~ (short path) ~ `con`) will fail but: write(`C:\` ~ (long path) ~ `con`) will succeed. This is just one issue I've noticed... there's probably more lurking. Also, according to the internet: write(chainPath(shortDirectory, "A "), "Win32 API strips trailing space"); readText(chainPath(shortDirectory, "A")); // Win32 API strips trailing space But: write(chainPath(longDirectory, "A "), "Win32 API strips trailing space"); readText(chainPath(longDirectory, "A")); // File not found write(chainPath(shortDirectory, "A.")); // fails write(chainPath(longDirectory, "A.")); // succeeds This is why I think the whole idea is bankrupt. This is why we should use the exact same behavior in all cases. Always use `\\?\` or never use it. Since Windows path handling is weird by default, I'd prefer always using `\\?\`. It's overhead, but not a huge amount of additional overhead compared to filesystem manipulation.
Re: Small @nogc experience report
On 08/09/18 11:07, Peter Alexander wrote:
I'd love to know if anyone is making good use of @nogc in a larger code
base and is happy with it. Weka.io?
No, sorry.
Actually, yes.
Well, sortof.
The main Weka codebase hardly uses any annotations of any kind. Not
@nogc nor others. This is in the process of being amended, somewhat, but
is not a high priority. We do use run-time detection of GC use. I.e. -
we've modified the druntime to invoke a callback if a GC allocation
takes place, and we then log that fact (with traceback). We then are
able to search logs for GC allocations and remove them.
So that's the "no" part.
As pointed out, one of the main motivations for running Mecca was to
clear up the strange solutions that have accumulated over the years. As
such, the Mecca code does have @nogc in much more wide use.
So, yes.
There is a catch, though. Writing Mecca with @nogc required
re-implementing quite a bit of druntime. Mecca uses its own exception
allocations (mkEx, just saw it's not yet documented, it's under
mecca.lib.exception). The same module also has "enforceNGC". We also
have our own asserts. This is partially to support our internal logging
facility, that needs a static list of formats, but it also solves a very
important problem with D's @nogc:
void func() @nogc {
assert(condition, string); // string is useless without actual info
about what went wrong.
assert(condition, format(string, arg, arg)); // No good - format is
not @nogc
ASSERT!"format"(condition, arg, arg); // @nogc and convenient
}
So, yes, we do use @nogc, but it took a *lot* of work to do it.
The good news is that mecca is available, and you can just dub it into
your project and use it, so you don't have to repeat that whole set of work.
Mecca was advertised mostly around the reactor. While it is a piece of
work I am very proud of, it is not the only part there, nor is it
necessary to use mecca's reactor if you want just the library. In fact,
nothing outside of mecca.reactor depends on the reactor running.
Hope this helps,
Shachar
Re: Small @nogc experience report
I've got plenty to say, but here is the long and the short of it: Use Mecca.
On 07/09/18 19:44, Peter Alexander wrote:
3. It was really frustrating that I had to make the compiler happy
before I was able to run anything again. Due to point #1 I had to move
code around to restructure things and wanted to make sure everything
continued working before all GC allocations were removed.
mecca.lib.reflection has "as".
https://weka-io.github.io/mecca/docs/mecca/lib/reflection/as.html
Here is how you use it:
void fun() @nogc {
as!"@nogc"( some code that is not @nogc );
}
4. I used std.algorithm.topNCopy, which is not @nogc. The error just
says "cannot call non-@nogc function [...]". I know there are efforts to
make Phobos more @nogc friendly, but seeing this error is like hitting a
brick wall. I wouldn't expect topNCopy to use GC, but as a user, what do
I do with the error? Having to dig into Phobos source is unpleasant.
Should I file a bug? What if it is intentionally not @nogc for some
subtle reason? Do I rewrite topNCopy?
5. Sometimes I wanted to add writeln to my code to debug things, but
writeln is not @nogc, so I could not. I could have used printf in
hindsight, but was too frustrated to continue.
mecca.log has logging facilities that are @nogc.
Now, to be fair, they are not actually @nogc, as by default it uses
writeln. It is, however, annotated with @nogc, for precisely the reasons
you encountered.
Shachar
Re: Visual Studio Code, Code-D and Microsoft's Visual C++ Debugger (cppvsdbg)
On Wednesday, 19 September 2018 at 14:37:32 UTC, Void-995 wrote: On Wednesday, 19 September 2018 at 14:29:27 UTC, Vladimir Panteleev wrote: [...] Unfortunately you can't make Natvis to understand D symbols as it's purely for C++. You can make C# extension for Visual Studio to do pretty much whatever, but that's not usable much as it doesn't work for VS Code from what I found (and C# are extremely complex for this matter from my point of view). I think Visual D is using Mago internally via DLLs or something, and doing it's own thing. Something like -gc would be great to have again (I guess I missed that one), even -gcpp (to pretend it's C++) that can be used by pretty printing in GDB (fairly simple changes in Python scripts from printing standard C++ stuff) and Microsoft's Debugger. So the classes, namespaces, containers, dynamic arrays can be easily represented with actually having structure and sense from C++ point of view. Forgot to mention that Microsoft added partial support (that may be more than sufficient for D case) to their C/C++ in VS code for both GDB and LLDB. So this one may be useful to make debugging more friendly on Linux and Mac as well!
Re: Visual Studio Code, Code-D and Microsoft's Visual C++ Debugger (cppvsdbg)
On Wednesday, 19 September 2018 at 14:29:27 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 14:22:13 UTC, Void-995 wrote: [...] Cool! What if that will be added as DMD command line option DMD used to have the switch -gc, which meant to emit debug information but pretend to be C as much as possible. It was removed as it wasn't considered necessary any more, but this looks like the perfect use case for it. I suppose it's not possible to make Natvis understand D symbols? Rainer Schuetze would be the person closest to this topic, I think. Doesn't Visual D already have something in it to get the VS debugger to understand some D types better? Unfortunately you can't make Natvis to understand D symbols as it's purely for C++. You can make C# extension for Visual Studio to do pretty much whatever, but that's not usable much as it doesn't work for VS Code from what I found (and C# are extremely complex for this matter from my point of view). I think Visual D is using Mago internally via DLLs or something, and doing it's own thing. Something like -gc would be great to have again (I guess I missed that one), even -gcpp (to pretend it's C++) that can be used by pretty printing in GDB (fairly simple changes in Python scripts from printing standard C++ stuff) and Microsoft's Debugger. So the classes, namespaces, containers, dynamic arrays can be easily represented with actually having structure and sense from C++ point of view.
Re: Visual Studio Code, Code-D and Microsoft's Visual C++ Debugger (cppvsdbg)
On Wednesday, 19 September 2018 at 14:22:13 UTC, Void-995 wrote: [...] Cool! What if that will be added as DMD command line option DMD used to have the switch -gc, which meant to emit debug information but pretend to be C as much as possible. It was removed as it wasn't considered necessary any more, but this looks like the perfect use case for it. I suppose it's not possible to make Natvis understand D symbols? Rainer Schuetze would be the person closest to this topic, I think. Doesn't Visual D already have something in it to get the VS debugger to understand some D types better?
Visual Studio Code, Code-D and Microsoft's Visual C++ Debugger (cppvsdbg)
Good time of day, everyone
Recently I've got myself into D Language again, but got a problem
with debugging on Windows, obviously. While Visual D with Mago
provides pretty comfortable workflow, I highly appreciate what
Code D did with it's more closer and tight integrating with
DScanner and DFormat, better DUB workflow and so on... The
biggest problem there is debugging on Windows OS. GDB from either
Microsoft's C/C++ plugin (ms-vscode.cpptools) or WebFreak's
Native Debug (webfreak.debug) didn't took me anywhere. Broken
Mago-MI integration in WebFreak's Native Debug (which I have
"fixed" locally after few moves around) didn't took me far as
well, the best I got was something bad about Stack Frame.
Meanwhile, the original Microsoft's Visual C++ Debugger
(cppvsdbg) is working pretty much flawless with D. With the
problem, of course, not being able to show what is inside of the
slice/dynamic arrays. Having "fun" with Natvis (The Visual Studio
Natvis framework lets you customize the way Visual Studio
displays native types in debugger variable windows) I found that
giving D types to catch won't work as Visual C++ Debugger
(cppvsdbg) is expecting valid C++ names to work with.
I mostly gave up there, but then I found interesting trick that
Visual Rust and Rust plugin for Visual Studio Code are using.
They added option to compiler to write debug information as
that's C++ code (so they doesn't have to come up with their own
debugger for Windows), with pretty simple trick in rustc and
simplest Natvis file they then have integrated into PDB that they
ship with it. So I've tried to apply that to DMD and see what
will happen.
Currently, as matter of experiment, I've changed it on my side
permanently, but thinking about adding new flag to DMD. The
change was about this big:
//
diff --git a/src/dmd/hdrgen.d b/src/dmd/hdrgen.d
index d9eca8721..a02c6bfef 100644
--- a/src/dmd/hdrgen.d
+++ b/src/dmd/hdrgen.d
@@ -814,8 +814,9 @@ public:
else
{
L1:
+buf.writestring("__dlang::__darray<");
visitWithMask(t.next, t.mod);
-buf.writestring("[]");
+buf.writestring(">");
}
}
//
Now, equipped with that one and fairly small Natvis file that can
be integrated into DMD somewhere and/or Code D, I've got nice
views with flawlessly working Visual C++ Debugger on my D code.
For matter of showcase, links with the screenshots:
1)
https://cdn.discordapp.com/attachments/463259852311887874/491953617545330689/unknown.png
2)
https://cdn.discordapp.com/attachments/463259852311887874/491956446469029888/unknown.png
This one can be further expanded with correct expanding of
string, wstring, dstring and/or any other built-in
interesting/specific/complex types from D. What people working on
DMD think about adding something likes this? Will it break
everything? What if that will be added as DMD command line option
to make life easier for end user on Windows for their own
projects (at least they may enable that temporary while
debugging, so it won't add side effects to anything else)?
P.S. Even without any optional switch that haven't broke Mago on
Visual D, so working with D, especially debugging, can become
far, far easier than it's now. Even with this small change I'm
able to continue working on my project without thinking twice why
I haven't stayed on C++.
Re: extern(C++, ns) is wrong
On 9/18/18 9:49 PM, Jonathan M Davis wrote: On Tuesday, September 18, 2018 6:22:55 PM MDT Manu via Digitalmars-d wrote: https://github.com/dlang/dmd/pull/8667 O_O Thank you Walter for coming to the party! Oh, wow. I sure wasn't expecting that. I thought that he'd made it pretty clear that a DIP was needed, and even then, it didn't seem likely that it would be accepted. This is awesome. I guess that he finally came around. I think a big part is that the implementation was done. I think there's a big difference between "I don't really love this, but crap, I'll have to implement it all" and "I don't really love this, but the implementation isn't too intrusive, and all I have to do is click the merge button, OK." I sure wish I had the skills to hack dmd, there are so many ideas I'd like to see implemented in the language :) Anyways, great to see this merged! -Steve
What's going on with the DMD nightlies releases ?
The downloads of nightlies is broken since at least 2 weeks now. What's going on ?
Re: Small @nogc experience report
On Friday, 7 September 2018 at 17:01:09 UTC, Meta wrote: So it seems that it's never worked. Looking at the implementation, it uses a std.container.BinaryHeap, so it'd require a small rewrite to work with @nogc. AFAICT, extending std.container with support for specifying you own @nogc (malloc-based) allocators is one way of making `topNCopy` not use the GC.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:13:31 UTC, Ecstatic Coder wrote: On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder wrote: None would ever be, considering you obviously have decided to ignore such a simple solution to the 260 character limit... Add "ad hominem" to your pile of fallacies, I guess. Now I will, thanks :) Once again, this forum proves to be very effective at removing any motivation from D users to get involved and contribute to the D language. Sorry, I didn't intend to make you feel unwelcome. But, you did say I was ignoring something, after I've addressed it twice in the same thread. That wasn't nice of you either. In any case, the solution in question won't work: https://forum.dlang.org/post/riuqfngyzfumehsmo...@forum.dlang.org
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 09:58:00 UTC, Vladimir Panteleev wrote: On my Windows VM, I get: C:\(long path here): The filename or extension is too long. (error 206) This seems like a completely reasonable error message to me, so I think we're good there already. It can be a long file name. Each individual component in the path still should not exceed MAX_PATH even when you use unparsed path.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 10:29:11 UTC, Vladimir Panteleev wrote: - GetFullPathName is documented as also having the MAX_PATH limit, but the framework seems to use it for normalization BEFORE prepending the prefix. https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea That part is erroneous :) Recent additions to msdn are too often incompetent, e.g. long path registry switch in windows 10 doesn't enable the feature, it's only the first of two gates, manifest is not optional. Don't ask me who does it. They can't even get backslashes right. Passing unparsed path to GetFullPathName makes no sense because such unparsed path must be already full normalized path, and it doesn't have MAX_PATH limit, probably because it doesn't talk to file system. - GetFullPathName has a big warning on it about how you shouldn't use it in multithreaded programs. I wonder about that too, shouldn't the system do the same to resolve the absolute path? Theoretically it can keep an open handle to the current directory and get its path from that.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 11:04:13 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky (Abscissa) wrote: 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5]. This is not a good criteria: relative paths whose pointing to objects whose absolute path exceeds MAX_PATH will fail, too. So, it looks like Phobos would need to expand relative paths unconditionally. Here is my test program: https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky (Abscissa) wrote: 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5]. This is not a good criteria: relative paths whose pointing to objects whose absolute path exceeds MAX_PATH will fail, too. So, it looks like Phobos would need to expand relative paths unconditionally.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 09:27:29 UTC, Vladimir Panteleev wrote: This might be a change which we won't be able to back out of if it turns out to be a bad idea, because then we break other classes of programs that depend on this change. See https://forum.dlang.org/post/eepblrtjmqzbtopyl...@forum.dlang.org for an example. Case in point: https://msdn.microsoft.com/ru-ru/office/mt762842%28v=vs.90%29?f=255&MSPPError=-2147217396
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 09:58:30 UTC, Kagamin wrote: On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir Panteleev wrote: Someone mentioned in this thread that .NET runtime does do the long-path workaround automatically. AFAIK, CoreFX does, but .net doesn't. .net did its own path normalization and length check, which can be turned off since 4.6.2. Thanks. I had a quick look. It looks pretty involved. Here are some of the relevant parts: https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L16-L43 https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L68-L95 https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/Path.Windows.cs#L37-L63 https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L82-L99 https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L118-L141 And for example directory deletion: https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/System.IO.FileSystem/src/System/IO/Directory.cs#L296-L300 https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/Interop/Windows/kernel32/Interop.RemoveDirectory.cs#L19-L23 Some things stood out to me: - GetFullPathName is documented as also having the MAX_PATH limit, but the framework seems to use it for normalization BEFORE prepending the prefix. https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea - GetFullPathName has a big warning on it about how you shouldn't use it in multithreaded programs. - The code seems to compare the length against 260 characters, but in my tests, the limit is actually about 12 characters shorter. The same file defines MaxShortDirectoryPath = 248, but that constant isn't used anywhere in the code. Maybe we shouldn't use this as a reference after all...
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir Panteleev wrote: Someone mentioned in this thread that .NET runtime does do the long-path workaround automatically. AFAIK, CoreFX does, but .net doesn't. .net did its own path normalization and length check, which can be turned off since 4.6.2.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky (Abscissa) wrote: Instead, what it really means is that our APIs should be designed to *REJECT* long paths with an appropriately meaningful error message On my Windows VM, I get: C:\(long path here): The filename or extension is too long. (error 206) This seems like a completely reasonable error message to me, so I think we're good there already.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 09:16:30 UTC, Nick Sabalausky (Abscissa) wrote: Essentially they boil down to "it is impossible to prove the algorithm is correct" (for both detecting when the path fix is needed, and fixing the path). If you're referring to the inability to deterministically reason about just what in the h*ll MS's API's actually do, then I agree. But the problem is, it's equally true of all Win APIs. Only way to fix that is to omit Win support entirely. It's not our job to fix it. Just provide a D interface to it, which already we do well. Otherwise, I disagree. I think it is not only provable, but also unnecessary to prove simply because such proof has never been necessary for Phobos, and there is nothing inherent to this problem which is inherently more complicated than anything already existing in Phobos (you can even omit the questionable modules like std.xml, it all still holds). No, we are mucking with data on the way between the user's program and the OS, because we think we can fix it. Not only should we not be doing that in the first place, but even if we get it right, it might still not be what the user wants. Otherwise, present counterexamples demonstrating the inherent ambiguity/non-provability. I don't understand what you mean here. Forcing the path transformation can introduce regressions, All phobos/compiler changes have the potential for regressions, plus we have unittests. Unless you can demonstrate how this necessarily goes above and beyond the risk from any other change in a way that cannot be sufficiently mitigated by tests, then the concern is irrelevant. This might be a change which we won't be able to back out of if it turns out to be a bad idea, because then we break other classes of programs that depend on this change. See https://forum.dlang.org/post/eepblrtjmqzbtopyl...@forum.dlang.org for an example. or make the situation worse on systems where it's not needed. Provide an example where the situation is made worse. 1. A user is happily using D on a system where the workaround is not needed. 2. A new D version comes out, with the workaround forcibly enabled. 3. The user's program is now broken. If you provide a specific implementation for the workaround you're envisioning, I could try to come up with more specific situations where it would fail. There's been lots of reasons mentioned in this thread where things can go wrong, and surely there will be more that we can't think of ahead of time.
Re: phobo's std.file is completely broke!
On 09/19/2018 04:41 AM, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky (Abscissa) wrote: What's the other issue(s)? Essentially they boil down to "it is impossible to prove the algorithm is correct" (for both detecting when the path fix is needed, and fixing the path). If you're referring to the inability to deterministically reason about just what in the h*ll MS's API's actually do, then I agree. But the problem is, it's equally true of all Win APIs. Only way to fix that is to omit Win support entirely. Otherwise, I disagree. I think it is not only provable, but also unnecessary to prove simply because such proof has never been necessary for Phobos, and there is nothing inherent to this problem which is inherently more complicated than anything already existing in Phobos (you can even omit the questionable modules like std.xml, it all still holds). Otherwise, present counterexamples demonstrating the inherent ambiguity/non-provability. Forcing the path transformation can introduce regressions, All phobos/compiler changes have the potential for regressions, plus we have unittests. Unless you can demonstrate how this necessarily goes above and beyond the risk from any other change in a way that cannot be sufficiently mitigated by tests, then the concern is irrelevant. or make the situation worse on systems where it's not needed. Provide an example where the situation is made worse.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:46:13 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir Panteleev wrote: If you're referring to NUL, COM1, COM2, etc, then this is completely orthogonal. Yes. How so? It is the same issue: paths with certain properties are valid on all platforms except on Windows. Phobos errors out when attempting to access/create them. A simple workaround is available: expand/normalize the path, prepend the UNC prefix, and use Unicode APIs. I just remembered, there is a third class of paths with these properties: paths containing directory components that begin or end with spaces. There are probably more... I think some special characters are also valid only in UNC paths. BTW, something follows from the above: write(`C:\` ~ (short path) ~ `con`) will fail but: write(`C:\` ~ (long path) ~ `con`) will succeed. This is just one issue I've noticed... there's probably more lurking. This is why I think the whole idea is bankrupt.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir Panteleev wrote: If you're referring to NUL, COM1, COM2, etc, then this is completely orthogonal. Yes. How so? It is the same issue: paths with certain properties are valid on all platforms except on Windows. Phobos errors out when attempting to access/create them. A simple workaround is available: expand/normalize the path, prepend the UNC prefix, and use Unicode APIs. I just remembered, there is a third class of paths with these properties: paths containing directory components that begin or end with spaces. There are probably more... I think some special characters are also valid only in UNC paths.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky (Abscissa) wrote: What's the other issue(s)? Essentially they boil down to "it is impossible to prove the algorithm is correct" (for both detecting when the path fix is needed, and fixing the path). Forcing the path transformation can introduce regressions, or make the situation worse on systems where it's not needed.
Re: One awesome GC feature we will use in Mir!
On Tuesday, 18 September 2018 at 23:01:46 UTC, Per Nordlöw wrote: On Tuesday, 18 September 2018 at 14:23:44 UTC, 9il wrote: I just remember that D's GC has NO_SCAN [1] attribute! I thought D libraries like Mir and Lubeck only had to care about when to call GC.addRange after allocations that contain pointers to GC-backed storage and GC.removeRange before their corresponding deallocations. But that's perhaps only when using non-GC-backed allocators (not using new), right? GC.addRange and GC.removeRange are necessary to mark memory blocks allocated outside of the GC when they may contain pointers to memory allocated by the GC. For example, if you have a @nogc container, the array backing the container can allocated vie libc's malloc. The users are free push elements allocated by the GC on this container, so to be safe, the container must use GC.addRange to tell the GC that the array may point to such elements. When the array grows, shrinks or the whole container is destoryed, the array needs to be passed to GC.removeRange, so the GC would be prevented from scanning freed memory. When you use the GC it automatically does the book keeping for you. It's only when you manually manage memory that you need to be careful whether this memory points to GC objects.
Re: phobo's std.file is completely broke!
On 09/19/2018 02:55 AM, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky (Abscissa) wrote: - Does it actually, necessarily perform those additional OS calls? We need to expand relative paths to absolute ones, for which we need to fetch the current directory. So in other words, NO, it does NOT necessarily perform additional OS calls. It ONLY performs an additional call if the given path is relative *AND* if we've decided to not simply reject too-long relative paths outright (which I'd be fine with as a compromise. At least it would be well-defined and enforced with a meaningful message.) - Is it really? Is what really what? If you mean the memory allocation, we do need a buffer to store the current directory. We also need to canonicalize away things like \..\, though we may be able to get away with it without allocating. So, in many cases, it's NOT really "a good deal of extra logic that performs additional OS calls and generates additional GC garbage". - If it actually does, are those additional, necessarily OS calls prohibitively expensive? They are certainly going to be less expensive that actual filesystem operations that hit the physical disk, Sounds like QED to me. Especially if the alternative is silently incorrect behaviour on an entirely realistic subset of cases. (Realistic enough that both the thread's OP and the bug report's OP each ran into purely by accident.) but it will still be an unwanted overhead in 99.9% of cases. That's an extremely exaggerated figure, and we've already established that the overhead is minor and often able to be elided. Weighted against the cost of incorrect behaviour on a subset of non-rejected inputs, I'd say that's a very clear "Yes, please!" The extreme minority of currently-hypothetical cases which require minimal overhead for individual file I/O operations are free to low-level optimize themselves as-needed. Correct behaviour should never be sacrificed for minor performance tweaks when the minor performance tweak can still be obtained through other means if absolutely necessary. In any case, the overhead is only one issue. What's the other issue(s)?
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky (Abscissa) wrote: Someone mentioned in this thread that .NET runtime does do the long-path workaround automatically. One thing we could do is copy EXACTLY what C# is doing. This is a complete textbook example of the "appeal to authority" fallacy. If an approach is valid, then it stands on its own merits regardless of whether or not Microsoft implemented it. If an approach in invalid, then it fails on its own demerits regardless of whether or not Microsoft implemented it. What MS has or hasn't implemented and released is completely irrelevant WRT validity and correctness. What it *might* be useful for is as a starting point for further exploration. But that is all. No, absolutely not. Microsoft is in charge of the implementation. We can't know that any deviation from Microsoft's algorithm will work in all situations, or all past/future implementations of the API. There is the considerable possibility that there are situations which we cannot foresee from our limited knowledge of the problem and Windows API implementation; on the other hand, Microsoft not only has complete knowledge of the implementation, but also controls its future. They have an incentive to keep the .NET algorithm working. If we deviate from the .NET algorithm and D breaks (but not C#), it is our fault. If we implement the .NET algorithm, then we are as good as C#. If it breaks, it's Microsoft's fault. You cannot evaluate any intrinsic merit here because the result is beyond your control. If one extra OS API call + allocation per std.file API call is unacceptable, then explain how it is unacceptable. I disagree that it is significant enough to be unacceptable. It is not unacceptable, but it is a drawback. If a user needs to optimize their already-working-for-all-accepted-inputs application, then they are free to do so. I argue that building this into the standard library's default behaviour amounts to mandatory premature optimization, prioritizing premature optimization over correctness. Prove me wrong. You could extend this argument to any severity of workarounds. Where do you draw the line? - Using paths longer than MAX_PATH is an exceptional situation. Putting the workaround in the main code path penalizes 99.9% of use cases. I have many filepaths on my system right now which exceed MAX_PATH in total length. I submit that this "penalty" you speak of is nothing more than a trivial performance boost at the expense of correctness. Furthermore, I submit that long paths which need extra optimization are MORE exceptional than long paths which do NOT need extra optimization. Optimization is the least concern. - The registry switch in newer Windows versions removes the need for this workaround, so systems with it enabled are penalized as well. Using the Phobos-based workaround on a system WITH the longpath setting supported and enabled results in slightly reduced performance (which can be overridden and optimized when necessary). I'm more concerned about differences in behavior. OTOH, NOT using the Phobos-based workaround on a system where the longpath setting is NOT supported *OR* NOT enabled results in erroneous behavior. I disagree that failure on paths exceeding MAX_PATH is necessarily erroneous behavior. The API reports an error given the user's path, so should Phobos. The superior default is clear: Use the workaround except where the workaround in known to be safe to omit. We don't even have an algorithm for determining for sure when the workaround is needed. - There is still the matter regarding special filenames, If you're referring to NUL, COM1, COM2, etc, then this is completely orthogonal. Yes. How so? It is the same issue: paths with certain properties are valid on all platforms except on Windows. Phobos errors out when attempting to access/create them. A simple workaround is available: expand/normalize the path, prepend the UNC prefix, and use Unicode APIs. as well as whet her the expected behavior is really to succeed and create paths inaccessible to most software, instead of failing. Ok, suppose we decide "Sure, we have reason to believe there may be a significant amount of software on Windows which fails to handle long paths and we want to ensure maximum compatibility with those admittedly broken programs." That's fine. I can get behind that. HOWEVER, that does NOT mean we should leave our APIs as they are, because currently, our APIs fail at that goal. Instead, what it really means is that our APIs should be designed to *REJECT* long paths with an appropriately meaningful error message - and a reasonable workaround - and NOT to blindly just pass them along as they currently do. This is not possible, because you need to precisely know how the implementation will handle the path. Considering the implementation's behavior can be configured by th
Re: phobo's std.file is completely broke!
They are certainly going to be less expensive that actual filesystem operations that hit the physical disk, but it will still be an unwanted overhead in 99.9% of cases. In any case, the overhead is only one issue. Seriously, checking the file path string *length* is above 260 characters to see if it needs to be fixed is not what I call an overhead. And IF the path is indeed too long, IN THOSE CASES personally I'd prefer that the D standard library fixes the path in order to make the disk/file operation succeed, than having my application crash, because I didn't know I had to put a "version ( Windows )" fix somewhere in my code. But hey, I may be wrong, software robustness and stability is often much overrated... ;)
Re: phobo's std.file is completely broke!
On 09/19/2018 02:26 AM, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky (Abscissa) wrote: [...] Someone mentioned in this thread that .NET runtime does do the long-path workaround automatically. One thing we could do is copy EXACTLY what C# is doing. This is a complete textbook example of the "appeal to authority" fallacy. If an approach is valid, then it stands on its own merits regardless of whether or not Microsoft implemented it. If an approach in invalid, then it fails on its own demerits regardless of whether or not Microsoft implemented it. What MS has or hasn't implemented and released is completely irrelevant WRT validity and correctness. What it *might* be useful for is as a starting point for further exploration. But that is all. However, there are still drawbacks to this: - There is still the matter of overhead (one OS API call (GetCurrentDirectory) and at least one GC allocation (for the current directory buffer)). If one extra OS API call + allocation per std.file API call is unacceptable, then explain how it is unacceptable. I disagree that it is significant enough to be unacceptable. If a user needs to optimize their already-working-for-all-accepted-inputs application, then they are free to do so. I argue that building this into the standard library's default behaviour amounts to mandatory premature optimization, prioritizing premature optimization over correctness. Prove me wrong. - Using paths longer than MAX_PATH is an exceptional situation. Putting the workaround in the main code path penalizes 99.9% of use cases. I have many filepaths on my system right now which exceed MAX_PATH in total length. I submit that this "penalty" you speak of is nothing more than a trivial performance boost at the expense of correctness. Furthermore, I submit that long paths which need extra optimization are MORE exceptional than long paths which do NOT need extra optimization. - The registry switch in newer Windows versions removes the need for this workaround, so systems with it enabled are penalized as well. Using the Phobos-based workaround on a system WITH the longpath setting supported and enabled results in slightly reduced performance (which can be overridden and optimized when necessary). Note that it is possible (and very simple) to detect this situation and handle it optimally by skipping the workaround. OTOH, NOT using the Phobos-based workaround on a system where the longpath setting is NOT supported *OR* NOT enabled results in erroneous behavior. Not something as trivial as slightly-degraded performance on IO access. The superior default is clear: Use the workaround except where the workaround in known to be safe to omit. - There is still the matter regarding special filenames, If you're referring to NUL, COM1, COM2, etc, then this is completely orthogonal. as well as whet her the expected behavior is really to succeed and create paths inaccessible to most software, instead of failing. Ok, suppose we decide "Sure, we have reason to believe there may be a significant amount of software on Windows which fails to handle long paths and we want to ensure maximum compatibility with those admittedly broken programs." That's fine. I can get behind that. HOWEVER, that does NOT mean we should leave our APIs as they are, because currently, our APIs fail at that goal. Instead, what it really means is that our APIs should be designed to *REJECT* long paths with an appropriately meaningful error message - and a reasonable workaround - and NOT to blindly just pass them along as they currently do. Either way, Phobos needs changed: Do you believe D should prevent its own software from being broken on long paths? Then Phobos should be modified to detect and fix long paths. Do you believe D should permit breakage on long paths and encourage its programs to play nicely with other non-D Windows software that is *also* broken on long paths? Then Phobos should be modified to detect and *reject* long paths. Either way, the current Phobos behavior is clearly the worst of both worlds and needs modification.
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir Panteleev wrote: On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder wrote: None would ever be, considering you obviously have decided to ignore such a simple solution to the 260 character limit... Add "ad hominem" to your pile of fallacies, I guess. Now I will, thanks :) Once again, this forum proves to be very effective at removing any motivation from D users to get involved and contribute to the D language. That's probably one of the keys of its success...
Re: phobo's std.file is completely broke!
On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky (Abscissa) wrote: - Does it actually, necessarily perform those additional OS calls? We need to expand relative paths to absolute ones, for which we need to fetch the current directory. - Is it really? Is what really what? If you mean the memory allocation, we do need a buffer to store the current directory. We also need to canonicalize away things like \..\, though we may be able to get away with it without allocating. - If it actually does, are those additional, necessarily OS calls prohibitively expensive? They are certainly going to be less expensive that actual filesystem operations that hit the physical disk, but it will still be an unwanted overhead in 99.9% of cases. In any case, the overhead is only one issue.
