Re: Head Const
On Friday, 19 February 2016 at 06:39:53 UTC, Walter Bright wrote: On 2/18/2016 9:46 PM, Jonathan M Davis wrote: On Thursday, 18 February 2016 at 22:40:32 UTC, Walter Bright wrote: On 2/18/2016 4:16 AM, Jonathan M Davis wrote: headconst may solve the extern(C++) problem, but it really doesn't solve the problems we have with const. 'mutable' doesn't really solve a problem, it just means that C++ 'const' is a documentation aid, not a guarantee. It's still a guarantee for those members that aren't mutable. It's allowing casting away const and mutating that totally blows the guarantees out of the water. That's why such casts are not allowed in D @safe code. C++ const does not come with mechanically checkable guarantees, and D's does. This makes all the difference. Except that if you're calling 3rd party code, it's free to cast away const and mutate and slap an @trusted on there, and if it's more than one level deep in the call stack, they could slap an @safe on the function you're actually calling, and you wouldn't have a clue that they're violating the type system - and it could be with something you passed to them. Yes, @safe helps some, but ultimately, you still rely on the programmer who wrote the code you're using to behave responsibly. Ultimately, the D compiler doesn't prevent code from mutating const any more than the C++ compiler does. The primary differences are that C++ considers it defined behaved, whereas D does not, and D has immutable to worry about, whereas C++ does not. And unless the compiler is optimizing based on const, as long as the data in question was not constructed as immutable, casting away const and mutating will work in D just as well as it does in C++. So, it's easy for a programmer to think that it's perfectly legitimate to cast away const and mutate and then think that it's @system purely because of the possibility of the data actually being immutable and not even realize that it's undefined behavior even when the data was constructed as mutable. Even Andrei made that mistake fairly recently: http://forum.dlang.org/post/n25qkc$2il8$1...@digitalmars.com He cast away const from the allocator member inside of a const member function so that he could provide access to it, since it was not part of the logical state of the container and really shouldn't have been const but had to be, because the container was const. @mutable is exactly what was needed in this case. But he (and Dicebot) thought that casting away const and mutating was defined behaved as long as the underlying data was mutable until I pointed out otherwise. http://forum.dlang.org/post/resphkhblryhrlznx...@forum.dlang.org While in theory, you can't cast away D's const and mutate, all that's preventing you is @safe, and it's clear that even expert D programmers who should arguably know better don't know better. I've heard (though haven't verified) that vibe.d casts away const to do reference counting, and it's clear from some of the stackoverflow answers that a number of D programmers think that casting away const and mutating is fine, because D is a systems language. Programmers are casting away const and mutating in practice, and the compiler is not preventing it. We're in exactly the same boat as C++ in that we rely on the programmer to be smart about casting away const in order for const to provide any guarantees. We just made it undefined behavior when they do cast away const and made it warn about it slightly better by making it @system. But it still works in practice, and folks are still doing it. So, ultimately, I don't think that the idea that D's const guarantees that the object isn't mutated via that reference holds water much better than it does with C++'s const, even if in theory it should. In both cases, it relies on the programmer to do the right thing, and the compiler can't do much if you insist on doing the wrong thing, thinking that you know what you're doing. And the fact that it actually works in practice to cast away const and mutate doesn't help matters any in preventing it. Allow @mutable, and no more mechanical checking in D. Recall that D supports opaque types, meaning types are not fully known to the compiler. Yes, which is why I said that a type with an @mutable member would be forced to be marked with @mutable as well (or something like @has_mutable if it's problematic to reuse the attribute on the class/struct). It would be the same as with an abstract class. So, then anything that wouldn't work with @mutable (e.g. constructing the type as immutable) could be statically prevented, even if the type were opaque, just like you can't construct an abstract class. it would obviously have to bubble up such that a type that contains an @mutable type would also have to be an @mutable type, but it would solve the opaque type problem. - Jonathan M Davis
Re: Poor memory allocation performance with a lot of threads on 36 core machine
On Thu, 2016-02-18 at 17:27 +, Chris Wright via Digitalmars-d wrote: > […] > > I would like to look into D's GC and parallelism more. I've started > on > mark/sweep parallelism but haven't made any worthwhile progress. > I'll > take this as my next task. It's more awkward because it requires > changes > to the runtime interface, which means modifying both DMD and the > runtime. The OpenJDK folk have more or less given up on traditional mark/sweep and even concurrent mark/sweep for JVM-based codes (most because there is still a need for a global stop the world collection at some points): the G1 algorithms that is now the default has a very different approach and appears to be a very significant improvement. See for example: http://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All and all the Oracle and OpenJDK blurb. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: [WIP] Native SQLite Database reader (works at CTFE)
On Thursday, 18 February 2016 at 21:09:15 UTC, Stefan Koch wrote: I am planing to open-source my native-sqlite database driver. (well currently it just reads them). However it works fully at CTFE. Interesting... I'd almost want a page with the ddoc information to glance over the API, and a couple code snippets of how you'd see it working. I'll look forward to seeing this when it's out :)
Re: Head Const
On 2/18/2016 9:46 PM, Jonathan M Davis wrote: On Thursday, 18 February 2016 at 22:40:32 UTC, Walter Bright wrote: On 2/18/2016 4:16 AM, Jonathan M Davis wrote: headconst may solve the extern(C++) problem, but it really doesn't solve the problems we have with const. 'mutable' doesn't really solve a problem, it just means that C++ 'const' is a documentation aid, not a guarantee. It's still a guarantee for those members that aren't mutable. It's allowing casting away const and mutating that totally blows the guarantees out of the water. That's why such casts are not allowed in D @safe code. C++ const does not come with mechanically checkable guarantees, and D's does. This makes all the difference. Allow @mutable, and no more mechanical checking in D. Recall that D supports opaque types, meaning types are not fully known to the compiler.
Re: Official compiler
On Fri, 19 Feb 2016 05:29:20 +, Ola Fosheim Grøstad wrote: > On Thursday, 18 February 2016 at 23:42:11 UTC, Chris Wright wrote: >> There are damages for patent infringement. There are higher damages for >> willful infringement. > > Iff you use it as a means for production. There is nothing illegal about > implementing patented techniques in source code (i.e. describing them) > and distributing it. That depends on where the patent was filed and where the lawsuit is being executed. >> regarding GCC. And thanks to how software patents generally are, it'd >> probably be regarding something that most C/C++ compilers need to >> implement and the most obvious implementation for that feature. > > If that is the case then there will be prior art that predates the > patent. Not if it's for a feature added to a C++ standard after the patent was filed. Not if it's for a feature that modern compilers consider standard but wasn't standard before the patent was created. Not if it's in a jurisdiction that uses first-to-file rather than first-to-invent.
Re: Head Const
On Friday, 19 February 2016 at 05:33:54 UTC, Doc wrote: On Thursday, 18 February 2016 at 11:57:59 UTC, Jonathan M Davis wrote: The more I look at it, the more I'm inclined to think that introducing @mutable for member variables with a corresponding, required attribute on the struct or class it's in (e.g. @has_mutable) is really what we need to be able to solve this problem and make D usable in some of these high performance cases that would be using the mutable keyword in C++. It solves the logical const problem without totally throwing away the compiler guarantees. Any type without @has_mutable functions as it always has, and the cases where @mutable/@has_mutable would be required would then work with const, gaining all of its benefits for the non-@mutable members, and it would allow the compiler to prevent you from doing stupid stuff like mutating immutable objects, because an object with @has_mutable couldn't be immutable. - Jonathan M Davis Could we use a special class Interface for this to limit the widespread use of a new keyword or attribute? I.e. classes could implement a special mutable RefCount (as an example) interface. Only code that refers to the object by it's mutable interface would be allowed to jailbreak its overall constness, and only for those members defined in the mutable interface. Maybe add a MutableInterface keyword or an attribute strictly valid in Interface declarations. Just a late night brainstorm. The mutable keyword is used for stuff that frequently _isn't_ part of the API of a type and thus doing something with interfaces doesn't make a lot of sense to me. And it's really the type that's constructed that needs to clearly have mutable members, not the interface, because the key thing is to avoid constructing immutable variables of that type. In addition, interfaces only apply to classes, which is far too limiting, even if it's otherwise a good idea. Having @mutable would not result in @mutable being plastered everywhere in D anymore than mutable is in C++. If someone used it that much, then there isn't much point in using const in the first place. Rather, certain types will have @mutable on at most a few of their members, and they'll have to have @mutable on the type itself (or something like @has_mutable if it's problematic to reuse the same attribute on the type), so it'll be obvious when it's used and will prevent the compiler from constructing such objects as immutable, since that would break immutability. I wouldn't expect it to be intrusive at all, but it's critical for certain types of objects to work with const at all, and if those objects don't work with const, then the code bases that they're in will avoid const, and generic code will be forced to avoid const regardless, because it could be used with most anything - including stuff that can't be const, whereas if we had @mutable, that problem would be reduced considerably and maybe even eliminated. - Jonathan M Davis
Re: Head Const
On Thursday, 18 February 2016 at 22:40:32 UTC, Walter Bright wrote: On 2/18/2016 4:16 AM, Jonathan M Davis wrote: headconst may solve the extern(C++) problem, but it really doesn't solve the problems we have with const. 'mutable' doesn't really solve a problem, it just means that C++ 'const' is a documentation aid, not a guarantee. It's still a guarantee for those members that aren't mutable. It's allowing casting away const and mutating that totally blows the guarantees out of the water. You can actually look at the type to see which members are mutable and know what might change, and usually it's stuff like mutexes that are perfectly okay to change, whereas with casting, you'd have to look at every spec of code that uses the object to see what it does if you want to be sure that a const object or particular member isn't mutated. And while I very much like the fact that D doesn't consider casting away const and mutating to be defined behavior and that it has stronger guarantees than C++, the reality of the matter is that in many cases, it ultimately provides far worse guarantees than C++. While C++'s const does have too many backdoors, at least when you use it, it catches accidental mutation (e.g. getting the arguments reversed when calling copy from the algorithm header will result in an error). But if you're forced to drop const entirely as tends to happen in D, then you don't get that. So, for some code, the stricter const in D is great, but for a lot of it, it makes things worse. In spite of the stronger guarantees, you ultimately end up with less protection. And since the compiler actually lets you cast away const and mutate in D to your heart's content without complaining at all (even if it's technically undefined behavior), there are plenty of folks that do it, because D doesn't provide mutable, and they assume that because you can cast away const, mutating it must be fine, just like with C++. So, while in principle, D's const provides better guarantees, in practice, it really doesn't. Casting away const would have to be illegal for it really provide those guarantees. And because it's so restrictive, it frequently gets avoided anyway. So, even if it doesn't get circumvented, it still fails to provide good guarantees, because it's not used. At least if we added @mutable, we'd be providing a controlled means of escaping const in the cases where it's needed, avoiding issues with immutable like you'd potentially get if you cast away const and mutated. And any code which didn't use @mutable would be able to rely on the full guarantees that we have now (the only problem being folks that cast away const and mutated, thinking that it was okay, but they'd have less incentive to if they had @mutable). In principle, I agree that having something like @mutable does weaken const, but in practice, it does a much better job of catching bugs, because then you're actually able to use const with more complicated objects. And unlike casting away const, @mutable is easily detected and accounted for. D's const simply doesn't work once you start doing stuff like putting allocators or reference counts into the mix. And while maybe we could not care about that when we thought that it was fine to use the GC with everything, I think that we have a much harder time holding that position now. As principled as D's current position on const may be, it's highly impractical. - Jonathan M Davis
Re: Head Const
On Thursday, 18 February 2016 at 11:57:59 UTC, Jonathan M Davis wrote: The more I look at it, the more I'm inclined to think that introducing @mutable for member variables with a corresponding, required attribute on the struct or class it's in (e.g. @has_mutable) is really what we need to be able to solve this problem and make D usable in some of these high performance cases that would be using the mutable keyword in C++. It solves the logical const problem without totally throwing away the compiler guarantees. Any type without @has_mutable functions as it always has, and the cases where @mutable/@has_mutable would be required would then work with const, gaining all of its benefits for the non-@mutable members, and it would allow the compiler to prevent you from doing stupid stuff like mutating immutable objects, because an object with @has_mutable couldn't be immutable. - Jonathan M Davis Could we use a special class Interface for this to limit the widespread use of a new keyword or attribute? I.e. classes could implement a special mutable RefCount (as an example) interface. Only code that refers to the object by it's mutable interface would be allowed to jailbreak its overall constness, and only for those members defined in the mutable interface. Maybe add a MutableInterface keyword or an attribute strictly valid in Interface declarations. Just a late night brainstorm. -Doc
Re: Official compiler
On Thursday, 18 February 2016 at 23:42:11 UTC, Chris Wright wrote: You testify it under oath, and you hope you look honest. You can show a lack of GCC source code on your home computer, possibly. If they actually have a strong case it will be highly unlikely that you have arrived at it independently. Of course, all you have to do is to remove the code and FSF will be happy. So if you let it go all the way to the court you can only blame yourself for being pedantic. FSF will only sue over a strong case that carries political weight. A loss in court is a PR disaster for FSF. There are damages for patent infringement. There are higher damages for willful infringement. Iff you use it as a means for production. There is nothing illegal about implementing patented techniques in source code (i.e. describing them) and distributing it. regarding GCC. And thanks to how software patents generally are, it'd probably be regarding something that most C/C++ compilers need to implement and the most obvious implementation for that feature. If that is the case then there will be prior art that predates the patent. If Walter had read the GCC source code from an infringing version after that case came to light, that's the sort of thing that can bring on triple damages. It depends on relative lawyer quality, of course, but it's much harder for the plaintiffs if there's no indication that you've accessed the GCC source code. It should help you, not hurt you, if you learnt about a technique from a widespread codebase from an organization that is known for avoiding patents. If anything that proves that you didn't pick it up from the filed patent and was in good faith? If the case came to light (e.g. you knew about it) and you didn't vet your own codebase then you will be to blame no matter where you got it from? But FSF would make sure they remove patented techniques from GCC so that scenario would be very unlikely. In other words, you are more likely to be hit by a bus when crossing the street. I find this kind of anxiety hysterical to be honest. The only thing I get out of this is that companies shouldn't admit to using open source codebases. Of course, one reason for avoiding reading other people's source code is that you have a client that makes it a requirement.
Re: OT: Need help with translation D-faq to other languages
On Thursday, 18 February 2016 at 18:20:48 UTC, Eugene Wissner wrote: On Wednesday, 17 February 2016 at 18:28:08 UTC, Suliman wrote: Hello, I wrote pretty good FAQ about D on Russian language. I need help with translation it to English and Germany and possible to other languages. http://dlang.ru/Why-D-is-Better I hope it people will like it, it will help to attract more people and help noobs better understand what D is. It's pretty short. I can do a German translation Yes, please!
Re: std.xml2 (collecting features)
On Thursday, 18 February 2016 at 10:18:18 UTC, Robert burner Schadek wrote: On Thursday, 18 February 2016 at 04:34:13 UTC, Alex Vincent wrote: I'm looking for a status update. DUB doesn't seem to have many options posted. I was thinking about starting a SAXParser implementation. I'm working on it, but recently I had to do some major restructuring of the code. Currently I'm trying to get this merged https://github.com/D-Programming-Language/phobos/pull/3880 because I had some problems with the encoding of test files. XML has a lot of corner cases, it just takes time. If you want to on some XML stuff, please join me. It is properly more productive working together than creating two competing implementations. Would you be interested in mentoring a student for the Google Summer of Code to do work on std.xml?
Re: Official compiler
On Thursday, 18 February 2016 at 20:28:41 UTC, David Nadlinger wrote: You can use rdmd with ldmd2 just as well (and presumably gdmd too). First I'm hearing of it.
Re: Vulkan bindings
On Thursday, 18 February 2016 at 03:39:30 UTC, Kapps wrote: On Thursday, 18 February 2016 at 03:38:42 UTC, Kapps wrote: This is what I did with OpenGL for my own bindings. It had some nice benefits like having the documentation be (mostly) accessible. Unfortunately, turns out the spec contains a lot of typos, including wrong arguments / function names. And I should clarify, ahead of time to generate a .d file, not at compile-time. :P Yea, by "directly", I meant using D templates and CTFE, not a script that generates a D file. For my own project, since I just need the function names, I'm using a Python script to generate a CSV file from the OpenGL spec, then importing/parsing that with D. It's neat, but slows down the build a lot. I haven't had any issues with typos, though.
Re: Another new io library
On 2/18/16 6:52 PM, Chad Joan wrote: Steve: My apologies in advance if I a misunderstood any of the functionality of your IO library. I haven't read any of the documentation, just this thread, and I my time is over-committed as usual. Understandable. Anyhow... I believe that when I am dealing with streams, >90% of the time I am dealing with data that is *structured* and *heterogeneous*. Here are some use-cases: 1. Parsing/writing configuration files (ex: XML, TOML, etc) 2. Parsing/writing messages from some protocol, possibly over a network socket (or sockets). Example: I am writing a PostgreSQL client and need to deserialize messages: http://www.postgresql.org/docs/9.2/static/protocol-message-formats.html 3. Serializing/deserializing some data structures to/from disk. Example: I am writing a game and I need to implement save/load functionality. 4. Serializing/deserializing tabular data to/from disk (ex: .CSV files). 5. Reading/writing binary data, such as images or video, from/to disk. This will probably involve doing a bunch of (3), which is kind of like (2), but followed by large homogenous arrays of some data (ex: pixels). 6. Receiving unstructured user input. This is my <10%. Note that (6) is likely to happen eventually but also likely to be minuscule: why are we receiving user input? Maybe it's just to store it for retrieval later. BUT, maybe we actually want it to DO something. If we want it to do something, then we need to structure it before code will be able to operate on it. (5) is a mix of structured heterogeneous data and structured homogenous data. In aggregate, this is structured heterogeneous data, because you need to do parsing to figure out where the arrays of homogeneous data start and end (and what they *mean*). This is why I think it will be much more important to have at least these two interfaces take front-and-center: A. The presence of a .popAs!(...) operation (mentioned by Wyatt in this thread, IIRC) for simple deserialization, and maybe for other miscellaneous things like structured user interaction. To me, this is a higher-level function. popAs cannot assume to know how to read what it is reading. If you mean something like reading an entire struct in binary form, that's not difficult to do. B. The ability to attach parsers to streams easily. This might be as easy as coercing the input stream into the basic encoding that the parser expects (ex: char/wchar/dchar Ranges for compilers, or maybe ubyte Ranges for our PostgreSQL client's network layer), though it might need (A) to help a bit first if the encoding isn't known in advance (text files can be represented in sooo many ways! isn't it fabulous!). This is the fundamental goal for my library -- enabling parsers to read data from a "stream" efficiently no matter how that data is sourced. I know your time is limited, but I would invite you to take a look at the convert program example that I created in my library. In it, I handle converting any UTF format to any other UTF format. https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d I understand that most unsuspecting programmers will arrive at a stream library expecting to immediately see an InputRange interface. This /probably/ is not what they really want at the end of the day. So, I think it will be very important for any such library to concisely and convincingly explain the design methodology and rationale early and aggressively. Neglect to do this, and the library and it's documentation will become a frustration and a violation of expectations (an "astonishment"). Do it right, and the library's documentation will become a teaching tool that leaves visitors feeling enlightened and empowered. Good points! I will definitely spend some time explaining this. Of course, I have to wonder if someone else has contrasting experiences with stream use-cases. Maybe they really would be frustrated with a range-agnostic design. I don't want to alienate this hypothetical individual either, so if this is you, then please share your experiences. I hope this helps and is worth making a bunch of you read a wall of text ;) Thanks for taking the time. -Steve
Re: [WIP] Native SQLite Database reader (works at CTFE)
On 02/18/2016 04:09 PM, Stefan Koch wrote: Hello, It is not quite ready to post in Announce, but I would like to inform you that I am planing to open-source my native-sqlite database driver. (well currently it just reads them). However it works fully at CTFE. so if you want to you can extract sourceCode out of a sql-database and make a mixin with it :D given you string imported the database file. That's very exciting! Are you planning a DConf talk on the topic? -- Andrei
Re: Yet another leak in the sinking ship of @safe
On 18.02.2016 21:57, H. S. Teoh via Digitalmars-d wrote: >If you want to verify guarantees, @safe should be specified by >inclusion and ideally, type safety should be proved formally (after >applying a few fixes). >This first requires a formal language semantics. >That's already a lot of effort, and after that you still don't have a >verified implementation. I'm not sure how feasible it is to do a formal proof with the current dmd code, since you'd have to freeze it, and as soon as the next PR is merged, the proof is potentially invalidated. The most obvious reason why it is infeasible to formally prove correctness of the current dmd code is that it is not correct.
Re: Head Const
On 2/18/2016 6:04 AM, Marc Schütz wrote: Rule 2 only forbids _static immutables_ with @mutable members, precisely because those could end up in read-only memory. I don't see how allocators are affected at all. An allocator would be mutable, or at least manage mutable memory. You could create a bunch of immutable data structures, then make the page they are in read-only.
Re: Another new io library
On Wednesday, 17 February 2016 at 06:45:41 UTC, Steven Schveighoffer wrote: It's no secret that I've been looking to create an updated io library for phobos. In fact, I've been working on one on and off since 2011 (ouch). ... Hi everyone, it's been a while. I wanted to chime in on the streams-as-ranges thing, since I've thought about this quite a bit in the past and discussed it with Wyatt outside of the forum. Steve: My apologies in advance if I a misunderstood any of the functionality of your IO library. I haven't read any of the documentation, just this thread, and I my time is over-committed as usual. Anyhow... I believe that when I am dealing with streams, >90% of the time I am dealing with data that is *structured* and *heterogeneous*. Here are some use-cases: 1. Parsing/writing configuration files (ex: XML, TOML, etc) 2. Parsing/writing messages from some protocol, possibly over a network socket (or sockets). Example: I am writing a PostgreSQL client and need to deserialize messages: http://www.postgresql.org/docs/9.2/static/protocol-message-formats.html 3. Serializing/deserializing some data structures to/from disk. Example: I am writing a game and I need to implement save/load functionality. 4. Serializing/deserializing tabular data to/from disk (ex: .CSV files). 5. Reading/writing binary data, such as images or video, from/to disk. This will probably involve doing a bunch of (3), which is kind of like (2), but followed by large homogenous arrays of some data (ex: pixels). 6. Receiving unstructured user input. This is my <10%. Note that (6) is likely to happen eventually but also likely to be minuscule: why are we receiving user input? Maybe it's just to store it for retrieval later. BUT, maybe we actually want it to DO something. If we want it to do something, then we need to structure it before code will be able to operate on it. (5) is a mix of structured heterogeneous data and structured homogenous data. In aggregate, this is structured heterogeneous data, because you need to do parsing to figure out where the arrays of homogeneous data start and end (and what they *mean*). This is why I think it will be much more important to have at least these two interfaces take front-and-center: A. The presence of a .popAs!(...) operation (mentioned by Wyatt in this thread, IIRC) for simple deserialization, and maybe for other miscellaneous things like structured user interaction. B. The ability to attach parsers to streams easily. This might be as easy as coercing the input stream into the basic encoding that the parser expects (ex: char/wchar/dchar Ranges for compilers, or maybe ubyte Ranges for our PostgreSQL client's network layer), though it might need (A) to help a bit first if the encoding isn't known in advance (text files can be represented in sooo many ways! isn't it fabulous!). I understand that most unsuspecting programmers will arrive at a stream library expecting to immediately see an InputRange interface. This /probably/ is not what they really want at the end of the day. So, I think it will be very important for any such library to concisely and convincingly explain the design methodology and rationale early and aggressively. Neglect to do this, and the library and it's documentation will become a frustration and a violation of expectations (an "astonishment"). Do it right, and the library's documentation will become a teaching tool that leaves visitors feeling enlightened and empowered. Of course, I have to wonder if someone else has contrasting experiences with stream use-cases. Maybe they really would be frustrated with a range-agnostic design. I don't want to alienate this hypothetical individual either, so if this is you, then please share your experiences. I hope this helps and is worth making a bunch of you read a wall of text ;) - Chad
Re: Official compiler
On Thu, 18 Feb 2016 22:41:46 +, Ola Fosheim Grøstad wrote: > On Thursday, 18 February 2016 at 22:22:57 UTC, Chris Wright wrote: >> With copyright, the fact that you created yours on your own is >> sufficient defense, assuming the courts agree. If by sheer coincidence >> you come up with code identical to what's in GCC, but you can show that >> you didn't take the code from GCC, you're in the clear. > > And how are you going to show that? You can't, because it is widespread. You testify it under oath, and you hope you look honest. You can show a lack of GCC source code on your home computer, possibly. >> Patents, well, you're infringing even if you didn't refer to any other >> source. But if you did look at another source, especially if you looked >> in the patent database, you open yourself up to increased damages. > > There are no damages for GCC. There are damages for patent infringement. There are higher damages for willful infringement. The patent doesn't have to be held by the FSF or a contributor to GCC. There might be a patent troll that sued a third party regarding GCC. And thanks to how software patents generally are, it'd probably be regarding something that most C/C++ compilers need to implement and the most obvious implementation for that feature. If Walter had read the GCC source code from an infringing version after that case came to light, that's the sort of thing that can bring on triple damages. It depends on relative lawyer quality, of course, but it's much harder for the plaintiffs if there's no indication that you've accessed the GCC source code.
Re: Yet another leak in the sinking ship of @safe
On Thu, 18 Feb 2016 19:17:27 +, Era Scarecrow wrote:
> On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer
> wrote:
>> On 2/18/16 1:30 PM, Timon Gehr wrote:
>>> No problem here. There is no way to assign to a void[] without doing
>>> 2.
>>
>> foo(void[] arr)
>> {
>>void[] arr2 = [1234, 5678, 91011];
>>arr[] = arr2[0 .. arr.length];
>> }
>
> Since void throws away type information (and all the safety
> related to it), would it be easier to simply require @safe code can't
> cast implicitly to void? It seems like explicit casting would take care
> of most of this, or disallowing to/from void converting period while in
> @safe code.
Casting *from* void[] is also a big issue. Disallow all implicit and
explicit casts between void[] and anything else to start, and we can look
at the rest case-by-caste.
We can probably cast to const(void)[] safely, and we can probably cast
arrays that contain no pointers to void[] safely. Casting from void[] to
const(T)[] where T contains no pointers (or arrays or functions or
delegates) should also be safe.
> To be honest, I think there's only 1 time I actually used a
> void[] in my code, and that was while writing a section in the BitArray
> replacement code years back in the case you wanted to use/read another
> block of data as the source for the BitArray. Beyond that I never
> touched it.
A lot of the IO stuff in Phobos uses void[]. std.socket is lousy with it.
I think the intention is that you can send arbitrary data over the wire
without having to explicitly marshal it into a ubyte[].
[OT] Some neat ideas from the Kotlin language
Just come across Kotlin today, and found some interesting ideas skimming through its tutorial: 1) Null check Kotlin has Optional types, suffixed with a '?'. Like 'Int?', same as in Swift. But instead of explicitly unwrapping them (e.g. var! in Swift, or var.unwrap() in Rust), Kotlin let you do this: var: Int? if (var != null) //You can use var here Skipping the null check will get you compile time error. You can even do this: if (var == null) return; //You can use var from now on 2) Smart cast This is a similar to previous one, instead of: var1: Object; var2 = cast(String)var1; You do this: if (var1 is String) //You can use var1 as a String here I think this two ideas are pretty neat, for more information, see: http://kotlinlang.org/docs/reference/typecasts.html
Re: Head Const
On Thursday, 18 February 2016 at 22:48:01 UTC, Walter Bright
wrote:
struct A { int* pi; }
and *pi will be mutable even though pi is __const.
Here's the big deal: when you have started on an implementation
with constness in, and then evolve the codebase and have to turn
some fields mutable or face a major rewrite, guess what you will
do?
You will look for ways to save time and cast away constness,
write some comment that it probably should be fixed later, but if
it looks like it is working it will never be fixed...
So you effectively cannot avoid "mutable". You can claim they
shouldn't do it, but you cannot enforce it.
Re: [WIP] Native SQLite Database reader (works at CTFE)
On Thursday, 18 February 2016 at 21:09:15 UTC, Stefan Koch wrote: Hello, It is not quite ready to post in Announce, but I would like to inform you that I am planing to open-source my native-sqlite database driver. (well currently it just reads them). However it works fully at CTFE. so if you want to you can extract sourceCode out of a sql-database and make a mixin with it :D given you string imported the database file. Mother of god! Looking forward to it!
Re: [OT] Re: Official compiler
On Thursday, 18 February 2016 at 20:18:14 UTC, David Nadlinger wrote: On Wednesday, 17 February 2016 at 22:57:20 UTC, Márcio Martins wrote: […] On a completely unrelated note, you aren't by any chance the Márcio Martins who is giving a talk at ETH in a couple of days, are you? — David No, I'm not.
Re: Head Const
On 2/18/2016 4:18 AM, Andrei Alexandrescu wrote:
On 02/17/2016 05:44 PM, Walter Bright wrote:
It would seem that implementing headconst as a type constructor would
let people who wanted mutable members have their way, without
introducing backdoors in const.
Doesn't seem that way to me, viz:
struct A
{
int i;
}
A __const(A) will have a __const(int) member.
struct A { int* pi; }
and *pi will be mutable even though pi is __const.
Re: Official compiler
On Thursday, 18 February 2016 at 22:33:15 UTC, Iain Buclaw wrote: On 18 February 2016 at 22:23, Jonathan M Davis via Digitalmars-d < digitalmars-d@puremagic.com> wrote: [...] Actually, I'm sure this is a great way to let bugs in. There's no saying what could happen if you switch compiler and turn the optimisations throttle to full. In 99% of cases, one would hope all is good. But the bigger the codebase you're dealing with, the more you should really use both side by side when testing to ensure that no heisenbugs creep in. Yep, that issue I reported a while ago with floating-point casts comes to mind.
Re: Head Const
On 2/18/2016 10:22 AM, Timon Gehr wrote: He wanted to embed a mutable reference count literally within a const object. Not a headconst object. I know. I pointed out how it could be done in a way to achieve the same effect. BTW, shared_ptr<> uses a pointer to the ref count.
Re: Official compiler
On Thursday, 18 February 2016 at 22:22:57 UTC, Chris Wright wrote: With copyright, the fact that you created yours on your own is sufficient defense, assuming the courts agree. If by sheer coincidence you come up with code identical to what's in GCC, but you can show that you didn't take the code from GCC, you're in the clear. And how are you going to show that? You can't, because it is widespread. Patents, well, you're infringing even if you didn't refer to any other source. But if you did look at another source, especially if you looked in the patent database, you open yourself up to increased damages. There are no damages for GCC.
Re: Head Const
On 2/18/2016 4:16 AM, Jonathan M Davis wrote: headconst may solve the extern(C++) problem, but it really doesn't solve the problems we have with const. 'mutable' doesn't really solve a problem, it just means that C++ 'const' is a documentation aid, not a guarantee.
Re: Another new io library
On Thu, Feb 18, 2016 at 03:20:58PM -0500, Steven Schveighoffer via Digitalmars-d wrote: > On 2/18/16 2:53 PM, Wyatt wrote: > >On Thursday, 18 February 2016 at 18:35:40 UTC, Steven Schveighoffer wrote: > > >>But the concept of what constitutes an "item" in a stream may not be > >>the "element type". That's what I'm getting at. > >> > >Hmm, I guess I'm not seeing it. Like, what even is an "item" in a > >stream? It sort of precludes that by definition, which is why we > >have to give it a type manually. What benefit is there to giving the > >buffer type separately from the window that gives you a typed slice > >into it? (I like that, btw.) > > An "item" in a stream may be a line of text, it may be a packet of > data, it may actually be a byte. But the compiler requires we type the > buffer as something rigid that it can work with. > > The elements of the stream are the basic fixed-sized units we use (the > array element type). The items are less concrete. [...] But array elements don't necessarily have to be fixed-sized, do they? For example, an array of lines can be string[] (or const(char)[][]). Of course, dealing with variable-sized items is messy, and probably rather annoying to implement. But it's *possible*, in theory. T -- People tell me that I'm paranoid, but they're just out to get me.
Re: Official compiler
On 18 February 2016 at 22:23, Jonathan M Davis via Digitalmars-d < digitalmars-d@puremagic.com> wrote: > On Thursday, 18 February 2016 at 20:28:41 UTC, David Nadlinger wrote: > >> On Thursday, 18 February 2016 at 17:56:32 UTC, Jonathan M Davis wrote: >> >>> […] if you want to be writing scripts in D (which is really useful), you >>> need rdmd, which means using dmd >>> >> >> You can use rdmd with ldmd2 just as well (and presumably gdmd too). >> > > Good to know. > > Clear only to somebody with x86-centric vision. I'm not claiming that the >> somewhat lower compile times aren't good for productivity. But being able >> to easily tap into the rich LLVM ecosystem or even just targeting the most >> widely used CPU architecture (in terms of units) is also something not to >> be forgotten when considering the development process. >> > > Having ldc is huge, but as long as you're targeting x86(_64) as one of > your platforms, developing with dmd is going to be faster thanks to the > fast compilation times. And if we can get dmd and ldc to be fully > compatible like they should be, then as long as your code is > cross-platform, it should be possible to develop it with dmd and then > target whatever you want with ldc - though obviously some stuff will have > to be done with ldc when it's something that dmd can't do (like a version > block targeting ARM), and anything that's going to ultimately be released > using ldc should be tested on it. But that fast compilation time is so > tremendous in the edit-test-edit cycle, that I just can't see using ldc as > the main compiler for development unless what you're doing isn't targeting > x86(_64) at all, or ldc isn't compatible enough with dmd to do most of the > development with dmd. > > But assuming that dmd and gdc/ldc are compatible, I would definitely argue > that the best way to do D development is to do most of the development with > dmd and then switch to gdc or ldc for production. That way, you get the > fast compilation times when you need it, and your final binary is better > optimized. > > - Jonathan M Davis > Actually, I'm sure this is a great way to let bugs in. There's no saying what could happen if you switch compiler and turn the optimisations throttle to full. In 99% of cases, one would hope all is good. But the bigger the codebase you're dealing with, the more you should really use both side by side when testing to ensure that no heisenbugs creep in.
Re: Official compiler
On Thu, 18 Feb 2016 21:39:45 +, Ola Fosheim Grøstad wrote: > On Thursday, 18 February 2016 at 21:30:29 UTC, Jonathan M Davis wrote: >> It's not a strawman. Walter has state previously that he's explicitly >> avoided looking at the source code for other compilers like gcc, >> because he doesn't want anyone to be able to accuse him of stealing >> code, copyright infringement, etc. > > Isn't this much more likely to happen if you don't look at the codebase > for other compilers? How do you know if someone submitting code isn't > just translating from GCC if you haven't looked at GCC? That's the exact opposite of true. With copyright, the fact that you created yours on your own is sufficient defense, assuming the courts agree. If by sheer coincidence you come up with code identical to what's in GCC, but you can show that you didn't take the code from GCC, you're in the clear. Patents, well, you're infringing even if you didn't refer to any other source. But if you did look at another source, especially if you looked in the patent database, you open yourself up to increased damages.
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 18:28:10 UTC, Alex Vincent wrote: Regarding control characters: If you give me a complete sample file, I can run it through Mozilla's UTF stream conversion and/or XML parsing code (via either SAX or DOMParser) to tell you how that reacts as a reference. Mozilla supports XML 1.0, but not 1.1. thanks you making the effort https://github.com/burner/std.xml2/blob/master/tests/eduni/xml-1.1/out/010.xml
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 20:24:18 UTC, H. S. Teoh wrote:
On Thu, Feb 18, 2016 at 07:25:16PM +, Jonathan M Davis via
Digitalmars-d wrote:
On Thursday, 18 February 2016 at 18:58:56 UTC, H. S. Teoh
wrote:
>On Thu, Feb 18, 2016 at 06:50:34PM +, Jonathan M Davis
>via Digitalmars-d wrote:
>>On Thursday, 18 February 2016 at 18:41:25 UTC, Steven
>>Schveighoffer wrote:
>[...]
>>>foo(void[] arr)
>>>{
>>> void[] arr2 = [1234, 5678, 91011];
>>> arr[] = arr2[0 .. arr.length];
>>>}
>>
>>Well, I'm not sure that that's actually not @safe.
>
>How can it possibly be @safe??? Consider:
>
>void main() @safe {
>Object[] objs = [ new Object() ];
>foo(objs);
>}
>
>Now the pointer to the Object instance has been corrupted.
Does it matter what state the void[] is in until you actually
attempt to cast it to something else? If you have
Object[] oArr = [ new Object ];
void[] vArr = oArr;
vArr = vArr[0 .. $ - 1];
it's not like that's going to cause any problems - not by
itself.
I think you misread Steven's code.
Yes. It looks like I did.
So the bottom line is that the array copy cannot be @safe.
Exactly. Passing it around is fine, but mutating it definitely is
not.
The larger question, is what, if anything, *can* you do with
void[] besides read-only operations, that doesn't break @safe?
Once something is (implicitly or otherwise) cast to void[], all
type information is forgotten, and there is no way, that I can
tell, to write to a void[] without causing @safe breakage. If
so, wouldn't it make sense to require that the type should be
const(void)[] rather than void[]?
Why? The problem isn't that void[] was passed it. It's that what
was done to it after it was passed in was not @safe. We need to
fix it so that the compiler doesn't consider mutating void[] or
casting away from it or doing anything with it that could corrupt
memory @safe, but passing it around is perfectly @safe, even if
it's not very useful by itself. So, I see no reason to make any
requirements about const. As long as dmd correctly catches the
operations that aren't @safe, the function which is passed the
void[] and does more than pass it around is going to be forced to
be @system anyway. So making any requirements about const(void[])
buys us nothing.
- Jonathan M Davis
Re: Official compiler
On Thursday, 18 February 2016 at 21:30:29 UTC, Jonathan M Davis wrote: It's not a strawman. Walter has state previously that he's explicitly avoided looking at the source code for other compilers like gcc, because he doesn't want anyone to be able to accuse him of stealing code, copyright infringement, etc. Isn't this much more likely to happen if you don't look at the codebase for other compilers? How do you know if someone submitting code isn't just translating from GCC if you haven't looked at GCC? If you have looked at GCC, then you can just choose a different implementation. :-) Anyway, the clean-virgin thing in programming is related to reverse engineering very small codebases where the implementation most likely is going to be very similar (like BIOS). So you have one team writing the spec and another team implementing the spec (with no communication between them).
Re: Another new io library
On 2/18/16 4:02 PM, H. S. Teoh via Digitalmars-d wrote: On Thu, Feb 18, 2016 at 03:20:58PM -0500, Steven Schveighoffer via Digitalmars-d wrote: On 2/18/16 2:53 PM, Wyatt wrote: On Thursday, 18 February 2016 at 18:35:40 UTC, Steven Schveighoffer wrote: But the concept of what constitutes an "item" in a stream may not be the "element type". That's what I'm getting at. Hmm, I guess I'm not seeing it. Like, what even is an "item" in a stream? It sort of precludes that by definition, which is why we have to give it a type manually. What benefit is there to giving the buffer type separately from the window that gives you a typed slice into it? (I like that, btw.) An "item" in a stream may be a line of text, it may be a packet of data, it may actually be a byte. But the compiler requires we type the buffer as something rigid that it can work with. The elements of the stream are the basic fixed-sized units we use (the array element type). The items are less concrete. [...] But array elements don't necessarily have to be fixed-sized, do they? For example, an array of lines can be string[] (or const(char)[][]). Of course, dealing with variable-sized items is messy, and probably rather annoying to implement. But it's *possible*, in theory. But the point of a stream is that it's contiguous data. A string[] has contiguous data that are pointers and lengths of a fixed size (sizeof(string) is fixed). This is not how you'd get data from a file or socket. Since this library doesn't discriminate what the data source provides (it will accept string[] as window type), it's possible. In this case, the element type might make sense as the range front type, but it's not a typical case. However, it might be interesting as, say, a message stream from one thread to another. -Steve
Re: Official compiler
On Thursday, 18 February 2016 at 20:24:31 UTC, David Nadlinger wrote: On Thursday, 18 February 2016 at 11:12:57 UTC, Jonathan M Davis wrote: And actually, he'd risk legal problems if he did, because he doesn't want anyone to be able to accuse him of taking code from gcc or llvm. That's a silly strawman, and you should know better than putting that forward as an argument by now. Walter is of course free to do whatever he pleases, and I would totally understand if his reason was just that it's hard to give something up you've worked on for a long time. But please don't make up argument trying to rationalize whatever personal decision somebody else made. You could literally copy LLVM source code into your application and sell it as a closed-source product without risking any copyright problems (if you comply with the very modest attribution clause of the license). It's not a strawman. Walter has state previously that he's explicitly avoided looking at the source code for other compilers like gcc, because he doesn't want anyone to be able to accuse him of stealing code, copyright infringement, etc. Now, that's obviously much more of a risk with gcc than llvm given their respective licenses, but it is a position that Walter has taken when the issue has come up, and it's not something that I'm making up. Now, if Walter were willing to give up on the dmd backend entirely, then presumably, that wouldn't be a problem anymore regardless of license issues, but he still has dmc, which uses the same backend, so I very much doubt that that's going to happen. - Jonathan M Davis
Re: Official compiler
On Thursday, 18 February 2016 at 20:28:41 UTC, David Nadlinger wrote: On Thursday, 18 February 2016 at 17:56:32 UTC, Jonathan M Davis wrote: […] if you want to be writing scripts in D (which is really useful), you need rdmd, which means using dmd You can use rdmd with ldmd2 just as well (and presumably gdmd too). Good to know. Clear only to somebody with x86-centric vision. I'm not claiming that the somewhat lower compile times aren't good for productivity. But being able to easily tap into the rich LLVM ecosystem or even just targeting the most widely used CPU architecture (in terms of units) is also something not to be forgotten when considering the development process. Having ldc is huge, but as long as you're targeting x86(_64) as one of your platforms, developing with dmd is going to be faster thanks to the fast compilation times. And if we can get dmd and ldc to be fully compatible like they should be, then as long as your code is cross-platform, it should be possible to develop it with dmd and then target whatever you want with ldc - though obviously some stuff will have to be done with ldc when it's something that dmd can't do (like a version block targeting ARM), and anything that's going to ultimately be released using ldc should be tested on it. But that fast compilation time is so tremendous in the edit-test-edit cycle, that I just can't see using ldc as the main compiler for development unless what you're doing isn't targeting x86(_64) at all, or ldc isn't compatible enough with dmd to do most of the development with dmd. But assuming that dmd and gdc/ldc are compatible, I would definitely argue that the best way to do D development is to do most of the development with dmd and then switch to gdc or ldc for production. That way, you get the fast compilation times when you need it, and your final binary is better optimized. - Jonathan M Davis
[WIP] Native SQLite Database reader (works at CTFE)
Hello, It is not quite ready to post in Announce, but I would like to inform you that I am planing to open-source my native-sqlite database driver. (well currently it just reads them). However it works fully at CTFE. so if you want to you can extract sourceCode out of a sql-database and make a mixin with it :D given you string imported the database file.
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 08:23:24PM +0100, Timon Gehr via Digitalmars-d wrote:
> On 18.02.2016 19:41, H. S. Teoh via Digitalmars-d wrote:
[...]
> > void breakSafety(void[] data) @safe
> > {
> > union U {
> > void[] d;
> > int[] arr;
> > }
> > U u;
> > u.d = data;
> > u.arr[0] = 0xFF; // kaboom
> > }
> >...
>
> That's in essence just a different way to cast void[] to int[].
> Steven's example disproves my previous claim though. Still, I don't
> think the conversion is the problem. It's the mutation of memory
> without type information.
Fair enough. But mutating memory *with* type information isn't
necessarily safer either:
void fun(int*[] intPtrs) {
union U {
int*[] ptrs;
long[] longs;
}
U u;
u.ptrs = intPtrs;
u.longs[0] = 12345; // oops
}
Basically, when it comes to @safe, one of the intrinsic requirements is
that arbitrary values must not be reinterpreted as pointers. In a sense,
it doesn't matter how you get there -- whether by implicit cast to
void[], or by unions, or whatever -- as soon as there is a way to
reinterpret an arbitrary value as a pointer, @safe is broken. The
arbitrary can be another pointer, and may even be a pointer to a type
with a compatible binary representation, but the point remains. For
example:
void safeFunc() @safe { ... }
void unsafeFunc() @system { ... }
union FuncPtrs {
void function() @safe fun1;
void function() @system fun2;
}
FuncPtrs ptrs;
ptrs.fun1 = &unsafeFunc();
ptrs.fun2();// oops
>From the POV of binary representation, both fun1 and fun2 are
essentially the same -- they are just function pointers: 32-bit or
64-bit addresses of some executable code. However, the breakage here is
caused by reinterpreting a pointer to a @system function as a pointer to
a @safe function. At no time do we lose type information, but as soon
as we have reintepretation of something as a pointer, @safe is out the
window.
[...]
> >>>2) To add salt to the wound, though, blatantly casting void[] to
> >>>T[] is allowed in @safe code, thus, readData() can even get away
> >>>with claiming to be @safe, when in fact it is anything but.
> >>>
> >>> https://issues.dlang.org/show_bug.cgi?id=15672
> >>
> >>This is the culprit.
> >
> >It's only one of many culprits. As long as there is any way around
> >@safe, the entire system of guarantees breaks down.
> >
> >
>
> If you want to verify guarantees, @safe should be specified by
> inclusion and ideally, type safety should be proved formally (after
> applying a few fixes).
> This first requires a formal language semantics.
> That's already a lot of effort, and after that you still don't have a
> verified implementation.
I'm not sure how feasible it is to do a formal proof with the current
dmd code, since you'd have to freeze it, and as soon as the next PR is
merged, the proof is potentially invalidated.
However, a good start is to make *everything* in the language @system by
default, and then have a whitelist of things that are @safe. Currently,
our implementation is to assume @safety by default and then add things
to a @system blacklist as we discover them. This is the wrong approach,
since we don't know what loopholes currently exist that can be
exploited, and the number of combinations of language features is far
too large for us to be ever sure that we've plugged all the holes. It's
better to start conservatively by assuming everything is @system, and as
people find more valid use cases that ought to be @safe, those can be
added to the whitelist after careful vetting. This is essentially the
idea behind:
https://issues.dlang.org/show_bug.cgi?id=15672
T
--
You have to expect the unexpected. -- RL
Re: Official compiler
On Thursday, 18 February 2016 at 10:42:33 UTC, Márcio Martins wrote: On Thursday, 18 February 2016 at 10:16:40 UTC, Radu wrote: [...] Walter doesn't have to give up working on DMD, right? Everyone could continue working on DMD, perhaps a few people could help on all three, I don't know... It's important if more people work on DMD and focus on polishing the frontend and language features, being the reference compiler, and used by all three compilers as well. What could potentially be important would be to backport key fixes/features from current frontend to LDC/GDC as well. As history tells, everything has to do with him moving D to more open waters. DMD is just a piece of that history and an excuse to keep working in a confort zone. It adds nothing strategically and it's just the last vestige of the old D world.
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 07:28:16PM +, Kagamin via Digitalmars-d wrote: > On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote: > >(*ahem*std.socket*cough*) liberally sprinkle @trusted on every function > >without regard to whether it's truly justified (e.g., see > >https://issues.dlang.org/show_bug.cgi?id=15672) > > How is bug 15672 related to std.socket? From quick glance at first > thousand lines of std.socket I don't see functions incorrectly marked > as trusted. Sorry, I pasted the wrong link. It should be: https://issues.dlang.org/show_bug.cgi?id=14137 T -- Some days you win; most days you lose.
Re: Official compiler
On 18.02.2016 21:24, David Nadlinger wrote: But please don't make up argument trying to rationalize whatever personal decision somebody else made. You could literally copy LLVM source code into your application and sell it as a closed-source product without risking any copyright problems (if you comply with the very modest attribution clause of the license). LLVM's license isn't the supposed problem. DMD's license is. You cannot copy DMD backend code to LLVM. By not contributing to other compilers, Walter stays in the clear in that regard. At least, that's how I understand the argument.
Re: Official compiler
On Thursday, 18 February 2016 at 17:56:32 UTC, Jonathan M Davis wrote: […] if you want to be writing scripts in D (which is really useful), you need rdmd, which means using dmd You can use rdmd with ldmd2 just as well (and presumably gdmd too). New users are frequently impressed by how fast dmd compiles code, and it's a big selling point for us. It's only later that benchmarking comes into play, and if want to do that, then use gdc or ldc. The download page already says to use gdc or ldc if you want better optimization. I'd claim that an equal number of people is put off by the sometimes abysmal performance of optimized DMD output in their initial tests and first toy projects. dmd is a clear winner as far as development goes. Clear only to somebody with x86-centric vision. I'm not claiming that the somewhat lower compile times aren't good for productivity. But being able to easily tap into the rich LLVM ecosystem or even just targeting the most widely used CPU architecture (in terms of units) is also something not to be forgotten when considering the development process. — David
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 07:25:16PM +, Jonathan M Davis via Digitalmars-d
wrote:
> On Thursday, 18 February 2016 at 18:58:56 UTC, H. S. Teoh wrote:
> >On Thu, Feb 18, 2016 at 06:50:34PM +, Jonathan M Davis via
> >Digitalmars-d wrote:
> >>On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer
> >>wrote:
> >[...]
> >>>foo(void[] arr)
> >>>{
> >>> void[] arr2 = [1234, 5678, 91011];
> >>> arr[] = arr2[0 .. arr.length];
> >>>}
> >>
> >>Well, I'm not sure that that's actually not @safe.
> >
> >How can it possibly be @safe??? Consider:
> >
> > void main() @safe {
> > Object[] objs = [ new Object() ];
> > foo(objs);
> > }
> >
> >Now the pointer to the Object instance has been corrupted.
>
> Does it matter what state the void[] is in until you actually attempt to
> cast it to something else? If you have
>
> Object[] oArr = [ new Object ];
> void[] vArr = oArr;
> vArr = vArr[0 .. $ - 1];
>
> it's not like that's going to cause any problems - not by itself.
I think you misread Steven's code. This has nothing to do with reducing
the length of an array. Read the code again. It has to do with
*overwriting* the contents of one void[] with another void[], which is a
legal and @safe operation under the current rules. Unfortunately, the
first void[] is an array of Object references, and the second void[] is
an array of ints, so this is overwriting pointers with arbitrary int
values.
Furthermore, as the caller of foo(), main() has no idea what has
happened to its Object[], because it continues to see the same area of
memory as Object[], even though it has been overwritten by an array of
ints. So when it next tries to dereference the Object[], which is
perfectly legal and does not involve any casting, it ends up
dereferencing an arbitrary int as a pointer instead. Ergo, @safe has
been compromised.
[...]
> >I think you missed the point of his example. :-) The point is that
> >it's perfectly legal to (1) cast an array of int to void[], and (2)
> >it's also perfectly legal to cast an array of anything to void[], and
> >(3) under current rules, it's perfectly legal to copy one void[] to
> >another void[].
> >
> >Arguably, (3) should not be allowed in @safe code. Which again brings
> >us back to the point, that if any function takes void[] as an
> >argument, is there *anything* it can do with the void[] other than
> >reading it, that *won't* break @safe?
> >
> >If there's *nothing* it can legally do with a void[] (other than
> >reading it) without violating @safe, then shouldn't it be a
> >requirement that all functions marked @safe must take const(void)[]
> >rather than void[]?
>
> It could pass it to something else without actually doing anything to
> it, in which case, I don't see why it couldn't be @safe. It's
> interpreting a void[] that's the problem. _That_ needs to be @system.
> Merely converting to void[] or passing it around is harmless.
No. Please look at Steven's code again. What it is doing, is (1)
implicitly casting Object[] to void[], which is perfectly legal
according to what you said. Then (2) it casts an int[] literal to
void[], again, another perfectly legal operation. Finally, (3) it copies
the contents of the second void[] to the first void[]. Since both arrays
are typed void[] at this point, the copy is perfectly legal (or so the
compiler thinks), and here is where the Object[] got overwritten by an
int[] and the compiler did not realize it.
Then back in main(), the code does not see void[] at all; all it sees is
the Object[] it has created all along. There is no interpretation of
void[] going on here at all. All that happened was (1) we passed
Object[] to void[], which is legal according to what you said, and (2)
we dereferenced the Object[], which is not suspect at all because, well,
it's an Object[], what could possibly go wrong? At this point, however,
we are dereferencing an illegal pointer, because foo() has corrupted our
Object[]. Note that *nowhere* in main() is there any reinterpretation of
void[] as Object[]. The only thing that led to this mess, as far as
main() is concerned, is that it passed an Object[] to a function that
takes void[].
In other words, from the caller's POV, as soon as you pass an array to a
function that takes void[], all bets are off, anything could happen to
your precious data, and @safety is no longer guaranteed. However, foo()
is marked @safe and the compiler does not complain. Also, inside foo()
we did nothing unseemly, according to current rules. All we did was to
convert two arrays to void[], which is perfectly legal and harmless, as
you claim, and copy data from one void[] to another void[], which, under
the current rules, is perfectly legal since we are just copying data
between two arrays *of the same type*. What could possibly go wrong with
copying data between two arrays of the same type?
The only problem is, they *aren't* of the same type after all. Just
because is(void[] == void[]) is tru
Re: Official compiler
On Thursday, 18 February 2016 at 11:12:57 UTC, Jonathan M Davis wrote: And actually, he'd risk legal problems if he did, because he doesn't want anyone to be able to accuse him of taking code from gcc or llvm. That's a silly strawman, and you should know better than putting that forward as an argument by now. Walter is of course free to do whatever he pleases, and I would totally understand if his reason was just that it's hard to give something up you've worked on for a long time. But please don't make up argument trying to rationalize whatever personal decision somebody else made. You could literally copy LLVM source code into your application and sell it as a closed-source product without risking any copyright problems (if you comply with the very modest attribution clause of the license). If anything, the problem is probably that the gdc and ldc folks could use more help, but dmd and Phobos suffer from that problem on some level as well, albeit probably not as acutely. The problem that many of us are seeing is that D development is unnecessarily defocussed by spreading out the effort between three different compilers. Of course, ideally we would have infinite manpower. A "special-case" compiler that boasts lower compile times for x86 development would definitely be nice to have then. But our resources aren't limitless, and as such the question whether we can afford to maintain such a "nice to have"-compiler is very relevant. Don't get me wrong, I understand that there is an argument to be made for the current situation. And, by the way, let me make very clear that even if I argue that sticking to DMD is a strategic mistake, this is not about personal things. I highly respect Walter as a compiler developer and like him as a person. But perpetuating ill-informed arguments really doesn't do this debate any good. — David
Re: Official compiler
On Thursday, 18 February 2016 at 15:36:42 UTC, Jonathan M Davis wrote: On Thursday, 18 February 2016 at 14:23:12 UTC, Márcio Martins wrote: I agree, but I don't see why this would have to change. It shouldn't change. Frontend development could happen on DMD as the *reference* compiler. And what exactly is the difference between the "official" compiler and the "reference" compiler supposed to be? A reference implementation is written to the spec in the simplest and clearest possible way so that it is bug free... It is not for production...
Re: Another new io library
On 2/18/16 2:53 PM, Wyatt wrote: On Thursday, 18 February 2016 at 18:35:40 UTC, Steven Schveighoffer wrote: But the concept of what constitutes an "item" in a stream may not be the "element type". That's what I'm getting at. Hmm, I guess I'm not seeing it. Like, what even is an "item" in a stream? It sort of precludes that by definition, which is why we have to give it a type manually. What benefit is there to giving the buffer type separately from the window that gives you a typed slice into it? (I like that, btw.) An "item" in a stream may be a line of text, it may be a packet of data, it may actually be a byte. But the compiler requires we type the buffer as something rigid that it can work with. The elements of the stream are the basic fixed-sized units we use (the array element type). The items are less concrete. And I think parsing/processing stream data works better by examining the buffer than shoehorning range functions in there. I think it's debatable. But part of stream semantics is being able to use it like a stream, and my BER toy was in that vein. Sorry again, this is probably not the place for it unless you try to replace the std.stream for real. I think stream semantics are what you should use. I haven't used std.stream, so I don't know what the API looks like. I assumed as! was something that returns a range of that type. Maybe I'm wrong? -Steve
[OT] Re: Official compiler
On Wednesday, 17 February 2016 at 22:57:20 UTC, Márcio Martins wrote: […] On a completely unrelated note, you aren't by any chance the Márcio Martins who is giving a talk at ETH in a couple of days, are you? — David
Re: Yet another leak in the sinking ship of @safe
On 2/18/16 2:25 PM, H. S. Teoh via Digitalmars-d wrote:
On Thu, Feb 18, 2016 at 01:58:24PM -0500, Steven Schveighoffer via
Digitalmars-d wrote:
On 2/18/16 11:37 AM, H. S. Teoh via Digitalmars-d wrote:
While @safe is a good idea in principle, the current implementation
is rather lackluster.
IMO, I think safe code should disallow casting that doesn't involve a
runtime function that verifies safety (such as casting an object to
another type, or invoking a @safe opCast), including casting array
types.
But isn't casting from, say, long[] to int[] @safe? Last time I checked,
druntime does actually convert .length correctly as well, so that
converting, say, int[] to long[] won't let you overrun the array by
accessing the last elements of the long[].
Probably. But there is always @trusted escapes. We are guaranteed
@safety if we just disallow casting whatsoever.
It doesn't sit well with me that casting is a standard mechanism to use
in @safe code. Casting is supposed to be a red flag. Even converting
between object types, I think casting is just a terrible requirement.
Maybe it's too restrictive, I don't know. It's unlikely to happen in any
case.
But obviously casting Object[] to long[] would be a very bad idea, and
should not be allowed in @safe. Scarily enough, this code currently
compiles without any complaint from the compiler:
void main() @safe
{
Object[] objs = [ new Object() ];
long[] longs = cast(long[]) objs;
longs[0] = 12345;
}
Yikes. Apparently you don't even need void[] to break @safe here. :-(
The difference here is that you have a cast. Casting to void[] doesn't
require one.
And implicit casts to void[] should be disallowed.
This would be a painful restriction, which is why I think it won't
happen :(
[...]
As far as I can tell, implicit casting to const(void)[] shouldn't be a
cause for concern -- if all you can do with it is to read the data
(e.g., a hexdump debugging function for printing out the byte
representation of something), there shouldn't be any problems. It's when
you write to the void[] that bad things begin to happen.
Possibly casting to const(T)[] should be OK for @safe code. But again, a
conservative approach that disallows casts is what I would do. Note that
you can't print out bytes from a void[], you would need a ubyte[].
-Steve
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 18:58:24 UTC, Steven
Schveighoffer wrote:
And implicit casts to void[] should be disallowed.
We could strike a sane medium: in a @safe function or when
calling a @safe function, implicit conversions of any array WITH
REFERENCES is not allowed, while other arrays may be
reinterpreted.
So you can still pass ubyte[], char[], long[], etc. to a @safe
void[] thing, but not Object[] or ubyte*[] or struct Foo {int *a
}[].
This should cover the vast majority of the sane use-cases for
Socket.receive, etc., anyway without being a problem.
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 19:28:16 UTC, Kagamin wrote: On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote: (*ahem*std.socket*cough*) liberally sprinkle @trusted on every function without regard to whether it's truly justified (e.g., see https://issues.dlang.org/show_bug.cgi?id=15672) How is bug 15672 related to std.socket? From quick glance at first thousand lines of std.socket I don't see functions incorrectly marked as trusted. The problem with std.socket is that calls to stuff like recv involve processing the data from void[], which is inherently unsafe, but the D functions calling them (e.g. receive) are marked as @trusted. It's conceptually similar to 15672 in that recv is taking the void* that it's given (from the void[]) and essentially converting it to ubyte* (or char*, since it's C), which is more or less equivalent to casting the void[] to ubyte[], which is what 15672 is about. However, even if 15672 is fixed, it wouldn't affect std.socket any, because it's the fact that a pointer to void[] is passed to a C function that converts it to ubyte*/char* that's the problem, not that void[] is converted to ubyte[] (since that never actually happens). - Jonathan M Davis
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 01:58:24PM -0500, Steven Schveighoffer via
Digitalmars-d wrote:
> On 2/18/16 11:37 AM, H. S. Teoh via Digitalmars-d wrote:
> >While @safe is a good idea in principle, the current implementation
> >is rather lackluster.
>
> IMO, I think safe code should disallow casting that doesn't involve a
> runtime function that verifies safety (such as casting an object to
> another type, or invoking a @safe opCast), including casting array
> types.
But isn't casting from, say, long[] to int[] @safe? Last time I checked,
druntime does actually convert .length correctly as well, so that
converting, say, int[] to long[] won't let you overrun the array by
accessing the last elements of the long[].
But obviously casting Object[] to long[] would be a very bad idea, and
should not be allowed in @safe. Scarily enough, this code currently
compiles without any complaint from the compiler:
void main() @safe
{
Object[] objs = [ new Object() ];
long[] longs = cast(long[]) objs;
longs[0] = 12345;
}
Yikes. Apparently you don't even need void[] to break @safe here. :-(
> And implicit casts to void[] should be disallowed.
>
> This would be a painful restriction, which is why I think it won't
> happen :(
[...]
As far as I can tell, implicit casting to const(void)[] shouldn't be a
cause for concern -- if all you can do with it is to read the data
(e.g., a hexdump debugging function for printing out the byte
representation of something), there shouldn't be any problems. It's when
you write to the void[] that bad things begin to happen.
T
--
Государство делает вид, что платит нам зарплату, а мы делаем вид, что работаем.
Re: Official compiler
On Thursday, 18 February 2016 at 11:41:26 UTC, Kai Nacke wrote: LLVM has about 2.5 million code lines. I am anything than sure if it is easy to improve compilation speed. I think you are a tad too pessimistic here. First, don't forget that there are some big LLVM customers for which low compile times are important too (remember all the buzz from when Clang first hit the scene?). Second, when was the last time you focussed on optimizing LDC -O0 compiler performance? There is currently a lot of low-hanging fruit, and then there are still the more involved options (such as making sure we use FastISel as much as possible). It might not end up quite as fast as DMD is right now. But imagine that Walter would have invested all the time he spent e.g. on implementing DWARF EH into optimizing the LDC frontend/glue layer/backend pass structure instead. Who knows, we might have an LDC-based compiler today that is faster than the DMD we currently have. — David
Re: Another new io library
On Thursday, 18 February 2016 at 18:35:40 UTC, Steven Schveighoffer wrote: On 2/18/16 12:08 PM, Wyatt wrote: I hadn't thought of this before, but if we accept that a stream is raw, untyped data, it may be best _not_ to provide a range interface directly. It's easy enough to alias source = sourceStream.as!ubyte; anyway, right? An iopipe is typed however you want it to be. Sorry, sorry, just thinking (too much?) in terms of the conceptual underpinnings. But I don't think we really disagree, either: if you don't give a stream a type it doesn't have one "naturally", so it's best to be explicit even if you're just asking for raw bytes. That's all I'm really saying there. But the concept of what constitutes an "item" in a stream may not be the "element type". That's what I'm getting at. Hmm, I guess I'm not seeing it. Like, what even is an "item" in a stream? It sort of precludes that by definition, which is why we have to give it a type manually. What benefit is there to giving the buffer type separately from the window that gives you a typed slice into it? (I like that, btw.) However, you have some issues there :) popFront doesn't return anything. Clearly, as!() returns the data! ;) But criminy, I do actually forget that ALL the damn time! (I blame Broadcom.) The worst part is I think I've even read the rationale for why it's like that and agreed with it with much nodding of the head and all that. :( And I think parsing/processing stream data works better by examining the buffer than shoehorning range functions in there. I think it's debatable. But part of stream semantics is being able to use it like a stream, and my BER toy was in that vein. Sorry again, this is probably not the place for it unless you try to replace the std.stream for real. -Wyatt
Re: Official compiler
On Thursday, 18 February 2016 at 17:52:10 UTC, Kai Nacke wrote: On Thursday, 18 February 2016 at 12:16:49 UTC, Radu wrote: As a casual user of the language I see that there is a fragmentation of resources and a waste in this regard with people developing in mainline, then some of you LDC guys catching up. As Iain already pointed out the main problem is (undocumented or weird) AST changes. This makes a merge sometimes painful. This can (and will) go better. This is IMHO the only "waste". Nobody of the LDC team does frontend development, we are all focused on the glue layer. My simple assumption is that if presumably the dmd backend is not maintained anymore, a lot of the core dmd people can focus on improving whatever problems the frontend or glue layers have. As far as I know only Walter (and Daniel I think) work on the backend. This is not "a lot of the core dmd people". This could only mean that you core LDC guys could focus on llvm backend optimizations (both code gen and performance related). I'm going to assume that those kind of performance optimizations are also constantly done by upstream llvm, so more win here. By chance I am an LLVM committer, too. But the LDC team only focuses on getting the glue library and the runtime library right. Adding new useful optimizations is hard work. The people working on it are either researchers or backed by a big company. Users will not magically turn to contributors if their perception is that there is always going to be a catch-up game to play somewhere. Not to mention that if one want's to get something in LDC, one has to commit it in mainline, which is DMD, you just multiplied the know-how someone needs to have to do some useful work... It depends on the feature you want. If you want a new language feature then yes. But then you do not change LDC, you change the language specification and therefore the reference compiler. You can add a lot of features without ever touching DMD frontend code. The sanitizers, for example. Or the not-yet-merged PR for profile-guided optimizations. And finally, just pointing people to ldc/gdc (always a version or 2 behind, another grief) each time dmd performance is poor, looks awfully wrong. I really find this "speed" argument doubtful. My experience is that if you really need performance you must *know* what you are doing. Just picking some code from a web site, compiling it and then complaining that the resulting binary is slower than that of language xy is not a serious approach. For a novice user, LDC can be discouraging: just type ldc2 -help-hidden. But you may need to know about these options to e.g. enable the right auto-vectorizer for your problem. I once wrote an MD5 implementation in pure Java which was substantially faster than the reference implementation in C from RFC 1321 (gcc -O3 compiled). C is not faster than Java if you know Java but not C. The same is true for D. I really like the compiler diversity. What I miss (hint!) is a program to verify the compiler/backend correctness. Just generate a random D program, compile with all 3 compilers and compare the output. IMHO we could find a lot of backend bugs this way. This would help all D compilers. Regards, Kai I think there are more involve in DMD in general, you need to count reviewers and all the infrastructure deployed. But even if only 2 of them are involved having them 100% focused on core D stuff would be a boon. I see a trend with this discussion: 1. Compiler speed. This is a clear win for DMD, but at the same time LDC doesn't benefit from consistent investment on performance tuning. This obviously is just speculation, but I think the performance gap can be substantially closed with more resources invested here, at least for un-optimized builds. 2. Speed of compiled code. People often suggest that DMD could close the gap here, but I see this as wishful thinking, just listing all the optimizations LLVM does it's just depressing for anyone wanting to move DMD closer to that, it is just game over in this regard. Plus, who is going to work on them except Walter? Does anyone want to invest in a dubious licensed backend? But the story is more complicated :) We are talking here about perception, LLVM is a well known and respectable backend, this is a win for people using or wanting to contribute to the language. Also, people forget that DMD is limited on numbers of architecture supported. My hope is that LDC will be on the same announcement page when a new DMD version is launched. When this will happen, common sense will just kill DMD. Apreciate all the hard work you all guys do!
Re: Official compiler
On Thursday, 18 February 2016 at 12:23:18 UTC, Jonathan M Davis wrote: if the dmd backend were dropped, […] that would further slow down the development of the frontend and not necessarily improve things overall. How would that be? — David
Re: Yet another leak in the sinking ship of @safe
Well, ok, getprotobyname indeed looks thread unsafe for some reason, but this can happen for any function.
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote: (*ahem*std.socket*cough*) liberally sprinkle @trusted on every function without regard to whether it's truly justified (e.g., see https://issues.dlang.org/show_bug.cgi?id=15672) How is bug 15672 related to std.socket? From quick glance at first thousand lines of std.socket I don't see functions incorrectly marked as trusted.
Re: Yet another leak in the sinking ship of @safe
On 18.02.2016 20:17, Era Scarecrow wrote:
On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
On 2/18/16 1:30 PM, Timon Gehr wrote:
No problem here. There is no way to assign to a void[] without doing 2.
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
Since void throws away type information (and all the safety related to
it), would it be easier to simply require @safe code can't cast
implicitly to void? It seems like explicit casting would take care of
most of this, or disallowing to/from void converting period while in
@safe code.
The conversion is fine. It just throws away type information. There's no
way to soundly type-check the block assignment after that, so that's the
operation that should be disallowed. This also prevents @safe code from
mutating untyped memory that was passed through from @system code
without preventing it to pass the block back to @trusted code.
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 18:58:56 UTC, H. S. Teoh wrote:
On Thu, Feb 18, 2016 at 06:50:34PM +, Jonathan M Davis via
Digitalmars-d wrote:
On Thursday, 18 February 2016 at 18:41:25 UTC, Steven
Schveighoffer wrote:
[...]
>foo(void[] arr)
>{
> void[] arr2 = [1234, 5678, 91011];
> arr[] = arr2[0 .. arr.length];
>}
Well, I'm not sure that that's actually not @safe.
How can it possibly be @safe??? Consider:
void main() @safe {
Object[] objs = [ new Object() ];
foo(objs);
}
Now the pointer to the Object instance has been corrupted.
Does it matter what state the void[] is in until you actually
attempt to cast it to something else? If you have
Object[] oArr = [ new Object ];
void[] vArr = oArr;
vArr = vArr[0 .. $ - 1];
it's not like that's going to cause any problems - not by itself.
As soon as you convert it back to Object[] or to int[] or
whatever, _then_ you're screwed. But converting from void[] to
anything is inherently @system regardless of what you did to the
array or where it came from. Now, given that reducing the length
like that is setting you up for a bad conversion, I can certainly
see an argument for reducing the length of a void[] being
@system, but in and of itself, it isn't going to corrupt memory.
It just sets the stage for it. As long as it stays void[],
everything is fine.
Now given that you can't possibly do anything useful with
reducing the length of void[] and that it does make it so that
future, @system operations would corrupt memory, I can certainly
see making it @system, but I dispute that it's really doing
anything unsafe on its own.
It's trying to interpret the void[] that's the problem.
Certainly, you can convert T[] to void[] and pass it around
all day without risking any memory corruption, so that should
definitely be @safe, and I don't see how reducing the length
of a void[] could actually cause memory corruption on its own.
It's when you cast the void[] to something else that you risk
things going south, and that's what needs to be @system. So,
I'm not sure that there's actually any reason for your example
code to not be @safe.
[...]
I think you missed the point of his example. :-) The point is
that it's
perfectly legal to (1) cast an array of int to void[], and (2)
it's also
perfectly legal to cast an array of anything to void[], and (3)
under
current rules, it's perfectly legal to copy one void[] to
another
void[].
Arguably, (3) should not be allowed in @safe code. Which again
brings us back to the point, that if any function takes void[]
as an argument, is there *anything* it can do with the void[]
other than reading it, that *won't* break @safe?
If there's *nothing* it can legally do with a void[] (other
than reading it) without violating @safe, then shouldn't it be
a requirement that all functions marked @safe must take
const(void)[] rather than void[]?
It could pass it to something else without actually doing
anything to it, in which case, I don't see why it couldn't be
@safe. It's interpreting a void[] that's the problem. _That_
needs to be @system. Merely converting to void[] or passing it
around is harmless. And as long as converting void[] to anything
else is considered @system like it should be, I don't see why it
would be necessary to care about whether a function accepts
void[] or not when considering @safety. Useless as it may be
void[] identity(void[] arr) @safe
{
return arr;
}
is perfectly @safe. If the compiler does its job, and the
function does cast the void[], then the function will be
considered @system unless it's marked @trusted, and all is well
without caring one whit about how the function was passed void[].
The problem with std.socket is not that it accepts void[] and
considers that @safe; it's that it passes that void[] to
something else, treating that as @safe, when it's not. And since
it's a C function that's being called, and it accesses the ptr
member of the void[], it's already @system, forcing @trusted to
be used to make the compiler happy. So, the problem is that
@trusted was used when it shouldn't have been, not that the type
system allowed void[] to be passed in without marking it as
@system. And since the function was marked @trusted, it's not
like having the compiler treat a function that accepted void[] as
@system would have helped anyway.
- Jonathan M Davis
Re: Yet another leak in the sinking ship of @safe
On 18.02.2016 19:41, H. S. Teoh via Digitalmars-d wrote:
On Thu, Feb 18, 2016 at 07:30:34PM +0100, Timon Gehr via Digitalmars-d wrote:
On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
[...]
1) Casting an array of elements with indirections (in this case
Object[]) to void[] is questionable at best, outright unsafe at
worst, as shown here. Even if we were to rewrite readData() and mark
it @trusted, it still raises the question of what a @trusted function
can legally do with void[], which is essentially a type-erased array,
that justifies being tagged as @trusted. How can a function do
anything that doesn't break @safety if all type information about the
array has been erased, and it essentially sees it only as a ubyte[]?
I'm inclined to say that @trusted functions should only be allowed to
receive const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
No problem here. There is no way to assign to a void[] without doing 2.
Sure there is:
void breakSafety(void[] data) @safe
{
union U {
void[] d;
int[] arr;
}
U u;
u.d = data;
u.arr[0] = 0xFF; // kaboom
}
...
That's in essence just a different way to cast void[] to int[]. Steven's
example disproves my previous claim though. Still, I don't think the
conversion is the problem. It's the mutation of memory without type
information.
2) To add salt to the wound, though, blatantly casting void[] to T[]
is allowed in @safe code, thus, readData() can even get away with
claiming to be @safe, when in fact it is anything but.
https://issues.dlang.org/show_bug.cgi?id=15672
This is the culprit.
It's only one of many culprits. As long as there is any way around
@safe, the entire system of guarantees breaks down.
If you want to verify guarantees, @safe should be specified by inclusion
and ideally, type safety should be proved formally (after applying a few
fixes). This first requires a formal language semantics.
That's already a lot of effort, and after that you still don't have a
verified implementation.
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 18:41:25 UTC, Steven
Schveighoffer wrote:
On 2/18/16 1:30 PM, Timon Gehr wrote:
No problem here. There is no way to assign to a void[] without
doing 2.
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
Since void throws away type information (and all the safety
related to it), would it be easier to simply require @safe code
can't cast implicitly to void? It seems like explicit casting
would take care of most of this, or disallowing to/from void
converting period while in @safe code.
To be honest, I think there's only 1 time I actually used a
void[] in my code, and that was while writing a section in the
BitArray replacement code years back in the case you wanted to
use/read another block of data as the source for the BitArray.
Beyond that I never touched it.
Re: Official compiler
On Thursday, 18 February 2016 at 17:52:10 UTC, Kai Nacke wrote: I really like the compiler diversity. What I miss (hint!) is a program to verify the compiler/backend correctness. Just generate a random D program, compile with all 3 compilers and compare the output. IMHO we could find a lot of backend bugs this way. This would help all D compilers. Regards, Kai reminds me of csmith https://embed.cs.utah.edu/csmith/ I believe Brian Schott had worked on something like this for D... Did that ever go anywhere?
Re: Yet another leak in the sinking ship of @safe
On 2/18/16 1:50 PM, Jonathan M Davis wrote:
On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
Well, I'm not sure that that's actually not @safe. It's trying to
interpret the void[] that's the problem. Certainly, you can convert T[]
to void[] and pass it around all day without risking any memory
corruption, so that should definitely be @safe, and I don't see how
reducing the length of a void[] could actually cause memory corruption
on its own. It's when you cast the void[] to something else that you
risk things going south, and that's what needs to be @system. So, I'm
not sure that there's actually any reason for your example code to not
be @safe.
You don't need to cast to use the original.
void bar() @safe
{
int *[] ptrs = new int*[1];
foo(ptrs);
*ptrs[0] = 5;
}
-Steve
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 06:55:18PM +, Jonathan M Davis via Digitalmars-d wrote: > On Thursday, 18 February 2016 at 18:41:58 UTC, H. S. Teoh wrote: [...] > >It's only one of many culprits. As long as there is any way around > >@safe, the entire system of guarantees breaks down. > > Of course, and we went about things the wrong way with @safe. It > should have been done via whitelisting, whereas we've done it via > blacklisting. Given that fact, we're pretty much permanently at risk > of @safe being broken. [...] https://issues.dlang.org/show_bug.cgi?id=12941 T -- Why are you blatanly misspelling "blatant"? -- Branden Robinson
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 06:50:34PM +, Jonathan M Davis via Digitalmars-d
wrote:
> On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
[...]
> >foo(void[] arr)
> >{
> > void[] arr2 = [1234, 5678, 91011];
> > arr[] = arr2[0 .. arr.length];
> >}
>
> Well, I'm not sure that that's actually not @safe.
How can it possibly be @safe??? Consider:
void main() @safe {
Object[] objs = [ new Object() ];
foo(objs);
}
Now the pointer to the Object instance has been corrupted.
> It's trying to interpret the void[] that's the problem. Certainly, you
> can convert T[] to void[] and pass it around all day without risking
> any memory corruption, so that should definitely be @safe, and I don't
> see how reducing the length of a void[] could actually cause memory
> corruption on its own. It's when you cast the void[] to something else
> that you risk things going south, and that's what needs to be @system.
> So, I'm not sure that there's actually any reason for your example
> code to not be @safe.
[...]
I think you missed the point of his example. :-) The point is that it's
perfectly legal to (1) cast an array of int to void[], and (2) it's also
perfectly legal to cast an array of anything to void[], and (3) under
current rules, it's perfectly legal to copy one void[] to another
void[].
Arguably, (3) should not be allowed in @safe code. Which again brings us
back to the point, that if any function takes void[] as an argument, is
there *anything* it can do with the void[] other than reading it, that
*won't* break @safe?
If there's *nothing* it can legally do with a void[] (other than reading
it) without violating @safe, then shouldn't it be a requirement that all
functions marked @safe must take const(void)[] rather than void[]?
T
--
Chance favours the prepared mind. -- Louis Pasteur
Re: Yet another leak in the sinking ship of @safe
On 2/18/16 11:37 AM, H. S. Teoh via Digitalmars-d wrote: While @safe is a good idea in principle, the current implementation is rather lackluster. IMO, I think safe code should disallow casting that doesn't involve a runtime function that verifies safety (such as casting an object to another type, or invoking a @safe opCast), including casting array types. And implicit casts to void[] should be disallowed. This would be a painful restriction, which is why I think it won't happen :( -Steve
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 18:41:58 UTC, H. S. Teoh wrote:
On Thu, Feb 18, 2016 at 07:30:34PM +0100, Timon Gehr via
Digitalmars-d wrote:
No problem here. There is no way to assign to a void[] without
doing 2.
Sure there is:
void breakSafety(void[] data) @safe
{
union U {
void[] d;
int[] arr;
}
U u;
u.d = data;
u.arr[0] = 0xFF; // kaboom
}
Well, unions with an array in them can't be @safe. That's clearly
a bug, regardless of whether void[] is involved or not.
Regardless, as far as I can tell, there is zero @safety problem
with converting to void[]. You'll never get corrupted memory with
that conversion. It's converting back that risks screwing
everything up. And that's what can't be @safe.
It's only one of many culprits. As long as there is any way
around @safe, the entire system of guarantees breaks down.
Of course, and we went about things the wrong way with @safe. It
should have been done via whitelisting, whereas we've done it via
blacklisting. Given that fact, we're pretty much permanently at
risk of @safe being broken.
- Jonathan M Davis
Re: Yet another leak in the sinking ship of @safe
On Thursday, 18 February 2016 at 18:41:25 UTC, Steven
Schveighoffer wrote:
On 2/18/16 1:30 PM, Timon Gehr wrote:
On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
While @safe is a good idea in principle, the current
implementation is
rather lackluster. Consider, for example:
void readData(void[] buffer) @safe
{
ubyte[] buf = cast(ubyte[]) buffer;
buf[0] = 0xFF;
}
void main() @safe
{
auto buffer = new Object[1];
readData(buffer);
}
There are (at least) two major problems here:
1) Casting an array of elements with indirections (in this
case
Object[]) to void[] is questionable at best, outright unsafe
at worst,
as shown here. Even if we were to rewrite readData() and mark
it
@trusted, it still raises the question of what a @trusted
function can
legally do with void[], which is essentially a type-erased
array, that
justifies being tagged as @trusted. How can a function do
anything that
doesn't break @safety if all type information about the array
has been
erased, and it essentially sees it only as a ubyte[]? I'm
inclined to
say that @trusted functions should only be allowed to receive
const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
No problem here. There is no way to assign to a void[] without
doing 2.
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
Well, I'm not sure that that's actually not @safe. It's trying to
interpret the void[] that's the problem. Certainly, you can
convert T[] to void[] and pass it around all day without risking
any memory corruption, so that should definitely be @safe, and I
don't see how reducing the length of a void[] could actually
cause memory corruption on its own. It's when you cast the void[]
to something else that you risk things going south, and that's
what needs to be @system. So, I'm not sure that there's actually
any reason for your example code to not be @safe.
- Jonathan M Davis
Re: Yet another leak in the sinking ship of @safe
On 18.02.2016 19:41, Steven Schveighoffer wrote:
On 2/18/16 1:30 PM, Timon Gehr wrote:
On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
While @safe is a good idea in principle, the current implementation is
rather lackluster. Consider, for example:
void readData(void[] buffer) @safe
{
ubyte[] buf = cast(ubyte[]) buffer;
buf[0] = 0xFF;
}
void main() @safe
{
auto buffer = new Object[1];
readData(buffer);
}
There are (at least) two major problems here:
1) Casting an array of elements with indirections (in this case
Object[]) to void[] is questionable at best, outright unsafe at worst,
as shown here. Even if we were to rewrite readData() and mark it
@trusted, it still raises the question of what a @trusted function can
legally do with void[], which is essentially a type-erased array, that
justifies being tagged as @trusted. How can a function do anything that
doesn't break @safety if all type information about the array has been
erased, and it essentially sees it only as a ubyte[]? I'm inclined to
say that @trusted functions should only be allowed to receive
const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
No problem here. There is no way to assign to a void[] without doing 2.
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
-Steve
Good point. :)
Re: Yet another leak in the sinking ship of @safe
On Thu, Feb 18, 2016 at 07:30:34PM +0100, Timon Gehr via Digitalmars-d wrote:
> On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
[...]
> >1) Casting an array of elements with indirections (in this case
> >Object[]) to void[] is questionable at best, outright unsafe at
> >worst, as shown here. Even if we were to rewrite readData() and mark
> >it @trusted, it still raises the question of what a @trusted function
> >can legally do with void[], which is essentially a type-erased array,
> >that justifies being tagged as @trusted. How can a function do
> >anything that doesn't break @safety if all type information about the
> >array has been erased, and it essentially sees it only as a ubyte[]?
> >I'm inclined to say that @trusted functions should only be allowed to
> >receive const(void)[] as parameter, not void[].
> >
> > https://issues.dlang.org/show_bug.cgi?id=15702
> >
>
> No problem here. There is no way to assign to a void[] without doing 2.
Sure there is:
void breakSafety(void[] data) @safe
{
union U {
void[] d;
int[] arr;
}
U u;
u.d = data;
u.arr[0] = 0xFF; // kaboom
}
> >2) To add salt to the wound, though, blatantly casting void[] to T[]
> >is allowed in @safe code, thus, readData() can even get away with
> >claiming to be @safe, when in fact it is anything but.
> >
> > https://issues.dlang.org/show_bug.cgi?id=15672
>
> This is the culprit.
It's only one of many culprits. As long as there is any way around
@safe, the entire system of guarantees breaks down.
T
--
Question authority. Don't ask why, just do it.
Re: Yet another leak in the sinking ship of @safe
On 2/18/16 1:30 PM, Timon Gehr wrote:
On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
While @safe is a good idea in principle, the current implementation is
rather lackluster. Consider, for example:
void readData(void[] buffer) @safe
{
ubyte[] buf = cast(ubyte[]) buffer;
buf[0] = 0xFF;
}
void main() @safe
{
auto buffer = new Object[1];
readData(buffer);
}
There are (at least) two major problems here:
1) Casting an array of elements with indirections (in this case
Object[]) to void[] is questionable at best, outright unsafe at worst,
as shown here. Even if we were to rewrite readData() and mark it
@trusted, it still raises the question of what a @trusted function can
legally do with void[], which is essentially a type-erased array, that
justifies being tagged as @trusted. How can a function do anything that
doesn't break @safety if all type information about the array has been
erased, and it essentially sees it only as a ubyte[]? I'm inclined to
say that @trusted functions should only be allowed to receive
const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
No problem here. There is no way to assign to a void[] without doing 2.
foo(void[] arr)
{
void[] arr2 = [1234, 5678, 91011];
arr[] = arr2[0 .. arr.length];
}
-Steve
Re: Another new io library
On 2/18/16 12:08 PM, Wyatt wrote:
On Thursday, 18 February 2016 at 15:44:00 UTC, Steven Schveighoffer wrote:
On 2/17/16 5:54 AM, John Colvin wrote:
On Wednesday, 17 February 2016 at 07:15:01 UTC, Steven Schveighoffer
wrote:
On 2/17/16 1:58 AM, Rikki Cattermole wrote:
A few things:
https://github.com/schveiguy/iopipe/blob/master/source/iopipe/traits.d#L126
why isn't that used more especially with e.g. window?
After all, window seems like a very well used word...
Not sure what you mean.
I don't like that a stream isn't inherently an input range.
This seems to me like a good place to use this abstraction by default.
What is front for an input stream? A byte? A character? A word? A line?
Why not just say it's a ubyte and then compose with ranges from there?
If I provide a range by element (it may not be ubyte), then that's
likely not the most useful range to have.
I hadn't thought of this before, but if we accept that a stream is raw,
untyped data, it may be best _not_ to provide a range interface
directly. It's easy enough to
alias source = sourceStream.as!ubyte;
anyway, right?
An iopipe is typed however you want it to be.
bufferedInput by default uses an ArrayBuffer!ubyte. You can have it use
any type of buffer you want, it doesn't discriminate. The only
requirement is that the buffer's window is a random-access range
(although I'm having thoughts that I should just require it to be an array).
But the concept of what constitutes an "item" in a stream may not be the
"element type". That's what I'm getting at.
This is why I think it's better to have the user specifically tell me
"this is how I want to range-ify this stream" rather than assume.
I think this makes more sense with TLV encodings, too. Thinking of
things like:
switch(source.as!(BERType).popFront){
case(UNIVERSAL|PRIMITIVE|UTF8STRING){
int len;
if(source.as!(BERLength).front & 0b10_00_00_00) {
// X.690? Never heard of 'em!
} else {
len = source.as!(BERLength).popFront;
}
return source.buffered(len).as!(string).popFront;
}
...etc.
}
Very cool looking!
However, you have some issues there :) popFront doesn't return anything.
And I think parsing/processing stream data works better by examining the
buffer than shoehorning range functions in there.
-Steve
Re: Yet another leak in the sinking ship of @safe
On 18.02.2016 17:37, H. S. Teoh via Digitalmars-d wrote:
While @safe is a good idea in principle, the current implementation is
rather lackluster. Consider, for example:
void readData(void[] buffer) @safe
{
ubyte[] buf = cast(ubyte[]) buffer;
buf[0] = 0xFF;
}
void main() @safe
{
auto buffer = new Object[1];
readData(buffer);
}
There are (at least) two major problems here:
1) Casting an array of elements with indirections (in this case
Object[]) to void[] is questionable at best, outright unsafe at worst,
as shown here. Even if we were to rewrite readData() and mark it
@trusted, it still raises the question of what a @trusted function can
legally do with void[], which is essentially a type-erased array, that
justifies being tagged as @trusted. How can a function do anything that
doesn't break @safety if all type information about the array has been
erased, and it essentially sees it only as a ubyte[]? I'm inclined to
say that @trusted functions should only be allowed to receive
const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
No problem here. There is no way to assign to a void[] without doing 2.
2) To add salt to the wound, though, blatantly casting void[] to T[] is
allowed in @safe code, thus, readData() can even get away with claiming
to be @safe, when in fact it is anything but.
https://issues.dlang.org/show_bug.cgi?id=15672
This is the culprit.
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 17:26:30 UTC, Adam D. Ruppe wrote: On Thursday, 18 February 2016 at 16:56:08 UTC, Robert burner Schadek wrote: unix file says it is a utf8 encoded file, but not BOM is present. the hex dump is "3C 66 6F 6F 3E C2 80 3C 2F 66 6F 6F 3E" Gah, I should have read this before replying... well, that does appear to be valid utf-8 why is it throwing an exception then? I'm pretty sure that byte stream *is* actually well-formed xml 1.0 and should pass utf validation as well as the XML well-formedness check. Regarding control characters: If you give me a complete sample file, I can run it through Mozilla's UTF stream conversion and/or XML parsing code (via either SAX or DOMParser) to tell you how that reacts as a reference. Mozilla supports XML 1.0, but not 1.1.
Re: Another new io library
On 2/18/16 12:16 PM, Wyatt wrote: On Thursday, 18 February 2016 at 16:36:37 UTC, Steven Schveighoffer wrote: Note, asInputRange may not do what you want here. If multiple zmqPollItems come in at once (I'm not sure how your socket works), the input range's front will provide the entire window of data, and flush it on popFront. Not so great! That's really not what I'd expect at all. :( (This isn't to say it doesn't make sense semantically, but I don't like how it feels.) The philosophy that I settled on is to create an iopipe that extends one "item" at a time, even if more are available. Then, apply the range interface on that. When I first started to write byLine, I made it a range. Then I thought, "what if you wanted to iterate by 2 lines at a time, or iterate by one line at a time, but see the last 2 for context?", well, then that would be another type, and I'd have to abstract out the functionality of line searching. So I decided to just make an abstract "asInputRange" and just wrap the functionality of extending data one line at a time. The idea is to make building blocks as simple and useful as possible. So what I think may be a good fit for your application (without knowing all the details) is to create an iopipe that delineates each message and extends exactly one message per call to extend. Then, you can wrap that in asInputRange, or create your own range which translates the actual binary data to a nicer object for each call to front. So something like: foreach(pollItem; zmqSocket.bufferedInput .byZmqPacket .asInputRange) I'm still not 100% sure that this is the right way to do it... Hm... if asInputRange took a template parameter of what type it should return, then asInputRange!zmqPacket could return zmqPacket(pipe.window) for front. That's kind of nice. I'm thinking I'll change the name byInputRange to byWindow, and add a byElement for an element-wise input range. Oh, I see. Naming. Naming is hard. Yes. It's especially hard when you haven't seen how others react to it :) -Steve
Re: Official compiler
On Thursday, 18 February 2016 at 15:58:14 UTC, Nick Sabalausky wrote: but for the second group, a language that's fast like C/C++ but not nearly as unproductive IS appealing, and even seems to be something they're often on the lookout for. I would agree with you if you could write D code using the most convenient style possible, compile using LDC, and get the same speed as C or C++. My experience suggests that is not going to happen. You're going to need some knowledge of the language to get the best performance.
Re: Head Const
On 18.02.2016 13:03, Walter Bright wrote: On 2/18/2016 2:47 AM, Timon Gehr wrote: On 18.02.2016 10:24, Walter Bright wrote: On 2/17/2016 11:58 PM, Timon Gehr wrote: const(headconst(T)) is the same as const(T), no? Er, yes? Jonathan wanted to embed a mutable reference count within a const object. Contrary to your suggestion, headconst won't help with that. Of course it will: headconst pointer -> pointer -> refcount -> const pointer -> ... constant stuff ... He wanted to embed a mutable reference count literally within a const object. Not a headconst object.
Re: OT: Need help with translation D-faq to other languages
On Wednesday, 17 February 2016 at 18:28:08 UTC, Suliman wrote: Hello, I wrote pretty good FAQ about D on Russian language. I need help with translation it to English and Germany and possible to other languages. http://dlang.ru/Why-D-is-Better I hope it people will like it, it will help to attract more people and help noobs better understand what D is. It's pretty short. I can do a German translation
Re: Vibe.d Copyright
On Thursday, 18 February 2016 at 16:14:09 UTC, Chris wrote: Just to say that the copyright notice on the vibe.d website should be updated. In the API it still says "Copyright © 2012-2015 RejectedSoftware e.K." In the license it still says "Copyright (c) 2012-2014, rejectedsoftware e.K." and RejectedSoftware is spelled differently. vibe.d bug reports belong here: https://github.com/rejectedsoftware/vibe.d/issues
Re: Official compiler
On Thursday, 18 February 2016 at 17:56:32 UTC, Jonathan M Davis wrote: Honestly, I think that dmd _should_ be the goto compiler. [snip] I agree with your response. That being said, it can't hurt to make things a bit more clear for new users. If you go to the download page, there is a more information button that takes you to the wiki. But the wiki page seems to just look like raw html. It doesn't look nearly as good as the page we are coming from. Assuming that is fixed, I would recommend two small other changes: 1) Put the first line from the wiki where it says "If you're a beginner DMD is the recommended choice, ..." on the top of the compiler page, 2) Replace the GDC and LDC "Strong optimization" lines in the download page with something a little clearer. Even "Stronger optimization than DMD" would be clearer.
Re: std.xml2 (collecting features)
On Thursday, 18 February 2016 at 10:18:18 UTC, Robert burner Schadek wrote: If you want to on some XML stuff, please join me. It is properly more productive working together than creating two competing implementations. Oh, I absolutely agree, independent implementation is a bad thing. (Someone should rename DRY as "don't repeat yourself or others"... but DRYOO sounds weird.) Where's your repo?
Re: Official compiler
On Thursday, 18 February 2016 at 17:52:10 UTC, Kai Nacke wrote: I really like the compiler diversity. What I miss (hint!) is a program to verify the compiler/backend correctness. Just generate a random D program, compile with all 3 compilers and compare the output. IMHO we could find a lot of backend bugs this way. This would help all D compilers. That would really be cool. - Jonathan M Davis
Re: Official compiler
On Thursday, 18 February 2016 at 16:47:16 UTC, Márcio Martins wrote: On Thursday, 18 February 2016 at 15:36:42 UTC, Jonathan M Davis wrote: On Thursday, 18 February 2016 at 14:23:12 UTC, Márcio Martins wrote: I agree, but I don't see why this would have to change. It shouldn't change. Frontend development could happen on DMD as the *reference* compiler. And what exactly is the difference between the "official" compiler and the "reference" compiler supposed to be? - Jonathan M Davis "official" carries a connotation of endorsement doesn't it? In other words, if you are given a choice of 3 and you know very little about each, which would you predict would give you a better user experience? Reference in this case is the one that most closely follows the bleeding edge of the language spec, which new users don't benefit a lot from. In this case it's also where all the frontend development would happen. But what we call it this doesn't really matter to end users. What I have been defending this far is that we could entertain the possibility that end users could be better off if we "suggested" they try out one of the other compilers before they try DMD. The easiest way to suggest that is to stamp "official" on one of the stronger alternatives. Once installers for LDC and GDC are on par with DMD, is there still a pragmatic reason to suggest DMD to new users? Given that all that DMD has going for it from the perspective of a new user is the compilation speed? For everyone else nothing would change, we'd go about our daily lives, using our favorite compiler as always. But meanwhile, people exploring and looking to try D could try out it's amazing features and get proof in first-hand, that these awesome features come at no efficiency cost, as advertised. Honestly, I think that dmd _should_ be the goto compiler. It's the fast one. It's the most up-to-date - especially right now as gdc and ldc have been trying to get to the point that they're using the new D frontend instead of the C++ one. gdc and ldc are great if you want to make sure that you're code is faster in production, but they're slower for actually get the code written, and AFAIK, if you want to be writing scripts in D (which is really useful), you need rdmd, which means using dmd (and I sure wouldn't want those to be compiled with gdc or ldc anyway - compilation speed matters way more in that case than it even does during development). New users are frequently impressed by how fast dmd compiles code, and it's a big selling point for us. It's only later that benchmarking comes into play, and if want to do that, then use gdc or ldc. The download page already says to use gdc or ldc if you want better optimization. I wouldn't want to use gdc or ldc for normal development unless I had to, and I wouldn't want to encourage others to either. dmd's speed is worth way too much when it comes to getting actual work done. And it's not like it generates slow code. It just doesn't generate code that's as fast as gdc or ldc generates, and when you get to the point that you need the fast binary, then use gdc or ldc. But use them as the default? Why? dmd is a clear winner as far as development goes. It's both faster and more up-to-date. It's just that gdc or ldc would be better for production code if you really need all the speed that you can get. We need to work towards getting and keeping gdc and ldc in sync with dmd so that they stop being behind like they typically are, and we do need to make sure that it's clear that gdc and ldc generate faster binaries. But I think that it would be a huge mistake to push either of them as the one that everyone should be using by default. - Jonathan M Davis
Re: Official compiler
On Thursday, 18 February 2016 at 17:23:09 UTC, rsw0x wrote: On Thursday, 18 February 2016 at 11:41:26 UTC, Kai Nacke wrote: On Thursday, 18 February 2016 at 10:45:54 UTC, Márcio Martins wrote: I suppose it's a lot easier to address the compilation speed issue in LDC/GDC, than to improve and maintain DMD's backend to the expected levels, right? LLVM has about 2.5 million code lines. I am anything than sure if it is easy to improve compilation speed. Regards, Kai Sorry for being off topic, Rustc(uses LLVM) has a parallel codegen compilation mode that decreases optimization for a (major AFAIK?) decrease in compilation time when compiling multiple files. Would it be possible for LDC to offer the same thing without a major rewrite? I'm unfamiliar with the LDC codebase which is why I ask. Probably worth noting that even with parallel codegen rustc is still far slower than ldc. reference: https://internals.rust-lang.org/t/default-settings-for-parallel-codegen/519 From time to time I dream about compiling modules in parallel. :-) This needs some investigation but I think it could be possible to spawn a thread per module you are compiling (after the frontend passes). Never digged deeper into this... Regards, Kai
Re: Official compiler
On Thursday, 18 February 2016 at 12:16:49 UTC, Radu wrote: As a casual user of the language I see that there is a fragmentation of resources and a waste in this regard with people developing in mainline, then some of you LDC guys catching up. As Iain already pointed out the main problem is (undocumented or weird) AST changes. This makes a merge sometimes painful. This can (and will) go better. This is IMHO the only "waste". Nobody of the LDC team does frontend development, we are all focused on the glue layer. My simple assumption is that if presumably the dmd backend is not maintained anymore, a lot of the core dmd people can focus on improving whatever problems the frontend or glue layers have. As far as I know only Walter (and Daniel I think) work on the backend. This is not "a lot of the core dmd people". This could only mean that you core LDC guys could focus on llvm backend optimizations (both code gen and performance related). I'm going to assume that those kind of performance optimizations are also constantly done by upstream llvm, so more win here. By chance I am an LLVM committer, too. But the LDC team only focuses on getting the glue library and the runtime library right. Adding new useful optimizations is hard work. The people working on it are either researchers or backed by a big company. Users will not magically turn to contributors if their perception is that there is always going to be a catch-up game to play somewhere. Not to mention that if one want's to get something in LDC, one has to commit it in mainline, which is DMD, you just multiplied the know-how someone needs to have to do some useful work... It depends on the feature you want. If you want a new language feature then yes. But then you do not change LDC, you change the language specification and therefore the reference compiler. You can add a lot of features without ever touching DMD frontend code. The sanitizers, for example. Or the not-yet-merged PR for profile-guided optimizations. And finally, just pointing people to ldc/gdc (always a version or 2 behind, another grief) each time dmd performance is poor, looks awfully wrong. I really find this "speed" argument doubtful. My experience is that if you really need performance you must *know* what you are doing. Just picking some code from a web site, compiling it and then complaining that the resulting binary is slower than that of language xy is not a serious approach. For a novice user, LDC can be discouraging: just type ldc2 -help-hidden. But you may need to know about these options to e.g. enable the right auto-vectorizer for your problem. I once wrote an MD5 implementation in pure Java which was substantially faster than the reference implementation in C from RFC 1321 (gcc -O3 compiled). C is not faster than Java if you know Java but not C. The same is true for D. I really like the compiler diversity. What I miss (hint!) is a program to verify the compiler/backend correctness. Just generate a random D program, compile with all 3 compilers and compare the output. IMHO we could find a lot of backend bugs this way. This would help all D compilers. Regards, Kai
Re: Any actively maintained qt bindings for D?
On Wednesday, 17 February 2016 at 21:20:09 UTC, Jeremy DeHaan wrote: On Wednesday, 17 February 2016 at 20:56:27 UTC, Rishub Nagpal wrote: Qtd hasn't been updated in 3 years Does anyone know of anactively maintained qt library? I think QML bindings exists, but as far as I know direct Qt bindings don't exist/aren't updated. I want to do a GSoC proposal for direct bindings to Qt, though. I have a lot of ideas for doing a binding to Qt proper, but the only way I can do it is if I have time. Time holds us all back :P
Re: Poor memory allocation performance with a lot of threads on 36 core machine
On Thu, 18 Feb 2016 13:00:12 +, Witek wrote: > So, the question is, why is D / DMD allocator so slow under heavy > multithreading? It's a global GC, possibly with a little per-thread pool. As part of the abortive Amber language project, I was looking into ways to craft per-thread GC. You need to tell the runtime whether a variable is marked as shared or __gshared and that's pretty much sufficient -- you can only refer to unshared variables from one thread, which means you can do a local collection stopping only one thread's execution. You can have one heap for each thread and one cross-thread heap. This work hasn't happened in D yet. I would like to look into D's GC and parallelism more. I've started on mark/sweep parallelism but haven't made any worthwhile progress. I'll take this as my next task. It's more awkward because it requires changes to the runtime interface, which means modifying both DMD and the runtime.
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 16:56:08 UTC, Robert burner Schadek wrote: unix file says it is a utf8 encoded file, but not BOM is present. the hex dump is "3C 66 6F 6F 3E C2 80 3C 2F 66 6F 6F 3E" Gah, I should have read this before replying... well, that does appear to be valid utf-8 why is it throwing an exception then? I'm pretty sure that byte stream *is* actually well-formed xml 1.0 and should pass utf validation as well as the XML well-formedness check.
Re: Official compiler
On Thursday, 18 February 2016 at 11:41:26 UTC, Kai Nacke wrote: On Thursday, 18 February 2016 at 10:45:54 UTC, Márcio Martins wrote: I suppose it's a lot easier to address the compilation speed issue in LDC/GDC, than to improve and maintain DMD's backend to the expected levels, right? LLVM has about 2.5 million code lines. I am anything than sure if it is easy to improve compilation speed. Regards, Kai Sorry for being off topic, Rustc(uses LLVM) has a parallel codegen compilation mode that decreases optimization for a (major AFAIK?) decrease in compilation time when compiling multiple files. Would it be possible for LDC to offer the same thing without a major rewrite? I'm unfamiliar with the LDC codebase which is why I ask. Probably worth noting that even with parallel codegen rustc is still far slower than ldc. reference: https://internals.rust-lang.org/t/default-settings-for-parallel-codegen/519
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 16:54:10 UTC, Robert burner Schadek wrote: It does not, it has no prolog and therefore no EncodingInfo. In that case, it needs to be valid UTF-8 or valid UTF-16 and it is a fatal error if there's any invalid bytes: https://www.w3.org/TR/REC-xml/#charencoding == It is a fatal error if an XML entity is determined (via default, encoding declaration, or higher-level protocol) to be in a certain encoding but contains byte sequences that are not legal in that encoding. Specifically, it is a fatal error if an entity encoded in UTF-8 contains any ill-formed code unit sequences, as defined in section 3.9 of Unicode [Unicode]. Unless an encoding is determined by a higher-level protocol, it is also a fatal error if an XML entity contains no encoding declaration and its content is not legal UTF-8 or UTF-16. ==
Re: Another new io library
On Thursday, 18 February 2016 at 16:36:37 UTC, Steven Schveighoffer wrote: On 2/18/16 11:07 AM, Wyatt wrote: This looks pretty all-right so far. Would something like this work? foreach(pollItem; zmqSocket.bufferedInput .as!(zmqPollItem) .asInputRange) Yes, that is the intent. All without copying. Great! Note, asInputRange may not do what you want here. If multiple zmqPollItems come in at once (I'm not sure how your socket works), the input range's front will provide the entire window of data, and flush it on popFront. Not so great! That's really not what I'd expect at all. :( (This isn't to say it doesn't make sense semantically, but I don't like how it feels.) I'm thinking I'll change the name byInputRange to byWindow, and add a byElement for an element-wise input range. Oh, I see. Naming. Naming is hard. -Wyatt
Re: Another new io library
On Thursday, 18 February 2016 at 15:44:00 UTC, Steven
Schveighoffer wrote:
On 2/17/16 5:54 AM, John Colvin wrote:
On Wednesday, 17 February 2016 at 07:15:01 UTC, Steven
Schveighoffer wrote:
On 2/17/16 1:58 AM, Rikki Cattermole wrote:
A few things:
https://github.com/schveiguy/iopipe/blob/master/source/iopipe/traits.d#L126
why isn't that used more especially with e.g. window?
After all, window seems like a very well used word...
Not sure what you mean.
I don't like that a stream isn't inherently an input range.
This seems to me like a good place to use this abstraction
by default.
What is front for an input stream? A byte? A character? A
word? A line?
Why not just say it's a ubyte and then compose with ranges
from there?
If I provide a range by element (it may not be ubyte), then
that's likely not the most useful range to have.
I hadn't thought of this before, but if we accept that a stream
is raw, untyped data, it may be best _not_ to provide a range
interface directly. It's easy enough to
alias source = sourceStream.as!ubyte;
anyway, right?
This is why I think it's better to have the user specifically
tell me "this is how I want to range-ify this stream" rather
than assume.
I think this makes more sense with TLV encodings, too. Thinking
of things like:
switch(source.as!(BERType).popFront){
case(UNIVERSAL|PRIMITIVE|UTF8STRING){
int len;
if(source.as!(BERLength).front & 0b10_00_00_00) {
// X.690? Never heard of 'em!
} else {
len = source.as!(BERLength).popFront;
}
return source.buffered(len).as!(string).popFront;
}
...etc.
}
Musing: I'd probably want a helper like popAs!() so I don't
forget popFront()...
-Wyatt
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 16:54:10 UTC, Robert burner Schadek wrote: unix file says it is a utf8 encoded file, but not BOM is present. the hex dump is "3C 66 6F 6F 3E C2 80 3C 2F 66 6F 6F 3E"
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 16:47:35 UTC, Adam D. Ruppe wrote: On Thursday, 18 February 2016 at 16:41:52 UTC, Robert burner Schadek wrote: for instance, quick often I find <80> in tests that are supposed to be valid xml 1.0. they are invalid xml 1.1 though What char encoding does the document declare itself as? It does not, it has no prolog and therefore no EncodingInfo. unix file says it is a utf8 encoded file, but not BOM is present.
Re: std.xml2 (collecting features) control character
On Thursday, 18 February 2016 at 16:41:52 UTC, Robert burner Schadek wrote: for instance, quick often I find <80> in tests that are supposed to be valid xml 1.0. they are invalid xml 1.1 though What char encoding does the document declare itself as?
Re: Official compiler
On Thursday, 18 February 2016 at 15:36:42 UTC, Jonathan M Davis wrote: On Thursday, 18 February 2016 at 14:23:12 UTC, Márcio Martins wrote: I agree, but I don't see why this would have to change. It shouldn't change. Frontend development could happen on DMD as the *reference* compiler. And what exactly is the difference between the "official" compiler and the "reference" compiler supposed to be? - Jonathan M Davis "official" carries a connotation of endorsement doesn't it? In other words, if you are given a choice of 3 and you know very little about each, which would you predict would give you a better user experience? Reference in this case is the one that most closely follows the bleeding edge of the language spec, which new users don't benefit a lot from. In this case it's also where all the frontend development would happen. But what we call it this doesn't really matter to end users. What I have been defending this far is that we could entertain the possibility that end users could be better off if we "suggested" they try out one of the other compilers before they try DMD. The easiest way to suggest that is to stamp "official" on one of the stronger alternatives. Once installers for LDC and GDC are on par with DMD, is there still a pragmatic reason to suggest DMD to new users? Given that all that DMD has going for it from the perspective of a new user is the compilation speed? For everyone else nothing would change, we'd go about our daily lives, using our favorite compiler as always. But meanwhile, people exploring and looking to try D could try out it's amazing features and get proof in first-hand, that these awesome features come at no efficiency cost, as advertised.
Re: std.xml2 (collecting features) control character
for instance, quick often I find <80> in tests that are supposed to be valid xml 1.0. they are invalid xml 1.1 though
Yet another leak in the sinking ship of @safe
While @safe is a good idea in principle, the current implementation is
rather lackluster. Consider, for example:
void readData(void[] buffer) @safe
{
ubyte[] buf = cast(ubyte[]) buffer;
buf[0] = 0xFF;
}
void main() @safe
{
auto buffer = new Object[1];
readData(buffer);
}
There are (at least) two major problems here:
1) Casting an array of elements with indirections (in this case
Object[]) to void[] is questionable at best, outright unsafe at worst,
as shown here. Even if we were to rewrite readData() and mark it
@trusted, it still raises the question of what a @trusted function can
legally do with void[], which is essentially a type-erased array, that
justifies being tagged as @trusted. How can a function do anything that
doesn't break @safety if all type information about the array has been
erased, and it essentially sees it only as a ubyte[]? I'm inclined to
say that @trusted functions should only be allowed to receive
const(void)[] as parameter, not void[].
https://issues.dlang.org/show_bug.cgi?id=15702
2) To add salt to the wound, though, blatantly casting void[] to T[] is
allowed in @safe code, thus, readData() can even get away with claiming
to be @safe, when in fact it is anything but.
https://issues.dlang.org/show_bug.cgi?id=15672
These two issues basically make the @safe annotation an empty promise,
because *anything* could happen entirely within the realm of @safe code,
and memory safety isn't guaranteed at all. It's bad enough that some
Phobos modules (*ahem*std.socket*cough*) liberally sprinkle @trusted on
every function without regard to whether it's truly justified (e.g., see
https://issues.dlang.org/show_bug.cgi?id=15672), now even if we exclude
@trusted functions from the mix, we can still break @safe without the
compiler even raising an eyebrow.
Sadly, the above issues are merely the tip of the iceberg. There are
many more holes in @safe, such as unions containing pointers, unions of
@safe and @system delegates, unaligned pointers, void initialization of
members with indirection, just to name a few. If we want the @safe ship
to float, we have a lot of work cut out for us.
--T
