Destroy All Memory Corruption
I'll be doing a reprise of my DConf 2020 talk on Destroy All Memory Corruption on April 21, 2021 at 7PM PST. https://nwcpp.org/ Except this time it'll be live, not prerecorded. All are welcome!
Re: Silicon Valley D Meetup - April 15, 2021 - "Compile Time Function Execution (CTFE)"
On Monday, 19 April 2021 at 09:06:17 UTC, FeepingCreature wrote: Right, I agree with all of this. I just think the way to get to it is to first allow everything, and then in a second step pare it down to something that does what people need while also being monitorable. This is as "simple" as merging every IO call the program does into its state count. I am not saying it is wrong for a new language to try out this philosophy, assuming you are willing to go through a series of major revisions of the language. But I think for a language where "breaking changes" is a made a big deal of, you want to stay conservative. In that regard, I agree it would be social, as in, if you clearly state upfront that your new language will come in major versions with major breakage at regular intervals then you should have more freedom to explore and end up with something much better. Which is not necessarily a deal breaker if you also support stable versions with some clear time window, like "version 2 is supported until 2030". So, yeah, it is possible. But you have to teach the programmers of your philosophy and make them understand that some versions of the language has a long support window and other versions are more short-lived. Maybe make it clear in the versioning naming-scheme perhaps. D's problem is that there is only one stable version, and that is the most recent one... that makes changes more difficult. Also, there are not enough users to get significant experience with "experimental features". What works for C++ when providing experimental features, might not work for smaller languages. - or if not, if D can make it work. At any rate, with a lot of features like implicit conversions, I think people would find that they're harmless and highly useful if they'd just try them for a while. A lot of features are harmless on a small scale. Python is a very flexible language and you can do a lot of stuff in it that you should not do. Despite this it works very well on a small scale. However, for Python to work on a larger scale it takes a lot of discipline (social constraints in place of technical constraints) and carefully chosen libraries etc. The use context matters, when discussing what is acceptable and what isn't. As such, people might have different views on language features and there might be no right/wrong solution. Implicit conversions is good for custom ADTs, but the interaction with overloading can be problematic, so it takes a lot of foresight to get it right. A geometry library can benefit greatly from implicit conversions, but you can run into problems when mixing libraries that overuse implicit conversions... So, it isn't only a question of whether implicit conversions is a bad thing or not, but how the language limits "chains of conversion" and overloads and makes it easy to predict for the programmer when looking a piece of code.
Re: Silicon Valley D Meetup - April 15, 2021 - "Compile Time Function Execution (CTFE)"
On Monday, 19 April 2021 at 08:46:05 UTC, Ola Fosheim Grøstad wrote: I think the downsides are conceptual and technical, not social. If you can implement a version counter then you get all kinds of problems, like first compilation succeeding, then the second compilation failing with no code changes. Also, you can no longer cache intermediate representations between compilations without a rather significant additional machinery. It is better to do this in a more functional way, so you can generate a file, but it isn't written to disk, it is an abstract entity during compilation and is turned into something concrete after compilation. So, anything that can be deduced from the input is fair game, but allowing arbitrary I/O is a completely different beast, compilation has to be idempotent. It should not be possible to write a program where the first compilation succeeds and the second compilation fails with no code changes between the compilation executions. Such failures should be limited to the build system so that you can quickly correct the problem. IMHO, a good productive language makes debugging easier, faster and less frequently needed. Anything that goes against that is a move in the wrong direction. Right, I agree with all of this. I just think the way to get to it is to first allow everything, and then in a second step pare it down to something that does what people need while also being monitorable. This is as "simple" as merging every IO call the program does into its state count. (For my lang, I just go and hash every part of the program during compilation. So caching works on the basis of what actually goes into the binary.) But my view is, you have those issues anyways! You need to run a caching dub server, and you need to run it on a Docker image, and you need to pretty much mirror every upstream package that your Docker install pulls in, *anyways.* You can't escape the vagaries of the build environment changing under you by limiting the language. And everything you keep out of the language - IO, downloads, arbitrary commands whose output is relevant - the rest of your build environment usually does regardless. (And it will mess you up at the first, slightest provocation. Bloody Ubuntu and its breaking changes in patch releases...) So my view is the other way around - make the language the single point of contact for *all of that stuff*, make CTFE powerful enough to hash static libraries and process header files live during compilation, so you can pull as much of the complexity as possible into the controlled environment of the compiler. And then when you know what you need there, take a step back and frameworkize it, so you can do change detection inside your single build system. You can't get the entire OS on board, but you can maybe get all or most of your language library developers on board. Anyways, even agreeing that you're right, "look, we tried it and it didn't work, in fact it was a disaster, see discussions here and here, or download version 2019-03-06 nightly to try how it went" is just inherently a stronger argument. to find out what works and what doesn't, and you can't gather experience with what people actually want to do and how it works in practice if you lock things down from the start. In That's ok for a prototype, but not for a production language. I think most long-term successful languages straddle a line here. For instance, looking at Rust, the use of nightly and stable channels allows the language to experiment while also keeping a guarantee that once it commits to a feature enough to merge it into stable, it won't change "overnight". D is trying to do a similar thing with DIPs and preview flags and deprecations, but the jury's still out on how well it's working - or if not, if D can make it work. At any rate, with a lot of features like implicit conversions, I think people would find that they're harmless and highly useful if they'd just try them for a while.
Re: Silicon Valley D Meetup - April 15, 2021 - "Compile Time Function Execution (CTFE)"
On Monday, 19 April 2021 at 06:37:03 UTC, FeepingCreature wrote: This is a social issue more than a technical one. The framework can help, by limiting access to disk and URLs and allowing tracing and hijacking, but ultimately you have to rely on code to not do crazy things. I think the downsides are conceptual and technical, not social. If you can implement a version counter then you get all kinds of problems, like first compilation succeeding, then the second compilation failing with no code changes. Also, you can no longer cache intermediate representations between compilations without a rather significant additional machinery. It is better to do this in a more functional way, so you can generate a file, but it isn't written to disk, it is an abstract entity during compilation and is turned into something concrete after compilation. to find out what works and what doesn't, and you can't gather experience with what people actually want to do and how it works in practice if you lock things down from the start. In That's ok for a prototype, but not for a production language. Most of my annoyances with D are issues where D isn't willing to take an additional step even though it would be technically very feasible. No implicit conversion for user-defined types, no arbitrary IO calls in CTFE, no returning AST trees from CTFE functions that are automatically inserted to create macros, and of course the cumbersome specialcased metaprogramming for type inspection instead of just letting us pass a type object to a ctfe function and calling methods on it. So, anything that can be deduced from the input is fair game, but allowing arbitrary I/O is a completely different beast, compilation has to be idempotent. It should not be possible to write a program where the first compilation succeeds and the second compilation fails with no code changes between the compilation executions. Such failures should be limited to the build system so that you can quickly correct the problem. IMHO, a good productive language makes debugging easier, faster and less frequently needed. Anything that goes against that is a move in the wrong direction.
Re: Silicon Valley D Meetup - April 15, 2021 - "Compile Time Function Execution (CTFE)"
On Sunday, 18 April 2021 at 04:41:44 UTC, Ola Fosheim Grostad wrote: On Sunday, 18 April 2021 at 00:38:13 UTC, Ali Çehreli wrote: I heard about safety issues around allowing full I/O during compilation but then the following points kind of convinced me: - If I am compiling a program, my goal is to execute that program anyway. What difference does it make whether the program's compilation is harmful vs. the program itself. I dont buy this, you can execute the code in a sandbox. Compilation should be idempotent, writing to disk/databases during compilation breaks this guarantee. I would not use a language that does not ensure this. This is a social issue more than a technical one. The framework can help, by limiting access to disk and URLs and allowing tracing and hijacking, but ultimately you have to rely on code to not do crazy things. And right now in D we just push this complexity out of the language and into the build system, because if you don't let people do crazy things, they just do crazy and also horribly hacky things instead. (*Cough* gen_ut_main *cough*.) In my opinion, the design approach should be to "default open, then restrict", rather than "default closed, then relax." This requires a certain willingness to break people's workflow, but if you default closed, you'll never get over your preconceptions, because you have to be able to do crazy things to find out what works and what doesn't, and you can't gather experience with what people actually want to do and how it works in practice if you lock things down from the start. In other words, I see no reason why "make one to throw it away" shouldn't apply to languages. Maybe we will decide one day to limit recursion for templates and CTFE, for instance, but if we do, it will be because of the experiences we gathered with unrestricted templates and the impact on compile times - if D had decided from day one to keep templates limited, we'd never have the wealth of experience and frameworks and selling points that D has with metaprogramming. You have to let people show you what they want to do and what comes of it, and you can't do that if you don't extend them enough rope to tangle themselves and their projects up first. There has to be a willingness to try and fail and backtrack. Most of my annoyances with D are issues where D isn't willing to take an additional step even though it would be technically very feasible. No implicit conversion for user-defined types, no arbitrary IO calls in CTFE, no returning AST trees from CTFE functions that are automatically inserted to create macros, and of course the cumbersome specialcased metaprogramming for type inspection instead of just letting us pass a type object to a ctfe function and calling methods on it. If some new language will overtake D (cx¹, fingers crossed!), it will be because of this design conservatism as much as any technological restriction in the frontend. ¹ I have a language, by the way! :) https://github.com/FeepingCreature/cx , but it's pre-pre-alpha. Native CTFE and macros are a beautiful thing though.
