Re: Flatten a range of static arrays
On 2/8/20 1:20 PM, ag0aep6g wrote: On 08.02.20 15:57, Steven Schveighoffer wrote: This kind of stuff is so difficult to reason about and develop as a library that people will just end up removing dip1000 from their compilation. I 100% agree that DIP 1000 is hard to reason about. It's pretty limited by design, and the implementation has so many bugs. If anyone has a better design (and implementation), I'd be all for that. I unfortunately don't. I suspect that making scope a type constructor might help, but I don't know enough about how dip1000 works to know for sure. I like the idea of dip1000 inferring most things. But the end result of joiner just getting inferred @system is so difficult to figure out why. I think before dip1000 is the default, we need some way to say, I expect inference to infer these calls as safe. And it should tell you why it can't. About just ditching the compiler switch: Then you can't even take the address of a local. Also, it's going to become the default eventually. Then the user will just switch to @trusted. I mean to have to figure out what joiner is actually doing, in conjunction with map, like you did, is so convoluted, nobody is going to spend that time. The easier thing is to give up and switch to system where you have to. I don't feel dip1000 is ready for the default mode until we have better ways to diagnose its issues, and concrete responses to things that really should be safe but aren't. -Steve
Re: Flatten a range of static arrays
On 08.02.20 07:27, ag0aep6g wrote:
On 08.02.20 02:38, ag0aep6g wrote:
Simplified, we're looking at this:
struct Joiner
{
int[3] _items;
int[] _current;
}
void main() @safe
{
Joiner j;
j._current = j._items[];
}
[...]
In the first reduction, `j` might be `scope`, but `j._items` has no
indirections. `scope` doesn't actually mean anything for it. It's just
an `int[3]` on the stack. So taking its address could be allowed.
But (the current implementation of) DIP 1000 is apparently too
conservative for that. It seems to treat pointers into a struct the same
as pointers to the whole thing.
My attempt at lifting this limitation:
https://github.com/dlang/dmd/pull/10773
Re: Flatten a range of static arrays
On 08.02.20 15:57, Steven Schveighoffer wrote: This kind of stuff is so difficult to reason about and develop as a library that people will just end up removing dip1000 from their compilation. I 100% agree that DIP 1000 is hard to reason about. It's pretty limited by design, and the implementation has so many bugs. If anyone has a better design (and implementation), I'd be all for that. About just ditching the compiler switch: Then you can't even take the address of a local. Also, it's going to become the default eventually.
Re: Flatten a range of static arrays
On 2/7/20 8:38 PM, ag0aep6g wrote:
If that code were allowed, you could do this:
struct Joiner
{
Joiner* p;
}
Joiner g;
void main() @safe
{
scope Joiner j;
() @trusted { j.p = } (); /* pretend it's allowed */
g = *j.p; /* dereference and copy */
}
Returning a copy of a dereferenced `scope` pointer is always allowed,
because `scope` only provides one level of protection.
Again, the limitations of dip1000 are apparent. Why can't I mark p as
only pointing at scope data? And in which cases do I need to do this (in
the true Joiner code)?
This kind of stuff is so difficult to reason about and develop as a
library that people will just end up removing dip1000 from their
compilation.
-Steve
Re: Flatten a range of static arrays
On 08.02.20 02:38, ag0aep6g wrote:
Simplified, we're looking at this:
struct Joiner
{
int[3] _items;
int[] _current;
}
void main() @safe
{
Joiner j;
j._current = j._items[];
}
I.e., a self-referential struct. Or most fundamentally:
struct Joiner
{
Joiner* p;
}
void main() @safe
{
Joiner j;
j.p = /* error */
}
`dmd -dip1000` complains about the marked line:
Error: reference to local variable j assigned to non-scope j.p
What if I mark `j` as `scope`? Then I should be able to assign a
reference-to-local to `j.p`. Indeed, the error goes away, but another
takes its place:
Error: cannot take address of scope local j in @safe function main
Right. That can't be allowed, because `scope` gives only one level of
protection, and `` would need two (one for `j` itself and one for the
thing it points at).
I went a bit overboard with that second reduction. The original struct
is self-referential, but it's not recursive. The errors are the same for
the first reduction. But it's not as clear that they're necessary.
In the first reduction, `j` might be `scope`, but `j._items` has no
indirections. `scope` doesn't actually mean anything for it. It's just
an `int[3]` on the stack. So taking its address could be allowed.
But (the current implementation of) DIP 1000 is apparently too
conservative for that. It seems to treat pointers into a struct the same
as pointers to the whole thing.
Re: Flatten a range of static arrays
On 08.02.20 01:17, Steven Schveighoffer wrote:
The original code is not invalid though. f is not valid, and that is a
bug, but the original code posted by nullptr should be fine by memory
safety standards.
Maybe. But then it should also work with `int[3] front;`.
If there is no possible way to do the original code with dip1000
attributes, then that's a bug with dip1000's design (would not be
surprised).
Or DIP 1000 just doesn't allow that kind of code. @safe (including DIP
1000) is not supposed to allow all de-facto safe programs.
Simplified, we're looking at this:
struct Joiner
{
int[3] _items;
int[] _current;
}
void main() @safe
{
Joiner j;
j._current = j._items[];
}
I.e., a self-referential struct. Or most fundamentally:
struct Joiner
{
Joiner* p;
}
void main() @safe
{
Joiner j;
j.p = /* error */
}
`dmd -dip1000` complains about the marked line:
Error: reference to local variable j assigned to non-scope j.p
What if I mark `j` as `scope`? Then I should be able to assign a
reference-to-local to `j.p`. Indeed, the error goes away, but another
takes its place:
Error: cannot take address of scope local j in @safe function main
Right. That can't be allowed, because `scope` gives only one level of
protection, and `` would need two (one for `j` itself and one for the
thing it points at).
If that code were allowed, you could do this:
struct Joiner
{
Joiner* p;
}
Joiner g;
void main() @safe
{
scope Joiner j;
() @trusted { j.p = } (); /* pretend it's allowed */
g = *j.p; /* dereference and copy */
}
Returning a copy of a dereferenced `scope` pointer is always allowed,
because `scope` only provides one level of protection.
Re: Flatten a range of static arrays
On 2/7/20 6:30 PM, ag0aep6g wrote:
On 08.02.20 00:10, nullptr wrote:
```
import std;
struct SomeRange
{
int[3] val;
enum empty = false;
auto popFront() @safe {}
ref auto front() @safe
{
return val;
}
}
void main() @safe
{
SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
}
```
I don't know how applicable this is to your use case, but this code
will compile and run under -dip1000.
That shouldn't compile. You have found a hole in DIP 1000.
struct SomeRange
{
int[3] val = [10, 20, 30];
ref auto front() @safe { return val; }
}
int[] f() @safe
{
SomeRange sr;
// return sr.val[]; /* error, as expected */
return sr.front[]; /* no error, but escapes reference to local */
}
void main() @safe
{
auto x = f();
import std.stdio;
writeln(x); /* Prints garbage. */
}
I'm too lazy right now to check if it's already in Bugzilla.
The original code is not invalid though. f is not valid, and that is a
bug, but the original code posted by nullptr should be fine by memory
safety standards.
If there is no possible way to do the original code with dip1000
attributes, then that's a bug with dip1000's design (would not be
surprised).
-Steve
Re: Flatten a range of static arrays
On 2/7/20 6:10 PM, nullptr wrote:
```
import std;
struct SomeRange
{
int[3] val;
enum empty = false;
auto popFront() @safe {}
ref auto front() @safe
{
return val;
}
}
void main() @safe
{
SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
}
```
I don't know how applicable this is to your use case, but this code will
compile and run under -dip1000.
I tried the AST button. for the front function I got:
return pure nothrow @nogc ref @safe int[3] front() return
Note the duplicate return attributes. When I type that in, it doesn't
compile, the return at the front is invalid.
When I remove that return, I get a safety error. So something is weird
about how the attributes are inferred. Either there is some magic
attribute not being printed, or an attribute that can't be specified
manually, or it's a bug.
-Steve
Re: Flatten a range of static arrays
On 08.02.20 00:10, nullptr wrote:
```
import std;
struct SomeRange
{
int[3] val;
enum empty = false;
auto popFront() @safe {}
ref auto front() @safe
{
return val;
}
}
void main() @safe
{
SomeRange().take(10).map!((return ref x) => x[]).joiner.writeln;
}
```
I don't know how applicable this is to your use case, but this code will
compile and run under -dip1000.
That shouldn't compile. You have found a hole in DIP 1000.
struct SomeRange
{
int[3] val = [10, 20, 30];
ref auto front() @safe { return val; }
}
int[] f() @safe
{
SomeRange sr;
// return sr.val[]; /* error, as expected */
return sr.front[]; /* no error, but escapes reference to local */
}
void main() @safe
{
auto x = f();
import std.stdio;
writeln(x); /* Prints garbage. */
}
I'm too lazy right now to check if it's already in Bugzilla.
Re: Flatten a range of static arrays
On Friday, 7 February 2020 at 22:55:29 UTC, Dennis wrote:
Oops, minimized a bit too much. Corrected test case:
```
import std;
struct S {
@safe:
int[3] front = [10, 20, 30];
bool empty = false;
void popFront() {empty = true;}
}
void main() @safe {
S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
It indeed still errors with -dip1000, but without -dip1000 it
compiles now interestingly.
```
import std;
struct SomeRange
{
int[3] val;
enum empty = false;
auto popFront() @safe {}
ref auto front() @safe
{
return val;
}
}
void main() @safe
{
SomeRange().take(10).map!((return ref x) =>
x[]).joiner.writeln;
}
```
I don't know how applicable this is to your use case, but this
code will compile and run under -dip1000.
Re: Flatten a range of static arrays
On Friday, 7 February 2020 at 21:40:36 UTC, Steven Schveighoffer
wrote:
S.popFront is not @safe, and S is not a template. So no
inferrence.
Oops, minimized a bit too much. Corrected test case:
```
import std;
struct S {
@safe:
int[3] front = [10, 20, 30];
bool empty = false;
void popFront() {empty = true;}
}
void main() @safe {
S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
It indeed still errors with -dip1000, but without -dip1000 it
compiles now interestingly.
Re: Flatten a range of static arrays
On 2/7/20 4:17 PM, Dennis wrote:
On Friday, 7 February 2020 at 20:55:14 UTC, nullptr wrote:
Depending on how your range is structured, it might be possible to
just mark front as returning by ref to make this work.
That's a good one. I can't make front() return by ref, but I can make
front a member variable of the range struct. Only problem:
@safe function ... cannot call @system function
std.algorithm.iteration.joiner!(...).joiner
I don't know why. I don't have time to delve into this at the moment but
if anyone wants to try, here's a minimal testcase:
```
import std;
struct S {
int[3] front = [10, 20, 30];
bool empty = false;
void popFront() {empty = true;}
}
void main() @safe {
S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
flags: -dip1000 -dip25
S.popFront is not @safe, and S is not a template. So no inferrence.
But I did that, and it still doesn't compile. Not sure why not. Another
fun dip1000 problem...
-Steve
Re: Flatten a range of static arrays
On Friday, 7 February 2020 at 20:55:14 UTC, nullptr wrote:
Depending on how your range is structured, it might be possible
to just mark front as returning by ref to make this work.
That's a good one. I can't make front() return by ref, but I can
make front a member variable of the range struct. Only problem:
@safe function ... cannot call @system function
std.algorithm.iteration.joiner!(...).joiner
I don't know why. I don't have time to delve into this at the
moment but if anyone wants to try, here's a minimal testcase:
```
import std;
struct S {
int[3] front = [10, 20, 30];
bool empty = false;
void popFront() {empty = true;}
}
void main() @safe {
S.init.map!((return ref x) => x[]).joiner.writeln;
}
```
flags: -dip1000 -dip25
Re: Flatten a range of static arrays
On Friday, 7 February 2020 at 20:31:47 UTC, Steven Schveighoffer wrote: The only solution I can provide is to wrap the static array into a range (maybe something like this exists in Phobos?): Thanks. I was hoping something like that existed in Phobos, but I can't find anything.
Re: Flatten a range of static arrays
On Friday, 7 February 2020 at 20:13:57 UTC, Dennis wrote: If I have an input range with element type `int[3]`, how do I easily turn it into a range of `int` so I can map it? If it were an int[3][] I could simply cast it to an int[] before mapping, but I don't want to eagerly turn it into an array. I thought of doing this: ``` range.map!(x => x[]).joiner.map!(x => x*2); ``` But it gives: Error: returning x[] escapes a reference to parameter x, perhaps annotate with return I tried doing: ``` map!((return x) => x[]) // same error map!((return ref x) => x[]) // does not match, map.front is not an lvalue ``` I can easily work around it with some more code, but I wonder if someone knows an easy solution. Depending on how your range is structured, it might be possible to just mark front as returning by ref to make this work.
Re: Flatten a range of static arrays
On 2/7/20 3:13 PM, Dennis wrote:
If I have an input range with element type `int[3]`, how do I easily
turn it into a range of `int` so I can map it?
If it were an int[3][] I could simply cast it to an int[] before
mapping, but I don't want to eagerly turn it into an array.
I thought of doing this:
```
range.map!(x => x[]).joiner.map!(x => x*2);
```
But it gives:
Error: returning x[] escapes a reference to parameter x, perhaps
annotate with return
This is correct. Consider that you did not mark x as ref. So what x
actually is is a function local. If you returned x[], you just returned
a reference to a local, which is about to go out of scope.
I tried doing:
```
map!((return x) => x[]) // same error
map!((return ref x) => x[]) // does not match, map.front is not an lvalue
```
I can easily work around it with some more code, but I wonder if someone
knows an easy solution.
This means your actual input range is not an array (map should forward
the lvalueness of it). This means you can't just slice, as you will
again return references to the local stack frame.
The only solution I can provide is to wrap the static array into a range
(maybe something like this exists in Phobos?):
struct SARange(T, size_t N)
{
T[N] storage;
size_t elem;
auto front() { return storage[elem]; }
void popFront() { ++elem; }
bool empty() { return elem >= N; }
}
auto asRange(T : E[N], E, size_t N)(T val) {
return SARange!(E, N)(val);
}
range.map!(x => x.asRange).joiner.map!(x => x*2);
-Steve
Flatten a range of static arrays
If I have an input range with element type `int[3]`, how do I easily turn it into a range of `int` so I can map it? If it were an int[3][] I could simply cast it to an int[] before mapping, but I don't want to eagerly turn it into an array. I thought of doing this: ``` range.map!(x => x[]).joiner.map!(x => x*2); ``` But it gives: Error: returning x[] escapes a reference to parameter x, perhaps annotate with return I tried doing: ``` map!((return x) => x[]) // same error map!((return ref x) => x[]) // does not match, map.front is not an lvalue ``` I can easily work around it with some more code, but I wonder if someone knows an easy solution.
