Re: shared arrray problem
On 11/20/2016 12:41 PM, ag0aep6g via Digitalmars-d-learn wrote:
On 11/20/2016 09:09 PM, Charles Hixson via Digitalmars-d-learn wrote:
Thinking it over a bit more, the item returned would need to be a
struct, but the struct wouldn't contain the array, it would just contain
a reference to the array and a start and end offset. The array would
need to live somewhere else, in the class (or struct...but class is
better as you don't want the array evaporating by accident) that created
the returned value. This means you are dealing with multiple levels of
indirection, so it's costly compared to array access, but cheap compared
to lots of large copies. So the returned value would be something like:
struct
{
private:
/** this is a reference to the data that lives elsewhere. It should
be a pointer, but I don't like the syntax*/
ubyte[] data;
intstart, end;///first and last valid indicies into data
public:
this (ubyte[] data, int start, int end)
{this.data = data; this.start = start; this.end = end;}
...
// various routines to access the data, but to limit the access to
the spec'd range, and
// nothing to change the bounds
}
Instead of extra 'start' and 'end' fields you can slice the array. A
dynamic array already is just a reference coupled with a length, i.e.
a pointer with restricted indexing. So you can slice the original
array with your offsets and create the struct with that slice.
I feel like there is a misunderstanding somewhere, but I'm not sure on
whose side. As far as I can tell, your understanding of dynamic arrays
may be lacking, or maybe I don't understand what you're getting at.
While you are definitely correct about slices, I really feel more
comfortable with the start and end fields. I keep being afraid some
smart optimizer is going to decide I don't really need the entire
array. This is probably unjustified, but I find start and end fields
easier to think about. I don't think either of us misunderstands what's
going on, we just feel differently about methods that are approximately
equivalent. Slices would probably be marginally more efficient (well,
certainly so as you'd need two less fields), so if this were a public
library there would be an excellent argument for doing it your way. I
keep trying to think of a reason that start and end fields are better,
and failing. All I've got is that I feel more comfortable with them.
Which is really the answer you already posted, but just a bit more
detail on the construct, and what it meant. (Yeah, I could allow types
other than ubyte as the base case, but I don't want to. I'm thinking of
this mainly as a means of sharing a buffer between applications where
different parts have exclusive access to different parts of the buffer,
and where the buffer will be written to a file with a single fwrite, or
since the underlying storage will be an array, it could even be
rawwrite). I don't want to specify any more than I must about how the
methods calling this will format the storage, and this means that those
with access to different parts may well use different collections of
types, but all types eventually map down to ubytes (or bytes), so ubytes
is the common ground. Perhaps I'll need to write inbuffer,outbuffer
methods/wrappings, but that's far in the future.
Sure, go with a specialized type instead of a template, if that makes
more sense for your use case. As far as I see, the concept is
independent of the element type, so it seemed natural to make it a
template, but a special type is perfectly fine and probably has less
pitfalls.
P.S.: The traits that I mentioned previously were those given by:
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
in your main routine. I assumed that they were validity tests. I don't
understand why they were static. I've never happened to use static
asserts, but I would assume that when they ran cla wouldn't be defined.
Those are tests to ensure that cla's length cannot be set and that it
cannot be appended to. The asserts check that the code does not
compile, i.e. that you cannot do those things. They're static simply
because they can be. The code inside is not executed at run-time.
Re: shared arrray problem
On 11/20/2016 09:09 PM, Charles Hixson via Digitalmars-d-learn wrote:
Thinking it over a bit more, the item returned would need to be a
struct, but the struct wouldn't contain the array, it would just contain
a reference to the array and a start and end offset. The array would
need to live somewhere else, in the class (or struct...but class is
better as you don't want the array evaporating by accident) that created
the returned value. This means you are dealing with multiple levels of
indirection, so it's costly compared to array access, but cheap compared
to lots of large copies. So the returned value would be something like:
struct
{
private:
/** this is a reference to the data that lives elsewhere. It should
be a pointer, but I don't like the syntax*/
ubyte[] data;
intstart, end;///first and last valid indicies into data
public:
this (ubyte[] data, int start, int end)
{this.data = data; this.start = start; this.end = end;}
...
// various routines to access the data, but to limit the access to
the spec'd range, and
// nothing to change the bounds
}
Instead of extra 'start' and 'end' fields you can slice the array. A
dynamic array already is just a reference coupled with a length, i.e. a
pointer with restricted indexing. So you can slice the original array
with your offsets and create the struct with that slice.
I feel like there is a misunderstanding somewhere, but I'm not sure on
whose side. As far as I can tell, your understanding of dynamic arrays
may be lacking, or maybe I don't understand what you're getting at.
Which is really the answer you already posted, but just a bit more
detail on the construct, and what it meant. (Yeah, I could allow types
other than ubyte as the base case, but I don't want to. I'm thinking of
this mainly as a means of sharing a buffer between applications where
different parts have exclusive access to different parts of the buffer,
and where the buffer will be written to a file with a single fwrite, or
since the underlying storage will be an array, it could even be
rawwrite). I don't want to specify any more than I must about how the
methods calling this will format the storage, and this means that those
with access to different parts may well use different collections of
types, but all types eventually map down to ubytes (or bytes), so ubytes
is the common ground. Perhaps I'll need to write inbuffer,outbuffer
methods/wrappings, but that's far in the future.
Sure, go with a specialized type instead of a template, if that makes
more sense for your use case. As far as I see, the concept is
independent of the element type, so it seemed natural to make it a
template, but a special type is perfectly fine and probably has less
pitfalls.
P.S.: The traits that I mentioned previously were those given by:
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
in your main routine. I assumed that they were validity tests. I don't
understand why they were static. I've never happened to use static
asserts, but I would assume that when they ran cla wouldn't be defined.
Those are tests to ensure that cla's length cannot be set and that it
cannot be appended to. The asserts check that the code does not compile,
i.e. that you cannot do those things. They're static simply because they
can be. The code inside is not executed at run-time.
Re: shared arrray problem
On 11/20/2016 03:42 AM, ag0aep6g via Digitalmars-d-learn wrote:
On 11/20/2016 04:34 AM, Charles Hixson via Digitalmars-d-learn wrote:
Whether you would call the change "break things for your code" might be
dubious. It would be effectively broken, even if technically my code
was doing the correct thing. But my code wouldn't be storing the data
that needed storing, so effectively it would be broken.
I don't see how it's dubious. It's an error by the user. When users
are given a dynamic array (and not by reference), they cannot expect
that your code sees changes to length. That's just not how arrays
work. When a user has that wrong expectation, and writes wrong code
because of it, then it's arguably their own fault. However, if you
want you can hold their hand a bit and make the mistake less likely.
"Write something
for yourself" is what I'd like to do, given that the language doesn't
have that built-in support, but I can't see how to do it.
Wrap the array in a struct that has indexing, but doesn't allow
setting the length or appending. Here's a quick prototype:
struct ConstLengthArray(E)
{
private E[] data;
this(E[] arr) { this.data = arr; }
ref inout(E) opIndex(size_t i) inout { return data[i]; }
@property size_t length() const { return data.length; }
}
void main()
{
auto cla = ConstLengthArray!ubyte([1, 2, 3, 4, 5]);
/* Mutating elements is allowed: */
cla[0] = 10;
assert(cla[0] == 10);
/* No setting length, no appending: */
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
}
You might want to add support for slicing, concatenation, etc. Maybe
allow implicit conversion to const(E[]), though that would also allow
conversion to const(E)[] and that has a settable length again
Thinking it over a bit more, the item returned would need to be a
struct, but the struct wouldn't contain the array, it would just contain
a reference to the array and a start and end offset. The array would
need to live somewhere else, in the class (or struct...but class is
better as you don't want the array evaporating by accident) that created
the returned value. This means you are dealing with multiple levels of
indirection, so it's costly compared to array access, but cheap compared
to lots of large copies. So the returned value would be something like:
struct
{
private:
/** this is a reference to the data that lives elsewhere. It
should be a pointer, but I don't like the syntax*/
ubyte[] data;
intstart, end;///first and last valid indicies into data
public:
this (ubyte[] data, int start, int end)
{this.data = data; this.start = start; this.end = end;}
...
// various routines to access the data, but to limit the access to
the spec'd range, and
// nothing to change the bounds
}
Which is really the answer you already posted, but just a bit more
detail on the construct, and what it meant. (Yeah, I could allow types
other than ubyte as the base case, but I don't want to. I'm thinking of
this mainly as a means of sharing a buffer between applications where
different parts have exclusive access to different parts of the buffer,
and where the buffer will be written to a file with a single fwrite, or
since the underlying storage will be an array, it could even be
rawwrite). I don't want to specify any more than I must about how the
methods calling this will format the storage, and this means that those
with access to different parts may well use different collections of
types, but all types eventually map down to ubytes (or bytes), so ubytes
is the common ground. Perhaps I'll need to write inbuffer,outbuffer
methods/wrappings, but that's far in the future.
P.S.: The traits that I mentioned previously were those given by:
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
in your main routine. I assumed that they were validity tests. I don't
understand why they were static. I've never happened to use static
asserts, but I would assume that when they ran cla wouldn't be defined.
N.B.: Even this much is just thinking about design, not something I'll
actually do at the moment. But this is a problem I keep coming up
against, so a bit of thought now seemed a good idea.
Re: shared arrray problem
On 11/20/2016 08:30 PM, Charles Hixson via Digitalmars-d-learn wrote: Well, that precise approach wouldn't work. (The traits aren't a part of the sturct, e.g.), What do you mean by "traits"?
Re: shared arrray problem
On 11/20/2016 03:42 AM, ag0aep6g via Digitalmars-d-learn wrote:
On 11/20/2016 04:34 AM, Charles Hixson via Digitalmars-d-learn wrote:
Whether you would call the change "break things for your code" might be
dubious. It would be effectively broken, even if technically my code
was doing the correct thing. But my code wouldn't be storing the data
that needed storing, so effectively it would be broken.
I don't see how it's dubious. It's an error by the user. When users
are given a dynamic array (and not by reference), they cannot expect
that your code sees changes to length. That's just not how arrays
work. When a user has that wrong expectation, and writes wrong code
because of it, then it's arguably their own fault. However, if you
want you can hold their hand a bit and make the mistake less likely.
"Write something
for yourself" is what I'd like to do, given that the language doesn't
have that built-in support, but I can't see how to do it.
Wrap the array in a struct that has indexing, but doesn't allow
setting the length or appending. Here's a quick prototype:
struct ConstLengthArray(E)
{
private E[] data;
this(E[] arr) { this.data = arr; }
ref inout(E) opIndex(size_t i) inout { return data[i]; }
@property size_t length() const { return data.length; }
}
void main()
{
auto cla = ConstLengthArray!ubyte([1, 2, 3, 4, 5]);
/* Mutating elements is allowed: */
cla[0] = 10;
assert(cla[0] == 10);
/* No setting length, no appending: */
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
}
You might want to add support for slicing, concatenation, etc. Maybe
allow implicit conversion to const(E[]), though that would also allow
conversion to const(E)[] and that has a settable length again.
Well, that precise approach wouldn't work. (The traits aren't a part of
the sturct, e.g.), but returning a struct (or perhaps a class) rather
than an actual array has promise. It could even allow separate callers
to have separate views of the data based on some sort of registered key,
which they could share on an as-needed basis. That's too much overhead
work for this project, but has promise for the more general problem.
Re: shared arrray problem
On 11/20/2016 04:34 AM, Charles Hixson via Digitalmars-d-learn wrote:
Whether you would call the change "break things for your code" might be
dubious. It would be effectively broken, even if technically my code
was doing the correct thing. But my code wouldn't be storing the data
that needed storing, so effectively it would be broken.
I don't see how it's dubious. It's an error by the user. When users are
given a dynamic array (and not by reference), they cannot expect that
your code sees changes to length. That's just not how arrays work. When
a user has that wrong expectation, and writes wrong code because of it,
then it's arguably their own fault. However, if you want you can hold
their hand a bit and make the mistake less likely.
"Write something
for yourself" is what I'd like to do, given that the language doesn't
have that built-in support, but I can't see how to do it.
Wrap the array in a struct that has indexing, but doesn't allow setting
the length or appending. Here's a quick prototype:
struct ConstLengthArray(E)
{
private E[] data;
this(E[] arr) { this.data = arr; }
ref inout(E) opIndex(size_t i) inout { return data[i]; }
@property size_t length() const { return data.length; }
}
void main()
{
auto cla = ConstLengthArray!ubyte([1, 2, 3, 4, 5]);
/* Mutating elements is allowed: */
cla[0] = 10;
assert(cla[0] == 10);
/* No setting length, no appending: */
static assert(!__traits(compiles, cla.length = 3));
static assert(!__traits(compiles, cla ~= 6));
}
You might want to add support for slicing, concatenation, etc. Maybe
allow implicit conversion to const(E[]), though that would also allow
conversion to const(E)[] and that has a settable length again.
Re: shared arrray problem
On 11/19/2016 05:52 PM, ag0aep6g via Digitalmars-d-learn wrote: On 11/20/2016 01:33 AM, Charles Hixson via Digitalmars-d-learn wrote: Yes. I was hoping someone would pop up with some syntax making the array, but not its contents, const or immutable, which I couldn't figure out how to do, and which is what I really hoped would be the answer, but it appears that this isn't part of the syntax. Yup, head const is not part of the language. You'd have to find a library solution or write something yourself. I really *can't* allow the length to be changed, Your emphasis suggests that user could break things for your code. They can't. Any changes to the length will only affect the slice on the user's end. They can only fool themselves. That may be bad enough to warrant a more restricted return type, but for your code it's safe to return a plain dynamic array. Whether you would call the change "break things for your code" might be dubious. It would be effectively broken, even if technically my code was doing the correct thing. But my code wouldn't be storing the data that needed storing, so effectively it would be broken. "Write something for yourself" is what I'd like to do, given that the language doesn't have that built-in support, but I can't see how to do it. I want to end up with a continuous array of ubytes of a given length with certain parts reserved to only be directly accessible to the defining class, and other parts accessible to the calling class(es). And the length of the array isn't known until run time. So I guess the only safe solution is to do an extra copy...which isn't a problem in this particular application as I only need to do it twice per file opening (once on opening, once on closing), but for other applications would be a real drag.
Re: shared arrray problem
On 11/20/2016 01:33 AM, Charles Hixson via Digitalmars-d-learn wrote: Yes. I was hoping someone would pop up with some syntax making the array, but not its contents, const or immutable, which I couldn't figure out how to do, and which is what I really hoped would be the answer, but it appears that this isn't part of the syntax. Yup, head const is not part of the language. You'd have to find a library solution or write something yourself. I really *can't* allow the length to be changed, Your emphasis suggests that user could break things for your code. They can't. Any changes to the length will only affect the slice on the user's end. They can only fool themselves. That may be bad enough to warrant a more restricted return type, but for your code it's safe to return a plain dynamic array.
Re: shared arrray problem
On 11/19/2016 01:50 PM, ag0aep6g via Digitalmars-d-learn wrote: On 11/19/2016 10:26 PM, Charles Hixson via Digitalmars-d-learn wrote: It's worse than that, if they modify the length the array may be reallocated in RAM so that the pointers held by the containing class do not point to the changed values. (Read the header comments...it's not nice at all.) Arguably, any D programmer must be aware that appending to a dynamic array potentially means making a copy of the data, and that changes to length are not visible to other views of the data. But it's an opportunity to mess up, for sure. You could return a wrapper around the array that supports editing the data but not changing the length or appending. Looks like std.experimental.typecons.Final [1] is supposed to be that wrapper. But in a little test I can still set the length. Not sure if that's a bug, or if Final has slightly different goals. [1] https://dlang.org/phobos/std_experimental_typecons.html#.Final Yes. I was hoping someone would pop up with some syntax making the array, but not its contents, const or immutable, which I couldn't figure out how to do, and which is what I really hoped would be the answer, but it appears that this isn't part of the syntax. If the array is constant, so is it's contents. I really *can't* allow the length to be changed, and if the array is reallocated, it won't get saved. But the contents of the array are intended to be changed by the calling routines. Again, for this particular problem the kludge of copying the values into the array works fine (and is what I've decided to do), but that's not a good general solution to this kind of problem.
Re: shared arrray problem
On 11/19/2016 10:26 PM, Charles Hixson via Digitalmars-d-learn wrote: It's worse than that, if they modify the length the array may be reallocated in RAM so that the pointers held by the containing class do not point to the changed values. (Read the header comments...it's not nice at all.) Arguably, any D programmer must be aware that appending to a dynamic array potentially means making a copy of the data, and that changes to length are not visible to other views of the data. But it's an opportunity to mess up, for sure. You could return a wrapper around the array that supports editing the data but not changing the length or appending. Looks like std.experimental.typecons.Final [1] is supposed to be that wrapper. But in a little test I can still set the length. Not sure if that's a bug, or if Final has slightly different goals. [1] https://dlang.org/phobos/std_experimental_typecons.html#.Final
Re: shared arrray problem
On 11/19/2016 11:10 AM, Nicolas Gurrola via Digitalmars-d-learn wrote:
On Saturday, 19 November 2016 at 18:51:05 UTC, Charles Hixson wrote:
ubyte[]header()@property {return fHead[4..$];}
This method should do what you want. You are only returning a slice of
the fHead array, so if the caller modifies the length it will only
affect of the return value, and not the length of fHead itself.
It's worse than that, if they modify the length the array may be
reallocated in RAM so that the pointers held by the containing class do
not point to the changed values. (Read the header comments...it's not
nice at all.)
More, the class explicitly requires the array to be a particular length
as it needs to fit into a spot in a binary file, so I really want to
forbid any replacement of the array for any reason. The first four
bytes are managed by the class and returned via alternate routines which
do not allow the original values to be altered, and this is necessary.
I could make them immutable, but actually what I do is return, e.g., an
int of 256 * fHead[0] + fHead[1], which is the length of the header.
It's an int to allow negative values to be returned in case of error.
So what I'll probably eventually decide on is some variation of
saveToHead, e.g.:
boolsaveToHeader(ubyte[] rec)
{ if(rec.length + 4 > dheader)returnfalse;
fHead[4..recLen + 4] = rec[0..$];
returntrue;
}
unless I can think of something better. Actually for this particular
case that's not a bad approach, but for the larger problem it's a lousy
kludge.
Re: shared arrray problem
On Saturday, 19 November 2016 at 18:51:05 UTC, Charles Hixson
wrote:
ubyte[]header()@property{return
fHead[4..$];}
This method should do what you want. You are only returning a
slice of the fHead array, so if the caller modifies the length it
will only affect of the return value, and not the length of fHead
itself.
