Re: What type does byGrapheme() return?
On Monday, 6 January 2020 at 08:39:19 UTC, Robert M. Münch wrote: On 2020-01-05 04:18:34 +, H. S. Teoh said: At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities. Done... my first bug report :-) I copied togehter all the findings from this thread. For the sake of completeness https://issues.dlang.org/show_bug.cgi?id=20483
Re: What type does byGrapheme() return?
On 2020-01-05 04:18:34 +, H. S. Teoh said: At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities. Done... my first bug report :-) I copied togehter all the findings from this thread. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Re: What type does byGrapheme() return?
On Sat, Jan 04, 2020 at 08:19:14PM +0100, Robert M. Münch via Digitalmars-d-learn wrote: > On 2019-12-31 21:36:56 +, Steven Schveighoffer said: > > > The fact that a Grapheme's return requires you keep the grapheme in > > scope for operations seems completely incorrect and dangerous IMO > > (note that operators are going to always have a ref this, even when > > called on an rvalue). So even though using ref works, I think the > > underlying issue here really is the lifetime problem. > > Thanks for all the answers, pretty enlighting even I'm not sure I get > everything 100%. > > So, what to do for now? File a bug-report? What needs to be fixed? At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities. > I'm using the ref approach for now, in hope it will be OK for my > use-case, which is converting a wstring to a grapheme[], alter the > array, and map it back to a wstring. Sounds like a lot of porcessing > for handling unicode text, but I don't think it gets a lot simpler > than that. [...] Unicode is a beast. Be glad that you can even do this in the first place. If I were writing this in C, I wouldn't even know where to begin! T -- No! I'm not in denial!
Re: What type does byGrapheme() return?
On 2019-12-31 21:36:56 +, Steven Schveighoffer said: The fact that a Grapheme's return requires you keep the grapheme in scope for operations seems completely incorrect and dangerous IMO (note that operators are going to always have a ref this, even when called on an rvalue). So even though using ref works, I think the underlying issue here really is the lifetime problem. Thanks for all the answers, pretty enlighting even I'm not sure I get everything 100%. So, what to do for now? File a bug-report? What needs to be fixed? I'm using the ref approach for now, in hope it will be OK for my use-case, which is converting a wstring to a grapheme[], alter the array, and map it back to a wstring. Sounds like a lot of porcessing for handling unicode text, but I don't think it gets a lot simpler than that. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Re: What type does byGrapheme() return?
On Tue, Dec 31, 2019 at 04:36:56PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
> On 12/31/19 4:22 PM, H. S. Teoh wrote:
[...]
> > import std;
> > void main() {
> > auto x = "Bla\u0301hbla\u0310h\u0309!";
> > auto r = x.byGrapheme;
> > writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
> > }
[...]
> > What did I do wrong?
>
> auto r = x.byGrapheme.array;
Haha, in my hurry I totally forgot about the .array. Mea culpa.
[...]
> The fact that a Grapheme's return requires you keep the grapheme in
> scope for operations seems completely incorrect and dangerous IMO
> (note that operators are going to always have a ref this, even when
> called on an rvalue). So even though using ref works, I think the
> underlying issue here really is the lifetime problem.
[...]
After my wrong recollection of the history surrounding indexOf vs.
countUntil, I'm not sure I can rely on my memory anymore, :-P but AIUI
Dmitri implemented it this way because he wanted to avoid allocations
(GC or otherwise) in the most common case of Grapheme containing just a
small number of code points (usually 1 or 2). When the number of
combining diacritics exceed the size of the Grapheme struct, then it
would quietly switch to malloc or some such for holding the data. My
guess is that this is the reason for passing to the wrapper range
returned by opSlice(). And possibly it's also to allow mutation of the
Grapheme via the returned slice?
Perhaps this whole approach should be looked at again. Certainly, unless
I'm missing something, it *ought* to be possible to implement Grapheme
in a way that doesn't require this scoped reference business.
T
--
The diminished 7th chord is the most flexible and fear-instilling chord. Use it
often, use it unsparingly, to subdue your listeners into submission!
Re: What type does byGrapheme() return?
On 12/31/19 4:22 PM, H. S. Teoh wrote:
On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
On 12/31/19 2:58 PM, H. S. Teoh wrote:
On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
e.g.:
writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);
[...]
Unfortunately this doesn't work. Somehow the ref parameter doesn't
match whatever it is std.algorithm.map is trying to pass to it.
Huh, it seemed to work for me. Got the full "Robert" with an R. map
does support ref-ness. Maybe you didn't put ref in the right place?
Here's my full non-working code:
import std;
void main() {
auto x = "Bla\u0301hbla\u0310h\u0309!";
auto r = x.byGrapheme;
writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
}
The compiler says:
/usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D
main.__lambda1 cannot deduce function from argument types !()(Grapheme),
candidates are:
test.d(5):__lambda1
/usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template
instance test.main.MapResult!(__lambda1, Result!string) error instantiating
test.d(5):instantiated from here: map!(Result!string)
What did I do wrong?
auto r = x.byGrapheme.array;
This is how Robert originally had it if you look a few messages up.
Otherwise, it's not an lvalue.
The fact that a Grapheme's return requires you keep the grapheme in
scope for operations seems completely incorrect and dangerous IMO (note
that operators are going to always have a ref this, even when called on
an rvalue). So even though using ref works, I think the underlying issue
here really is the lifetime problem.
-Steve
Re: What type does byGrapheme() return?
On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
> On 12/31/19 2:58 PM, H. S. Teoh wrote:
> > On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
> > Digitalmars-d-learn wrote:
> > > e.g.:
> > >
> > > writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);
> > [...]
> >
> > Unfortunately this doesn't work. Somehow the ref parameter doesn't
> > match whatever it is std.algorithm.map is trying to pass to it.
>
> Huh, it seemed to work for me. Got the full "Robert" with an R. map
> does support ref-ness. Maybe you didn't put ref in the right place?
Here's my full non-working code:
import std;
void main() {
auto x = "Bla\u0301hbla\u0310h\u0309!";
auto r = x.byGrapheme;
writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
}
The compiler says:
/usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D
main.__lambda1 cannot deduce function from argument types !()(Grapheme),
candidates are:
test.d(5):__lambda1
/usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template
instance test.main.MapResult!(__lambda1, Result!string) error instantiating
test.d(5):instantiated from here: map!(Result!string)
What did I do wrong?
T
--
What do you get if you drop a piano down a mineshaft? A flat minor.
Re: What type does byGrapheme() return?
On 12/31/19 2:58 PM, H. S. Teoh wrote:
On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
e.g.:
writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);
[...]
Unfortunately this doesn't work. Somehow the ref parameter doesn't match
whatever it is std.algorithm.map is trying to pass to it.
Huh, it seemed to work for me. Got the full "Robert" with an R. map does
support ref-ness. Maybe you didn't put ref in the right place?
-Steve
Re: What type does byGrapheme() return?
On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
> On 12/30/19 6:31 PM, H. S. Teoh wrote:
> > On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via
> > Digitalmars-d-learn wrote:
[...]
> > Haha, it's actually right there in the Grapheme docs for the opSlice
> > overloads:
> >
> > Random-access range over Grapheme's $(CHARACTERS).
> >
> > Warning: Invalidates when this Grapheme leaves the scope,
> > attempts to use it then would lead to memory corruption.
> >
> > Looks like when you use .map over the Grapheme, it gets copied into
> > a temporary, which gets invalidated when map.front returns.
> > Somewhere we're missing a 'scope' qualifier...
[...]
> Then the original example should be fixable by putting "ref" in for
> all the lambdas.
>
> But this is kind of disturbing. Why does the grapheme do this? The
> original data is not scoped.
Honestly I have no idea, but glancing at the code in std.uni reveals
that the returned slice is actually a wrapper object that contains a
pointer to the parent Grapheme object. So if the parent was a
temporary and goes out of scope before the wrapper does, we're left with
a dangling pointer.
> e.g.:
>
> writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);
[...]
Unfortunately this doesn't work. Somehow the ref parameter doesn't match
whatever it is std.algorithm.map is trying to pass to it.
T
--
They say that "guns don't kill people, people kill people." Well I think the
gun helps. If you just stood there and yelled BANG, I don't think you'd kill
too many people. -- Eddie Izzard, Dressed to Kill
Re: What type does byGrapheme() return?
On 12/30/19 6:31 PM, H. S. Teoh wrote:
On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:
[...]
I suspect the cause is that whatever Grapheme.opSlice returns is going
out-of-scope when used with .map, that's why it's malfunctioning.
[...]
Haha, it's actually right there in the Grapheme docs for the opSlice
overloads:
Random-access range over Grapheme's $(CHARACTERS).
Warning: Invalidates when this Grapheme leaves the scope,
attempts to use it then would lead to memory corruption.
Looks like when you use .map over the Grapheme, it gets copied into a
temporary, which gets invalidated when map.front returns. Somewhere
we're missing a 'scope' qualifier...
Then the original example should be fixable by putting "ref" in for all
the lambdas.
But this is kind of disturbing. Why does the grapheme do this? The
original data is not scoped.
e.g.:
writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);
-Steve
Re: What type does byGrapheme() return?
On Mon, Dec 30, 2019 at 03:31:31PM -0800, H. S. Teoh via Digitalmars-d-learn wrote: > On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn > wrote: > [...] > > I suspect the cause is that whatever Grapheme.opSlice returns is > > going out-of-scope when used with .map, that's why it's > > malfunctioning. > [...] > > Haha, it's actually right there in the Grapheme docs for the opSlice > overloads: > > Random-access range over Grapheme's $(CHARACTERS). > > Warning: Invalidates when this Grapheme leaves the scope, > attempts to use it then would lead to memory corruption. > > Looks like when you use .map over the Grapheme, it gets copied into a > temporary, which gets invalidated when map.front returns. Somewhere > we're missing a 'scope' qualifier... [...] Indeed, compiling with dmd -dip1000 produces this error message: test.d(15): Error: returning g.opSlice() escapes a reference to parameter g, perhaps annotate with return /usr/src/d/phobos/std/algorithm/iteration.d(499):instantiated from here: MapResult!(__lambda1, Grapheme[]) test.d(15):instantiated from here: map!(Grapheme[]) Not the most helpful message (the annotation has to go in Phobos code, not in user code), but it does at least point to the cause of the problem. T -- What doesn't kill me makes me stranger.
Re: What type does byGrapheme() return?
On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn wrote: [...] > I suspect the cause is that whatever Grapheme.opSlice returns is going > out-of-scope when used with .map, that's why it's malfunctioning. [...] Haha, it's actually right there in the Grapheme docs for the opSlice overloads: Random-access range over Grapheme's $(CHARACTERS). Warning: Invalidates when this Grapheme leaves the scope, attempts to use it then would lead to memory corruption. Looks like when you use .map over the Grapheme, it gets copied into a temporary, which gets invalidated when map.front returns. Somewhere we're missing a 'scope' qualifier... T -- War doesn't prove who's right, just who's left. -- BSD Games' Fortune
Re: What type does byGrapheme() return?
On Sun, Dec 29, 2019 at 01:19:09PM +0100, Robert M. Münch via
Digitalmars-d-learn wrote:
> On 2019-12-27 19:44:59 +, H. S. Teoh said:
[...]
> > If you want to add/delete/change graphemes, what you *really* want
> > is to use an array of Graphemes:
> >
> > Grapheme[] editableGraphs;
> >
> > You can then splice it, insert stuff, delete stuff, whatever.
> >
> > When you're done with it, convert it back to string with something
> > like this:
> >
> > string result = editableGraphs.map!(g => g[]).joiner.to!string;
>
> I played around with this approach...
>
> string r1 = "Robert M. Münch";
> // Code-Units = 16
> // Code-Points = 15
> // Graphemes = 15
>
> Grapheme[] gr1 = r1.byGrapheme.array;
> writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string);
> // Text = obert M. Münch
> writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring);
> // wText = obert M. Münch
> writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring);
> // dText = obert M. Münch
>
> Why is the first letter missing? Is this a bug?
[...]
I suspect there's a scope-related bug/issue somewhere here. I did some
experiments and discovered that using foreach to iterate over a
Grapheme[] is OK, but somehow when using Grapheme[] with .map to slice
over each one, I get random UTF-8 encoding errors and missing
characters.
I suspect the cause is that whatever Grapheme.opSlice returns is going
out-of-scope when used with .map, that's why it's malfunctioning. The
last time I looked at the Grapheme code, there's a bunch of
memory-related stuff involving dtors that's *probably* the cause of this
problem.
Please file a bug for this.
T
--
Life is complex. It consists of real and imaginary parts. -- YHL
Re: What type does byGrapheme() return?
On Friday, 27 December 2019 at 17:26:58 UTC, Robert M. Münch wrote: ... There are set of range interfaces that can be used to mask range type. Check for https://dlang.org/library/std/range/interfaces/input_range.html for starting point, and for https://dlang.org/library/std/range/interfaces/input_range_object.html for wrapping any range to those interfaces. Note: resulting wrapped range is an object and has reference semantics, beware of using it directly with other range algorithms as they can consume your range. Best regards, Alexandru.
Re: What type does byGrapheme() return?
On 2019-12-27 19:44:59 +, H. S. Teoh said:
Since graphemes are variable-length in terms of code points, you can't
exactly *edit* a range of graphemes -- you can't replace a 1-codepoint
grapheme with a 6-codepoint grapheme, for example, since there's no
space in the underlying string to store that.
Hi, my idea was that when I use a grapheme range, it will abstract away
that graphemes consist of different sized code-points. And the docs at
https://dlang.org/phobos/std_uni.html#byGrapheme show an example using
this kind of range:
auto gText = text.byGrapheme;
gText.take(3);
gText.drop(3);
But maybe I need to get a better understanding of the ranges stuff too...
If you want to add/delete/change graphemes, what you *really* want is to
use an array of Graphemes:
Grapheme[] editableGraphs;
You can then splice it, insert stuff, delete stuff, whatever.
When you're done with it, convert it back to string with something like
this:
string result = editableGraphs.map!(g => g[]).joiner.to!string;
I played around with this approach...
string r1 = "Robert M. Münch";
// Code-Units = 16
// Code-Points = 15
// Graphemes = 15
Grapheme[] gr1 = r1.byGrapheme.array;
writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string);
// Text = obert M. Münch
writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring);
// wText = obert M. Münch
writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring);
// dText = obert M. Münch
Why is the first letter missing? Is this a bug?
--
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Re: What type does byGrapheme() return?
On 2019-12-27 17:54:28 +, Steven Schveighoffer said:
This is the rub with ranges. You need to use typeof. There's no other
way to do it, because the type returned by byGrapheme depends on the
type of Range.
Hi, ok, thanks a lot and IMO these are the fundamental important
information for people using D (beginners, causual programmers, etc.)
to understand how things fit together.
If you know what type Range is, it would be:
struct S
{
typeof(string.init.byGrapheme()) gText;
// or
alias GRange = typeof(string.init.byGrapheme());
GRange gText;
}
Ah... I didn't know that I can use a basic type "string" combined with
".init" to manually build the type. Neat...
Subbing in whatever your real range for `string`. Or if it's the result
of a bunch of adapters, use the whole call chain with typeof.
Ok, and these are good candidates for alias definitions to avoid
re-typing it many times.
Why not just declare the real range type? Because it's going to be
ugly, especially if your underlying range is the result of other range
algorithms. And really, typeof is going to be the better mechanism,
even if it's not the best looking thing.
I think I got it... thanks a lot.
--
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Re: What type does byGrapheme() return?
On Fri, Dec 27, 2019 at 06:26:58PM +0100, Robert M. Münch via Digitalmars-d-learn wrote: > I love these documentation lines in the D docs: > > auto byGrapheme(Range)(Range range) > > How should I know what auto is? Why not write the explicit type so > that I know what to expect? When declaring a variable as class/struct > member I can't use auto but need the explicit type... > > I used typeof() but that doesn't help a lot: > > gText = [Grapheme(53, 0, 0, 72057594037927936, [83, , 1)]Result!string > > I want to iterate a string byGrapheme so that I can add, delete, > change graphemes. [...] Since graphemes are variable-length in terms of code points, you can't exactly *edit* a range of graphemes -- you can't replace a 1-codepoint grapheme with a 6-codepoint grapheme, for example, since there's no space in the underlying string to store that. If you want to add/delete/change graphemes, what you *really* want is to use an array of Graphemes: Grapheme[] editableGraphs; You can then splice it, insert stuff, delete stuff, whatever. When you're done with it, convert it back to string with something like this: string result = editableGraphs.map!(g => g[]).joiner.to!string; T -- The irony is that Bill Gates claims to be making a stable operating system and Linus Torvalds claims to be trying to take over the world. -- Anonymous
Re: What type does byGrapheme() return?
On 12/27/19 12:26 PM, Robert M. Münch wrote:
I love these documentation lines in the D docs:
auto byGrapheme(Range)(Range range)
How should I know what auto is? Why not write the explicit type so that
I know what to expect? When declaring a variable as class/struct member
I can't use auto but need the explicit type...
I used typeof() but that doesn't help a lot:
gText = [Grapheme(53, 0, 0, 72057594037927936, [83, , 1)]Result!string
I want to iterate a string byGrapheme so that I can add, delete, change
graphemes.
This is the rub with ranges. You need to use typeof. There's no other
way to do it, because the type returned by byGrapheme depends on the
type of Range.
If you know what type Range is, it would be:
struct S
{
typeof(string.init.byGrapheme()) gText;
// or
alias GRange = typeof(string.init.byGrapheme());
GRange gText;
}
Subbing in whatever your real range for `string`. Or if it's the result
of a bunch of adapters, use the whole call chain with typeof.
Why not just declare the real range type? Because it's going to be ugly,
especially if your underlying range is the result of other range
algorithms. And really, typeof is going to be the better mechanism, even
if it's not the best looking thing.
-Steve
