Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On 28/12/21 07:21, Bart via lazarus wrote:
On Mon, Dec 27, 2021 at 6:35 PM Marco van de Voort via lazarus
wrote:
The expression seems to be 1 when the top bits are 10 iow when it is a
follow bytes of utf8, that is what the comment says, and I as far as I
can see the signedness doesn't matter.
Basically to me that seems to be a branchless version of
if (p[i] and %1100)=%1000 then
inc(result);
This is how I understood that paert of the code as well.
Just a side node: all this assumes that the UTF8 is correct (in the
strict sense).
Now for the part that tries to do calculations on blocks of 32 or 64 bits.
It uses a multiplication in that part.
That seems a bit odd as this code tries to do evrything with ands,
nots, and shifts.
Would this approach (in that part of the code) also work?
X = 32 or 64 bit block
1: AND X with EIGHTYMASK
2: SHR 7: gives (A)
3: NOT X
4: SHR 6: gives (B)
5: (A) AND (B): gives (C)
Basically we do the same as for a 1-byte block;
Now we have a pattern where any 1 tells us this byte was a following byte
A 32-bit example:
(Invalid sequence but that does not matter here)
X = %11xx %01xx %00xx %10xx
(Leading-ASCII-ASCII-Following, so count of following bytes must be 1)
AND with EIGTHYMASK gives %1000 % % %1000
SHR 7 gives: %0001 % % %0001 (A)
NOT %11xx %01xx %00xx %10xx = %00yy %10yy
%11yy %01yy
SHR 6 gives: % %yy10 %yy11 %yy01 (B)
(A) and (B) gives: % % % %0001 (C)
All non-following bytes turn into %
The count of following bytes is PopCnt(C)
As long as PopCnt is available as a machine instruction, this will be
faster than the (nx * ONEMASK) calculation I think.
(I would hope this is the case for all platforms Lazarus supports, if
not the call to PopCnt could be ifdef-ed.)
So basically change this:
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
{$push}{$overflowchecks off} // "nx * ONEMASK" causes an
arithmetic overflow.
Result += (nx * ONEMASK) >> ((sizeof(PtrInt) - 1) * 8);
{$pop}
into
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
Result := Result + PopCnt(PtrUInt(nx)); /PopCnt only accepts
unsigend parameter
Since bit manipulating is not my strongpoint, please comment.
I ran my code that calls UTF8LengthFast with a euro through the debugger
to see what path through UTF8LengthFast was followed. The debugger
skipped over all the loops until the very last one:
// Take care of any left-over bytes.
while ixOK, time to simplify. I tried writing a short pascal program that did
the bit shifting with constants:
c := (226 shr 7) and ((not 226) shr 6);
writeln('c='+inttostr(c)+' 226.');
c := (130 shr 7) and ((not 130) shr 6);
writeln('c='+inttostr(c)+' 130.');
c := (172 shr 7) and ((not 172) shr 6);
writeln('c='+inttostr(c)+' 172.');
This produces the correct result:
c=0 226.
c=1 130.
c=1 172.
Next I tried using a pint8:
const
s = '€';
var
p: PChar;
c: Byte;
pn8: pint8 absolute p;
begin
c := (pn8^ shr 7) and ((not pn8^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(Byte(pn8^)));
Inc(pn8);
c := (pn8^ shr 7) and ((not pn8^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(Byte(pn8^)));
Inc(pn8);
c := (pn8^ shr 7) and ((not pn8^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(Byte(pn8^)));
end;
This produces:
c=252 226
c=253 130
c=253 172
Clearly wrong. Someone mentioned signed vs unsigned, so that seemed a
logical next step. I tried using a pbyte.
const
s = '€';
var
pb: PByte absolute p;
c: Byte;
p: PChar;
begin
p := PChar(s);
c := (pb^ shr 7) and ((not pb^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(pb^));
Inc(pb);
c := (pb^ shr 7) and ((not pb^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(pb^));
Inc(pb);
c := (pb^ shr 7) and ((not pb^) shr 6);
writeln('c='+inttostr(c)+' '+IntToStr(pb^));
end;
This produces:
c=0 226
c=1 130
c=1 172
Correct result. So it appears that using a pint8 produces the wrong
result on aarch64, but it doesn't on x86_64. It's not clear why, though.
So it appears to me that an unsigned pointer type is required in
UTFLengthFast.
--
___
lazarus mailing list
lazarus@lists.lazarus-ide.org
https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On Mon, Dec 27, 2021 at 10:02 PM Noel Duffy via lazarus
wrote:
> It's not just the euro, though. It's any utf-8 sequence.
What I meant was that a single '€' (or any other single UTF8
"character") will not enter the mentioned block.
Can you add some debug statements to display the values of the it
uses in the calculation like I did in the 4th message in this thread?
function UTF8LengthFast(p: PChar; ByteCount: PtrInt): PtrInt;
const
{$ifdef CPU32}
ONEMASK =$01010101;
EIGHTYMASK=$80808080;
{$endif}
{$ifdef CPU64}
ONEMASK =$0101010101010101;
EIGHTYMASK=$8080808080808080;
{$endif}
var
pnx: PPtrInt absolute p; // To get contents of text in PtrInt
blocks. x refers to 32 or 64 bits
pn8: pint8 absolute pnx; // To read text as Int8 in the initial and
final loops
ix: PtrInt absolute pnx; // To read text as PtrInt in the block loop
nx: PtrInt; // values processed in block loop
i,cnt,e: PtrInt;
begin
Result := 0;
e := ix+ByteCount; // End marker
// Handle any initial misaligned bytes.
cnt := (not (ix-1)) and (sizeof(PtrInt)-1);
if cnt>ByteCount then
cnt := ByteCount;
for i := 1 to cnt do
begin
// Is this byte NOT the first byte of a character?
writeln('pn8^ = ',byte(pn8^).ToBinString);
writeln('pn8^ shr 7 = ',Byte(Byte(pn8^) shr 7).ToBinString);
writeln('not pn8^ = ',Byte(not pn8^).ToBinString);
writeln('(not pn8^) shr 6 = ',Byte((not pn8^) shr 6).ToBinString);
writeln;
Result += (pn8^ shr 7) and ((not pn8^) shr 6);
inc(pn8);
end;
// Handle complete blocks
for i := 1 to (ByteCount-cnt) div sizeof(PtrUInt) do
begin
// Count bytes which are NOT the first byte of a character.
{
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
{$push}{$overflowchecks off} // "nx * ONEMASK" causes an
arithmetic overflow.
Result += (nx * ONEMASK) >> ((sizeof(PtrInt) - 1) * 8);
{$pop}
}
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
Result := Result + PopCnt(PtrUInt(nx));
inc(pnx);
end;
// Take care of any left-over bytes.
while ixhttps://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On 28/12/21 01:47, Juha Manninen via lazarus wrote: On Mon, Dec 27, 2021 at 1:44 AM Noel Duffy via lazarus < lazarus@lists.lazarus-ide.org> wrote: I need some help getting to the root of a problem with incorrect results on Apple hardware (M1, aarch64) for the function UTF8LengthFast in lazutf8. On MacOS, when given a string containing one or more UTF8 characters, UTF8LengthFast returns wildly incorrect results. On Fedora, the function returns the correct answer. You mean both MacOS and Fedora run on the same aarch64 CPU? Oh no, Fedora runs on an Intel x86_64. I don't think there's a Fedora that runs on aarch64. I included the results from Fedora just to show that the code does work in some places. It must be a Big endian / Little endian issue. IIRC it can be adjusted in ARM CPUs. Why do MacOS and Linux use a different setting there? I have no idea. Well, as Florian said above, the M1 is little-endian. So it doesn't appear to be an endian issue. https://developer.apple.com/documentation/apple-silicon/porting-your-macos-apps-to-apple-silicon -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On 28/12/21 04:39, Bart via lazarus wrote: On Mon, Dec 27, 2021 at 3:41 PM Juha Manninen via lazarus wrote: It must be a Big endian / Little endian issue. IIRC it can be adjusted in ARM CPUs. Why do MacOS and Linux use a different setting there? I have no idea. On second thought: if the function returns grabage for just a single '€', the code for that should not enter the pasrt where it handles blocks of size PtrInt and does masking with EIGHTYMASK etc. It's not just the euro, though. It's any utf-8 sequence. -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On Mon, Dec 27, 2021 at 6:35 PM Marco van de Voort via lazarus
wrote:
> The expression seems to be 1 when the top bits are 10 iow when it is a
> follow bytes of utf8, that is what the comment says, and I as far as I
> can see the signedness doesn't matter.
>
> Basically to me that seems to be a branchless version of
>
> if (p[i] and %1100)=%1000 then
>
> inc(result);
This is how I understood that paert of the code as well.
Just a side node: all this assumes that the UTF8 is correct (in the
strict sense).
Now for the part that tries to do calculations on blocks of 32 or 64 bits.
It uses a multiplication in that part.
That seems a bit odd as this code tries to do evrything with ands,
nots, and shifts.
Would this approach (in that part of the code) also work?
X = 32 or 64 bit block
1: AND X with EIGHTYMASK
2: SHR 7: gives (A)
3: NOT X
4: SHR 6: gives (B)
5: (A) AND (B): gives (C)
Basically we do the same as for a 1-byte block;
Now we have a pattern where any 1 tells us this byte was a following byte
A 32-bit example:
(Invalid sequence but that does not matter here)
X = %11xx %01xx %00xx %10xx
(Leading-ASCII-ASCII-Following, so count of following bytes must be 1)
AND with EIGTHYMASK gives %1000 % % %1000
SHR 7 gives: %0001 % % %0001 (A)
NOT %11xx %01xx %00xx %10xx = %00yy %10yy
%11yy %01yy
SHR 6 gives: % %yy10 %yy11 %yy01 (B)
(A) and (B) gives: % % % %0001 (C)
All non-following bytes turn into %
The count of following bytes is PopCnt(C)
As long as PopCnt is available as a machine instruction, this will be
faster than the (nx * ONEMASK) calculation I think.
(I would hope this is the case for all platforms Lazarus supports, if
not the call to PopCnt could be ifdef-ed.)
So basically change this:
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
{$push}{$overflowchecks off} // "nx * ONEMASK" causes an
arithmetic overflow.
Result += (nx * ONEMASK) >> ((sizeof(PtrInt) - 1) * 8);
{$pop}
into
nx := ((pnx^ and EIGHTYMASK) shr 7) and ((not pnx^) shr 6);
Result := Result + PopCnt(PtrUInt(nx)); /PopCnt only accepts
unsigend parameter
Since bit manipulating is not my strongpoint, please comment.
--
Bart
--
___
lazarus mailing list
lazarus@lists.lazarus-ide.org
https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
Op 12/27/2021 om 4:39 PM schreef Bart via lazarus: pn8^ =11100010 //first byte (pn8^ shr 7) = //<<-- I would have expected that to be 0001 ? Depends on if pn8^ is signed or not, for a signed shift it makes sense. The definition as pint8 (instead of puint8) is an odd choice. The expression seems to be 1 when the top bits are 10 iow when it is a follow bytes of utf8, that is what the comment says, and I as far as I can see the signedness doesn't matter. Basically to me that seems to be a branchless version of if (p[i] and %1100)=%1000 then inc(result); ...which counts all utf8 follow bytes, and then subtracts it from the number of bytes in a string to find the number of utf8 sequences/codepoints. Maybe the absolute stuff confuses somehow? Also make sure the input is 100% the same by printing the values of the bytes of the input string. -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
Am 27.12.2021 um 13:28 schrieb Bart via lazarus: On Mon, Dec 27, 2021 at 12:44 AM Noel Duffy via lazarus wrote: I need some help getting to the root of a problem with incorrect results on Apple hardware (M1, aarch64) for the function UTF8LengthFast in lazutf8. Your M1 architecture is BigEndian perhaps? (I really have no idea) No. It is little endian. -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On Mon, Dec 27, 2021 at 3:41 PM Juha Manninen via lazarus wrote: > It must be a Big endian / Little endian issue. IIRC it can be adjusted in ARM > CPUs. > Why do MacOS and Linux use a different setting there? I have no idea. On second thought: if the function returns grabage for just a single '€', the code for that should not enter the pasrt where it handles blocks of size PtrInt and does masking with EIGHTYMASK etc. (The part of the code that might be endianness dependant). It should go to one of the 2 loops that simply does: Result += (pn8^ shr 7) and ((not pn8^) shr 6); That part should not depend on endianness at all. On Win32 a sigle '€' will result in something like this: pn8^ =11100010 //first byte (pn8^ shr 7) = //<<-- I would have expected that to be 0001 ? (not pn8^)=00011101 (not pn8^) shr 6 = Add: (pn8^ shr 7) and ((not pn8^) shr 6)=0 pn8^ =1010 //second byte (pn8^ shr 7) = (not pn8^)=0101 (not pn8^) shr 6 =0001 Add: (pn8^ shr 7) and ((not pn8^) shr 6)=1 pn8^ =10101100 //third and last byte of '€' (pn8^ shr 7) = (not pn8^)=01010011 (not pn8^) shr 6 =0001 Add: (pn8^ shr 7) and ((not pn8^) shr 6)=1 B.t.w. I find the code in Utf8LengthFast difficult to read. Personally I dislike the C-ism of += and >> (even more so if both >> and shr is used). -- Bart -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On Mon, Dec 27, 2021 at 1:44 AM Noel Duffy via lazarus < lazarus@lists.lazarus-ide.org> wrote: > I need some help getting to the root of a problem with incorrect results > on Apple hardware (M1, aarch64) for the function UTF8LengthFast in lazutf8. > > On MacOS, when given a string containing one or more UTF8 characters, > UTF8LengthFast returns wildly incorrect results. On Fedora, the function > returns the correct answer. > You mean both MacOS and Fedora run on the same aarch64 CPU? It must be a Big endian / Little endian issue. IIRC it can be adjusted in ARM CPUs. Why do MacOS and Linux use a different setting there? I have no idea. If somebody can figure out how to port the function, good. Otherwise we can make UTF8LengthFast call the standard UTF8Length using an IFDEF when needed. What is the correct define then? Juha -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] SQLDBRestBridge memory consumption attn Michael
On Mon, 27 Dec 2021, zeljko wrote: Hi all, fpc-3.2.2, lazarus trunk...found that sqldbrestdispatcher does not free some unused resources, so sqldbrestserver memory alloc grows and after few thousand connections eats more than 1GB of memory. Just found in TSQLDBRestDispatcher.HandleResourceRequest that H: TSQLDBRestDBHandler is created for each request but freed when dispatcher is destroyed. So, dispatcher in this case can contain thousands of TSQLDBRestDBHandler unused components (could not find that dispatcher reuse any of this). I've added H.Free at last place in finally section and it seem that memory does not grow too much at all (still grows but slower, probably there's still unused resources around). Solution would be to reuse TSQLDBRestDBHandler's or free them. Pls look at https://forum.lazarus.freepascal.org/index.php/topic,51938.msg382021.html#msg382021 Forum is down, but I distinctly remember fixing this memleak. It was not on my main development machine, but in a Windows VM. Probably I forgot to commit that at the time :/ I applied a fix (free the handler), and fixed some compiler hints/warnings as well. I did a test, and I can't reproduce memleaks with the current demo application (the -m option exists for this very purpose), but that doesn't mean there are no memleaks left, there are many different execution paths :/ Michael. -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] Lazarus server maintenance
The mailing lists should be available again Marc On 24-12-2021 08:30, Marc Weustink via lazarus wrote: Hi, On Monday 27 December 9.00 CET (8.00 GMT) the Lazarus server will be down for maintenance. This affects the following services: * Lazarus website * Lazarus mailinglists * Lazarus online package manager * Lazarus and FreePascal forum Thanks, Marc -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
Re: [Lazarus] UTF8LengthFast returning incorrect results on AARCH64 (MacOS)
On Mon, Dec 27, 2021 at 12:44 AM Noel Duffy via lazarus wrote: > I need some help getting to the root of a problem with incorrect results > on Apple hardware (M1, aarch64) for the function UTF8LengthFast in lazutf8. Your M1 architecture is BigEndian perhaps? (I really have no idea) -- Bart -- ___ lazarus mailing list lazarus@lists.lazarus-ide.org https://lists.lazarus-ide.org/listinfo/lazarus
