Re: [fpc-devel] Changes to TProcess

[Wayne Sherman via fpc-devel]

On Wed, Jan 3, 2024 at 5:29 AM Michael Van Canneyt wrote:
> I merged a significant evolution of the TProcess component to FPC trunk...
> There is now a testsuite for the TProcess command, so everything was tested.

A much more flexible TProcess plus tests. Your work is appreciated. Thanks.  :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Interesting short article about optimisation

[Wayne Sherman via fpc-devel]

On Sat, Nov 25, 2023 at 3:10 PM J. Gareth Moreton wrote:
> I just stumbled across this article about micro-architecture-specific
> optimisations in ARM:
> https://www.phoronix.com/news/ARM64-Linux-No-Uarch-Opts
>
> They briefly mention x86_64, and I agree it's good to avoid
> micro-architecture-specific optimisations and now it makes me wonder
> where the line is drawn, like the recent introduction of the optimizer
> hint regarding LEA instructions being slow on some middle-aged CPUs.

For general optimization, direct most of the effort on the most
commonly used CPUs and architectures, while ensuring satisfactory
performance for others. Allow specific optimizations for atypical
cases, as long as they don't negatively impact the most common
scenarios. Before investing much effort, check what optimizations will
be accepted upstream (unless one is willing to maintain that code
separately).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Record representation clauses

[Wayne Sherman via fpc-devel]

On Sun, Nov 12, 2023 at 9:14 PM dave--- wrote:
> I am currently working on defining records to describe memory-mapped
> registers on the Raspberry Pi's arm/aarch64 processor. It is painful and
> brittle to do this in Pascal. Recent versions of Ada/GNAT allow one to
> define both field location and byte-order in a record representation clause.

Some short code examples that show what you are trying to do would be helpful.

ARM is little endian, does that differ from the byte-order of some registers?

Some relevant discussions:
https://forum.lazarus.freepascal.org/index.php/topic,46017.0.html
https://forum.lazarus.freepascal.org/index.php?topic=49492.0
https://forum.lazarus.freepascal.org/index.php/topic,19107.msg113485.html#msg113485
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] fnmatch (linux), where is the external code/symbol located?

[Wayne Sherman via fpc-devel]

On Wed, Oct 25, 2023 Alfred wrote:
> AFAIK, the GTK fnmatch can be found here:
> https://codebrowser.dev/gtk/include/fnmatch.h.html
> Additional source-link:
> https://gitlab.gnome.org/rburton/gdk-pixbuf/...

Ok, I searched again in the official/upstream repo and found the
function fnmatch and file fnmatch.h.  In 2002 fnmatch.c was changed to
remove the function name "fnmatch" and replace it with
"gtk_fnmatch_intern" and "_gtk_fnmatch".  In the same commit fnmatch.h
was deleted.  The fpc header import apparently did not pick up these
changes.

GTK commit:
https://gitlab.gnome.org/GNOME/gtk/-/commit/73b15ba391b3a533786e2a2f4f80274b80866822
https://gitlab.gnome.org/GNOME/gtk/-/commit/73b15ba391b3a533786e2a2f4f80274b80866822#8af485fae929f582a6202cf54a88b00a0cc9824b_60_78

System fnmatch wasn't going to be UTF-8 clean, neither was our version.

Fri Dec 13 17:45:40 2002  Owen Taylor  

* gtk/fnmatch.c gtk/gtkprivate.h gtk/gtkfilesel.c:
System fnmatch wasn't going to be UTF-8 clean, neither
was our version. Redo our fnmatch.c to be UTF-8, add
test cases, fix all sorts of bugs inherited
from the antique GNU fnmatch code. Change interface
to get rid of fnmatch.h constants. Fixes basic
non-workingness of filesel with non-ASCII filenames.

* gtk/fnmatch.h: No longer needed.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] fnmatch (linux), where is the external code/symbol located?

[Wayne Sherman via fpc-devel]

On Tue, Oct 17, 2023 at 12:32 AM Marco van de Voort wrote:
> - Since it has perfectly fine definition in libc, my guess that this is
> some old workaround of GTK2 for some defunct version (or multiple
> varying prototypes) of fnmatch, something might have been resolved a
> decade ago already.
>
> - the symbol is never used in Lazarus to my best knowledge. Simply
> omitting the line from the GTK2 header should allow you to continue with
> your patch evaluation.

fnmatch is a libc function and AFAICT, GTK never exported a function
with the name "fnmatch".  GTK has a file named fnmatch.c which
declares "_gtk_fnmatch" with different parameters than libc fnmatch.
There is no GTK header file named fnmatch.h.  GTK "_gtk_fnmatch" is
only exposed in gtkprivate.h (likely for private / internal use only).

_gtk_fnmatch current version:
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gtk/fnmatch.c#L249
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gtk/gtkprivate.h#L63

_gtk_fnmatch 2004 version (just before fpc did their gtk header conversion):
https://gitlab.gnome.org/GNOME/gtk/-/blob/3127f29ef626097dd861376f36a0914a20f4598c/gtk/fnmatch.c#L253
https://gitlab.gnome.org/GNOME/gtk/-/blob/3127f29ef626097dd861376f36a0914a20f4598c/gtk/gtkprivate.h#L102

To promote good housekeeping, and prevent strange linking behavior and
errors I propose the removal of the incorrect declaration from
./packages/gtk2/src/gtk+/gtk/fnmatch.inc:

function fnmatch(__pattern: AnsiChar; __string: AnsiChar;
  __flags: gint): gint; cdecl; external gtklib;

or remove the entire file (or move it out of gtk2 and put it where it belongs?):
./packages/gtk2/src/gtk+/gtk/fnmatch.inc

Trunk fpc fnmatch.inc version:
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/gtk2/src/gtk+/gtk/fnmatch.inc

Original 2005 fpc fnmatch.inc version:
https://gitlab.com/freepascal.org/fpc/source/-/blob/3c9295f2856b775a6db83a2fde40b9992b492cb9/packages/extra/gtk2/gtk+/gtk/fnmatch.inc
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] x86_64 SHA1 implementation

[Wayne Sherman via fpc-devel]

> assembly version of SHA-1 for x86_64

Another example which I thought is beautiful in form and simplicity (I
don't know how it compares for performance):
https://github.com/nayuki/Nayuki-web-published-code/blob/master/fast-sha1-hash-implementation-in-x86-assembly/sha1-fast-x8664.S
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] x86_64 SHA1 implementation

[Wayne Sherman via fpc-devel]

> OpenSSL x86 64-bit assembly code
>
https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-x86_64.pl

To generate plain assembly from the OpenSSL perl asm script requires
downloading these files to the same folder:
https://github.com/openssl/openssl/tree/master/crypto/perlasm
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] x86_64 SHA1 implementation

[Wayne Sherman via fpc-devel]

J. Gareth Moreton via fpc-devel  wrote:
> So this past week I've been building on Rika's work by adding an
> assembly version of SHA-1 for x86_64 to complement Rika's i386 version.
> So far I've successfully made a version that runs twice as fast as the
> Pascal code.  I hoped to go even faster by making use of the SSE2
> instruction set...

In 2010 Intel published SSE3 code to improve SHA1 performance.  Later
that year it was incorporated into OpenSSL ASM code.  The OpenSSL code
also includes AVX and SHA acceleration extensions.

Intel Article:
https://www.intel.com/content/www/us/en/developer/articles/technical/improving-the-performance-of-the-secure-hash-algorithm-1.html

Brief on Intel SHA extensions (also works for AMD Zen and later CPUs)
https://en.wikipedia.org/wiki/Intel_SHA_extensions

OpenSSL x86 64-bit assembly code and performance chart
https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-x86_64.pl

##
# Current performance is summarized in following table. Numbers are
# CPU clock cycles spent to process single byte (less is better).
#
#   x86_64 SSSE3AVX[2]
# P49.05   -
# Opteron   6.26   -
# Core2 6.55   6.05/+8% -
# Westmere  6.73   5.30/+27%-
# Sandy Bridge  7.70   6.10/+26%4.99/+54%
# Ivy Bridge6.06   4.67/+30%4.60/+32%
# Haswell   5.45   4.15/+31%3.57/+53%
# Skylake   5.18   4.06/+28%3.54/+46%
# Bulldozer 9.11   5.95/+53%
# Ryzen 4.75   3.80/+24%1.93/+150%(**)
# VIA Nano  9.32   7.15/+30%
# Atom  10.3   9.17/+12%
# Silvermont13.1(*)9.37/+40%
# Knights L 13.2(*)9.68/+36%8.30/+59%
# Goldmont  8.13   6.42/+27%1.70/+380%(**)
#
# (*) obviously suboptimal result, nothing was done about it,
# because SSSE3 code is compiled unconditionally;
# (**) SHAEXT result
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Division nodes

[Wayne Sherman via fpc-devel]

On Thu, May 11, 2023 at 11:42 AM J. Gareth Moreton wrote:
> This is the code block in question (ncnv.pas, starting at line 3397)

The git "blame" function shows who last made changes:
https://gitlab.com/freepascal.org/fpc/source/-/blame/main/compiler/ncnv.pas?page=4#L3396

Most of that code was added 2 years ago in this commit:
https://gitlab.com/freepascal.org/fpc/source/-/commit/ea11517d27fa00f40b626e47213f0caa8832d155
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Compiler optimization techniques

[Wayne Sherman via fpc-devel]

Explaining my fast 6502 code generator
"To learn how optimizing compilers are made, I built one targeting the
6502 architecture. In a bizarre twist, my compiler generates faster
code than GCC, LLVM, and every other compiler I compared it to..."

https://pubby.games/codegen.html
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Tue, Jan 17, 2023 at 7:08 AM Wayne Sherman wrote:
> A REST class should know how to generate a complete and correct JSON
> representation, so there must be a way to distinguish REST properties
> from non-REST.  Which properties have been modified is not sufficient.

Rephrasing that last sentence:  Tracking "which properties have been
modified" is sufficient for a limited use case.  But with a way to
distinguish REST properties and some other relatively small
adjustments, TBaseClass could be more generally useful for many other
use cases.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Tue, Jan 17, 2023 at 1:23 AM Michael Van Canneyt wrote:
> Yes. The reason is the code generator:
>
> Ideally, every property simply has a unique index specifier.
>
> But when generating classes, the code generator has no way of knowing how
> many properties exist in parent classes, so it must start at 0.
>
>(
>one could imagine helping the code generator by specifying a start point
>per class:
>1000
>2000
>3000
>etc.
>)

If non-REST properties are always not published the codegen could
generate unique index specifiers.  Also, if 0 is not used as an index
specifier, then any non-zero index specifier could be used to indicate
a REST property. (* but I have more to say about this below)

Wayne wrote:
> > function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
> > begin
> >  Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
> > end;
> >
> > The name list (from which NameIndex gets its value) includes the
> > published property names of all the parents plus the names in the
> > current class, so for a REST property in a descendant class with Index
> > specifiers starting at 0:
> >  GetParentPropCount+(IndexSpecifier shr IndexShift) = NameIndex

Michael wrote:
> Yes.
> This was long ago, but I seem to remember that I changed that code in 
> production,
> so it would use the same mechanism as MarkPropertyChanged, because there were
> published properties not part of the REST scheme, and in that case, the 
> equation does
> not hold true (becasue nameindex includes non-REST properties).
> That change does not seem to have made it in FPC.

A REST class should know how to generate a complete and correct JSON
representation, so there must be a way to distinguish REST properties
from non-REST.  Which properties have been modified is not sufficient.
I thought any property with an index specifier could be considered
REST, but the first property has an index specifier of 0 which matches
other properties with no index specifier.  One of the extra bit flags
in the index specifier could be used to indicate a REST property since
bits 0-2 seem to be reserved for future use.  But since the mechanism
for persisting objects is based on typeinfo and published properties,
I think the rule should be that ALL REST properties are published and
ALL non-REST properties are NOT published.  Why would you need to
publish any non-REST properties?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Mon, Jan 16, 2023 at 12:16 PM Wayne Sherman wrote:
> I need clarification about the auto generated class index
> specifiers.  Do they always start at 0 in each descendant
> class, or are they unique across all descendant classes?
>
> TBaseObject --> TChild --> TGrandChild
>
> TChild = class(TBaseObject)
> ...
>   published
> property Field1: string index 0...
> property Field2: string index 8...
>   end;
>
> TGrandChild = class(TChild)
> ...
>   published
> property Field3: string index 16...  // does this continue at 16
> or start at 0 again?
> property Field4: string index 24...
>   end;

It appears the index specifiers restart at 0 in each descendant class.
Which is why this code works:

procedure TBaseObject.MarkPropertyChanged(AIndex: Integer);
begin
  If Assigned(FBits) then
FBits.SetOn(GetParentPropCount+(AIndex shr IndexShift));
end;

function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
begin
  Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
end;

The name list (from which NameIndex gets its value) includes the
published property names of all the parents plus the names in the
current class, so for a REST property in a descendant class with Index
specifiers starting at 0:
  GetParentPropCount+(IndexSpecifier shr IndexShift) = NameIndex
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Sun, Jan 15, 2023 at 9:28 PM Michael Van Canneyt wrote:
> On Sun, 15 Jan 2023, Wayne Sherman wrote:
> > On Sun, Jan 15, 2023 at 9:24 AM Michael Van Canneyt wrote:
> >> On Sat, 14 Jan 2023, Wayne Sherman wrote:
> >>
> >>> I see a couple of problems TBaseObject.SaveToJSON
> >>>
> >>> 1) TBaseObject.SaveToJSON cannot distinguish properties that are part
> >>> of the REST protocol from properties that are not part of it.  It only
> >>> knows that properties which have been modified are part of the rest
> >>> protocol, but properties which have not been modified might be part of
> >>> the REST protocol or might not be.  For example, a client receives a
> >>> JSON object from a server (via LoadFromJSON) and wants to persist the
> >>> complete JSON object to disk, database, or send it to another server.
> >>> But, at present, there is no way to save all the properties that are
> >>> part of the REST protocol without getting contaminated with non-REST
> >>> properties.
> >>
> >> The non-REST properties will never be marked as changed, so they will not 
> >> be
> >> saved.
> >
> > LoadFromJSON calls StartRecordPropertyChanges and
> > StopRecordPropertyChanges.  Both of these destroy the change records.
> > At present, there is no opportunity to save modified records after
> > loading.  If that was fixed I think it would be a step in the right
> > direction.
>
> It seems to me we're talking cross-purpose. From my point of view,
> the code works as designed. It has been in use in production since many
> years (in fact, since it was committed to FPC).
>
> Can you please make a small example that demonstrates the problem you are
> experiencing, so I can look at it ?

Ok, I will see if I can create a demo.  I need clarification about the
auto generated class index specifiers.  Do they always start at 0 in
each descendant class, or are they unique across all descendant
classes?

TBaseObject --> TChild --> TGrandChild

TChild = class(TBaseObject)
...
  published
property Field1: string index 0...
property Field2: string index 8...
  end;

TGrandChild = class(TChild)
...
  published
property Field3: string index 16...  // does this continue at 16
or start at 0 again?
property Field4: string index 24...
  end;
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Sun, Jan 15, 2023 at 9:24 AM Michael Van Canneyt wrote:
> On Sat, 14 Jan 2023, Wayne Sherman wrote:
>
> > I see a couple of problems TBaseObject.SaveToJSON
> >
> > 1) TBaseObject.SaveToJSON cannot distinguish properties that are part
> > of the REST protocol from properties that are not part of it.  It only
> > knows that properties which have been modified are part of the rest
> > protocol, but properties which have not been modified might be part of
> > the REST protocol or might not be.  For example, a client receives a
> > JSON object from a server (via LoadFromJSON) and wants to persist the
> > complete JSON object to disk, database, or send it to another server.
> > But, at present, there is no way to save all the properties that are
> > part of the REST protocol without getting contaminated with non-REST
> > properties.
>
> The non-REST properties will never be marked as changed, so they will not be
> saved.

LoadFromJSON calls StartRecordPropertyChanges and
StopRecordPropertyChanges.  Both of these destroy the change records.
At present, there is no opportunity to save modified records after
loading.  If that was fixed I think it would be a step in the right
direction.

Ref:
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/restbase.pp#L1148

Taking a step further, it would be advantageous if TBaseObject and
descendants know specifically which of their properties are REST
protocol properties independent of whether they have been modified or
not.  For example, if I want to dump the JSON for an empty object to
see the fields, I would like to do this:

KeepNote := TNote.Create(nil);  //Google Keep Note
KeepNote.SaveAsJSON(saveAllProperties);

{
  "name": "",
  "createTime": "",
  "updateTime": "",
  "trashTime": "",
  "trashed": false,
  "attachments": [
{ }
  ],
  "permissions": [
{ }
  ],
  "title": "",
  "body": { }
}
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

I see a couple of problems TBaseObject.SaveToJSON

1) TBaseObject.SaveToJSON cannot distinguish properties that are part
of the REST protocol from properties that are not part of it.  It only
knows that properties which have been modified are part of the rest
protocol, but properties which have not been modified might be part of
the REST protocol or might not be.  For example, a client receives a
JSON object from a server (via LoadFromJSON) and wants to persist the
complete JSON object to disk, database, or send it to another server.
But, at present, there is no way to save all the properties that are
part of the REST protocol without getting contaminated with non-REST
properties.

2) Lack of control.  With TBaseObject.SaveToJSON it would be nice to
control what is saved (Save all REST properties, or Save modified REST
properties).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

Michael wrote:
> >> Because there may be other published properties that are not part of the
> >> REST protocol. You may for instance decide to add published properties to a
> >> base class that describe where to save the object in a local database.

Wayne wrote:
> > Ok, that clears up that.  If I understand correctly, the behavior
> > should be as follows:
> >
> > (restbase.pp)
> > In TBaseObject descendants, SaveToJSON only saves object properties
> > defined directly in the final descendant class and none of the
> > properties of parent classes.  This is implemented by only marking
> > properties as modified that are defined directly in the final
> > descendant class and skipping properties of all parent classes (which
> > always appear to be unmodified, and thus not saved).  This prevents
> > properties which are not part of the REST protocol from contaminating
> > the generated JSON.  It follows that REST protocol properties should
> > always be defined in the final TBaseObject descendant class (and not
> > in any base or intermediate classes).

Michael wrote:
> While in practice most REST classes will be the final classes,
> it should be possible to support inheritance in rest classes,
> that is why the 'ParentPropertyCount' exists and is taken into account.

My summary above is incorrect.  How about this?:

In TBaseObject descendants, SaveToJSON only saves object properties
which have had their properties marked as modified using
MarkPropertyChanged (modified flags are stored internally using
TBits).  Properties which have not been marked as modified will not be
saved to JSON.  This prevents properties which are not part of the
REST protocol from contaminating the generated JSON.

Auto generated code that defines TBaseObject descendant classes
declares properties with index specifiers and setter methods that call
MarkPropertyChanged.  When a property is changed in these auto
generated classes, that property will end up in the output of
SaveToJSON.  Properties which are a valid part of the REST protocol,
but which have not been changed are not included in the generated
JSON.

Properties can be defined in a TBaseObject final descendant class or
parent classes and be included in the JSON output (as long as
MarkPropertyChanged has been called with the corresponding index for
that property).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Sat, Jan 14, 2023 at 12:36 PM Michael Van Canneyt wrote:
> On Sat, 14 Jan 2023, Wayne Sherman wrote:
> > On Sat, Jan 14, 2023 at 10:34 AM Michael Van Canneyt wrote:
> >> On Sat, 14 Jan 2023, Wayne Sherman wrote:
> >>> 2) Doesn't each object already have the parent properties included in
> >>> its own properties by inheritance?
> >>
> >> Why this question ?
> >> An object does not know the property count of the parent.
> >
> > True, but why does it need to know the property count of the parent?
> >
> >  TBaseClass = class
> >  private
> >FField1: integer;
> >  protected
> >FPropModifiedFlags: TBits;
> >  published
> >property Field1: integer read FField1 write FField1;
> >  end;
> >
> > GetTypeData(TBaseClass.ClassInfo)^.PropCount = 1
> >
> >  TDescendant = class(TBaseClass)
> >  private
> >FField2: integer;
> >  published
> >property Field2: integer read FField2 write FField2;
> >  end;
> >
> > GetTypeData(TDescendant.ClassInfo)^.PropCount = 2
> >
> > TDescendant has two properties (one inherited and one declared
> > directly) and let's say it can record if any of the properties have
> > been modified.  So why does it have to know about the number of
> > properties it inherited vs the properties it declares directly?
>
> Because there may be other published properties that are not part of the
> REST protocol. You may for instance decide to add published properties to a
> base class that describe where to save the object in a local database.

Ok, that clears up that.  If I understand correctly, the behavior
should be as follows:

(restbase.pp)
In TBaseObject descendants, SaveToJSON only saves object properties
defined directly in the final descendant class and none of the
properties of parent classes.  This is implemented by only marking
properties as modified that are defined directly in the final
descendant class and skipping properties of all parent classes (which
always appear to be unmodified, and thus not saved).  This prevents
properties which are not part of the REST protocol from contaminating
the generated JSON.  It follows that REST protocol properties should
always be defined in the final TBaseObject descendant class (and not
in any base or intermediate classes).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Sat, Jan 14, 2023 at 10:34 AM Michael Van Canneyt wrote:
> On Sat, 14 Jan 2023, Wayne Sherman wrote:
> > 3) Why does MarkPropertyChanged offset the index and
> > IsPropertyModified does not take the offset into account?
>
> They should be the same. I don't remember why I created the two methods
> differently.

For property index specifiers, it seems that NameIndex might be the
wrong index used in IsPropertyModified:

function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
begin
  Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
end;

and possibly the rtl docs are incorrect(?):
   https://www.freepascal.org/docs-html/rtl/typinfo/tpropinfo.html
   Index - Index for array properties
   NameIndex - Index for indexed properties  << IS THIS CORRECT?

In the typinfo.pp unit, a comment indicates that NameIndex has to do
with property names (i.e. not the index specifier)
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/rtl/objpas/typinfo.pp#L1689
// Don't overwrite properties with the same name
if PropList^[TP^.NameIndex]=nil then
  PropList^[TP^.NameIndex]:=TP;
// Point to TP next propinfo record.
// Located at Name[Length(Name)+1] !
TP:=aligntoptr(PPropInfo(pointer(@TP^.Name)+PByte(@TP^.Name)^+1));
Dec(Count);
  end;

NameIndex in Delphi typeinfo also has to do with Names of properties
and Index has the information on indexed properties:
   https://docwiki.embarcadero.com/Libraries/Sydney/en/System.TypInfo.TPropInfo
   Index - The index of the property. It is used as a parameter to the
GetProc and SetProc methods on indexed properties.
   NameIndex - The offset into the Name field where the property's name starts.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Sat, Jan 14, 2023 at 10:34 AM Michael Van Canneyt wrote:
> On Sat, 14 Jan 2023, Wayne Sherman wrote:
> > 2) Doesn't each object already have the parent properties included in
> > its own properties by inheritance?
>
> Why this question ?
> An object does not know the property count of the parent.

True, but why does it need to know the property count of the parent?

  TBaseClass = class
  private
FField1: integer;
  protected
FPropModifiedFlags: TBits;
  published
property Field1: integer read FField1 write FField1;
  end;

GetTypeData(TBaseClass.ClassInfo)^.PropCount = 1

  TDescendant = class(TBaseClass)
  private
FField2: integer;
  published
property Field2: integer read FField2 write FField2;
  end;

GetTypeData(TDescendant.ClassInfo)^.PropCount = 2

TDescendant has two properties (one inherited and one declared
directly) and let's say it can record if any of the properties have
been modified.  So why does it have to know about the number of
properties it inherited vs the properties it declares directly?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

On Fri, Jan 13, 2023 at 11:48 PM Michael Van Canneyt wrote:
> Markpropertychanged is called in the setter of the properties generated by
> the code generator: Check all generated units.

Thanks for your explanations, it is getting more clear how modified
property tracking is being used.

More clarification please.  MarkPropertyChanged offsets the property
changed index (to TBits) by the ClassParent total property count, but
IsPropertyModified does not take this offset into account.  Consider
this example:

TBaseObject --> TGoogleBaseObject --> TSchema --> TMySchema
TBaseObjectClass = Class of TBaseObject;

If the object is TMySchema, in GetParentPropCount the TBits index gets
offset by (see reference code below):
  TBaseObjectClass(TSchema).GetTotalPropCount;
If the object is TSchema, the TBits index gets offset by:
  TBaseObjectClass(TGoogleBaseObject).GetTotalPropCount;

Questions:
1) Since the ClassParent is always cast as TBaseObjectClass isn't this
the same as just TBaseObjectClass(Self.ClassType)?

2) Doesn't each object already have the parent properties included in
its own properties by inheritance?

3) Why does MarkPropertyChanged offset the index and
IsPropertyModified does not take the offset into account?

Reference for above questions:

class function TBaseObject.GetParentPropCount: Integer;
begin
  if (ClassParent=TBaseObject) or (ClassParent=Nil) then
Result:=0
  else
Result:=TBaseObjectClass(ClassParent).GetTotalPropCount;
end;

procedure TBaseObject.MarkPropertyChanged(AIndex: Integer);
begin
  If Assigned(FBits) then
FBits.SetOn(GetParentPropCount+(AIndex shr IndexShift));
end;

function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
begin
  Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
end;


___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

Is there a better way to track changed TBaseObject properties (and get
rid of property index declarations)?

Lazarus knows which properties on a form have been changed from their
default value and only properties which do not match the default are
saved in the lfm file.  Can the same thing be used with TBaseObject
properties?

On Fri, Jan 13, 2023 at 8:59 PM Wayne Sherman  wrote:
>
> Some related misunderstandings:
>
> 1) When StopRecordPropertyChanges is called it destroys all the
> change records (stored in TBits) in which case SaveToJSON saves ALL
> the properties even if they were not loaded or modified previously.
>
> 2) Nothing in restbase.pp or TBaseObject calls MarkPropertyChanged
> anyway, so no properties will get marked as modified during
> TGoogleRestDescription.LoadFromJSON.
>
> procedure TBaseObject.MarkPropertyChanged(AIndex: Integer);
> begin
>   If Assigned(FBits) then
> FBits.SetOn(GetParentPropCount+(AIndex shr IndexShift));
> end;
>
> 3) Both MarkPropertyChanged (above) and IsPropertyModified (below)
> require the properties declarations to have indexes and
> TGoogleRestDescription does not have indexes on any of its properties.
>
> function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
> begin
>   Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
> end;
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

Some related misunderstandings:

1) When StopRecordPropertyChanges is called it destroys all the
change records (stored in TBits) in which case SaveToJSON saves ALL
the properties even if they were not loaded or modified previously.

2) Nothing in restbase.pp or TBaseObject calls MarkPropertyChanged
anyway, so no properties will get marked as modified during
TGoogleRestDescription.LoadFromJSON.

procedure TBaseObject.MarkPropertyChanged(AIndex: Integer);
begin
  If Assigned(FBits) then
FBits.SetOn(GetParentPropCount+(AIndex shr IndexShift));
end;

3) Both MarkPropertyChanged (above) and IsPropertyModified (below)
require the properties declarations to have indexes and
TGoogleRestDescription does not have indexes on any of its properties.

function TBaseObject.IsPropertyModified(Info: PPropInfo): Boolean;
begin
  Result:=Not Assigned(FBits) or FBits.Bits[Info^.NameIndex]
end;
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] restbase.pp LoadFromJSON calling StopRecordPropertyChanges;

[Wayne Sherman via fpc-devel]

I am trying to check the googleapiconv "TGoogleRestDescription" object
contents (which is descended from restbase TBaseObject) by using
"SaveToJSON".  But it always saves an empty object, even when the
properties have values assigned to them (via LoadFromJSON).

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/googleapi/generator/googlediscoverytopas.pp#L303

SaveToJSON calls SavePropertyToJSON which checks IsPropertyModified:

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/restbase.pp#L1310

If the property has not been modified it is skipped and not saved.
But ALL of the properties are being skipped since none of them have
been marked as modified (even though they were actually modified
during the loading).

The reason this happens is because LoadFromJSON calls
"StopRecordPropertyChanges" before loading and none of the properties
get marked as "changed".

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/restbase.pp#L1333

Is this the correct behavior?  It seems to me the code should be:

  StartRecordPropertyChanges;
  For D in JSON Do
LoadPropertyFromJSON(D.Key,D.Value);
  StopRecordPropertyChanges;

(or alternately get rid of the Start/Stop record property calls
completely from LoadFromJSON and let the ObjectOptions control it:
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/restbase.pp#L559
TBaseObject.Create sets ooStartRecordingChanges by default.)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] googleapiconv issues

[Wayne Sherman via fpc-devel]

The google api binding generator "googleapiconv" is generating empty
files for the api pascal bindings and it also causes access violations
randomly.  I have not found the issue with AVs yet, but I found out
why it is creating empty source code files.

restcodegen.pp TRestCodeGenerator was refactored in 2018 to use pascodegen:
- TRestCodeGenerator = Class(TComponent)
+ TRestCodeGenerator = Class(TPascalCodeGenerator)

ref:  
https://gitlab.com/freepascal.org/fpc/source/-/commit/d7fa0b19988f753873f498b9424dcf82ba1b271e#b4a1b63d04b721fa6db35bc226d44a63a21fda40

Before the refactoring, TRestCodeGenerator.SaveToStream used to call
"Execute;" if the source code string list was empty:
procedure TRestCodeGenerator.SaveToStream(const AStream : TStream);
begin
  if (FSource.Count=0) then
Execute;
  FSource.SaveToStream(AStream)
end;

ref:  
https://gitlab.com/freepascal.org/fpc/source/-/blob/0eddef3d093c53fca0526b7bc558ffeef42a79df/packages/fcl-web/src/base/restcodegen.pp#L186

Now "Execute;" is not being called and no code is generated because
pascodegen.pp "TPascalCodeGenerator.SaveToStream" looks like this:
procedure TPascalCodeGenerator.SaveToStream(const AStream : TStream);
begin
  FSource.SaveToStream(AStream)
end;

ref:  
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-base/src/pascodegen.pp#L255

Two options to fix this are:
1)  Add the "FSource" check and "Execute;" statements to
TPascalCodeGenerator.SaveToStream
  or
2) Add "Execute;" to TGoogleAPIConverter.DoConversion
(googleapiconv.pp) before it calls "SaveToStream" here:
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/googleapi/generator/googleapiconv.pp#L575

Question:  Do you prefer I discuss issues like this here on the
mailing list or instead open a bug report and discuss in the bug
report?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Make file questions (Makefile.fpc, fpmake.pp, etc)

[Wayne Sherman via fpc-devel]

On Tue, Jan 3, 2023 at 12:17 AM Sven Barth wrote:
>> 1) How do I generate the fpmake.pp file?  Or is it created by manual coding?
>
> This is created manually and is *the* source for how to build a package.
>
>> 2) Does "Makefile" automatically get generated during a build or do I
>> have to manually run FPCMake?
>
> You need to run fpcmake manually.

Got it.  So after moving files and/or directories around in
packages/some-package/ one has to:
  1) Edit fpmake.pp to make sure it has the correct paths and files
  2) Fix up any examples or lazarus project files with the correct
paths. (if needed)
  3) Run fpcmake -w -Tall from base package directory to regenerate Makefile

Thanks for the assistance.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Make file questions (Makefile.fpc, fpmake.pp, etc)

[Wayne Sherman via fpc-devel]

Each package directory (./fpc-source/packages/some-package/) typically
has these files:
   fpmake.pp  (where does this come from?)
   Makefile  (this is generated by FPCMake?)
   Makefile.fpc  (config file for FPCMake to create Makefile?)

Some package directories also have:
   Makefile.fpc.fpcmake  (?)

Some questions please:
1) How do I generate the fpmake.pp file?  Or is it created by manual coding?
2) Does "Makefile" automatically get generated during a build or do I
have to manually run FPCMake?
3) What is "Makefile.fpc.fpcmake" used for and how is it created?
4) Can "fpmake" completely build fpc and all the packages without using "make"?

Thanks in advance,
Wayne Sherman
(developing on Xubuntu 20.04 64-bit)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Incompatible assignments but no compile error (char array and shortstring)

[Wayne Sherman via fpc-devel]

Gareth aka. Kit wrote:
> What happens if you run the program with these
> false negatives?  If you get a buffer overrun, then
> it is pretty serious.

When running in debug mode, I get no range check or overflow errors.
A 9-byte sized value when assigned to an 8-byte sized variable gets
truncated to 8-bytes.

I missed these at first, but when I compile I am getting these warnings:
const
  ShortStringConst8c: TShortString8 = '123456789';   //did not get
expected compile error (bug?)
  (unit1.pas(53,50) Warning: String "123456789" is longer than "8")

  ShortStringConst8d: TShortString8 = StringConst9;  //did not get
expected compile error (bug?)
  (unit1.pas(54,51) Warning: String "123456789" is longer than "8")

begin
...
  ShortStringVar8 := '123456789';   //did not get expected compile error (bug?)
  (unit1.pas(92,19) Warning: String literal has more characters than
short string length)

But no compile warnings or errors on these:
begin
...
  CharArrayVar8 := '123456789';   //did not get expected compile
errors or warnings
  CharArrayVar8 := StringConst9;  //did not get expected compile
errors or warnings
  ShortStringVar8 := CharArrayVar9; //did not get expected compile
errors or warnings
  ShortStringVar8 :=
TCharArray9(['1','2','3','4','5','6','7','8','9']); //did not get
expected compile errors or warnings
  ShortStringVar8 := CharArrayConst9;  //did not get expected compile
errors or warnings

So it looks like the compiler does not treat string compatible types
as strictly as other types and when assigning it copies as many
characters as will fit.  But it is not consistent in generating
warnings.


On Sat, Apr 16, 2022 at 2:24 PM J. Gareth Moreton via fpc-devel
 wrote:
>
> It does look like a bug on the surface.  What happens if you run the
> program with these false negatives?  If you get a buffer overrun, then
> it is pretty serious.
>
> Gareth aka. Kit
>
> On 16/04/2022 21:35, Wayne Sherman via fpc-devel wrote:
> > Tested with fpc 3.3.1 trunk (as of 2022-Mar-12) and 3.2.2 stable.
> > Ubuntu 20.04 64-bit
> >
> > Good compile time error checking is one of the wonderful things about
> > Pascal.  But some char array and shortstring assignments which are not
> > size compatible do not produce a compile error as expected.  Please
> > see below.
> >
> > {$mode objfpc}{$H+}
> > type
> >TCharArray8= packed array[0..7] of Char;
> >TCharArray9= packed array[0..8] of Char;
> >TShortString8  = string[8];
> >
> > const
> >StringConst8 = '12345678';
> >StringConst9 = '123456789';
> >
> >CharArrayConst8a: TCharArray8 = '12345678';
> >CharArrayConst8b: TCharArray8 = StringConst8;
> >CharArrayConst8c: TCharArray8 = ('1','2','3','4','5','6','7','8');
> >
> >//CharArrayConst8d: TCharArray8 = '123456789';   //compile error as 
> > expected
> >//CharArrayConst8e: TCharArray8 = StringConst9;  //compile error as 
> > expected
> >//CharArrayConst8f: TCharArray8 =
> > ('1','2','3','4','5','6','7','8','9'); //compile error as expected
> >
> >ShortStringConst8a: TShortString8 = '12345678';
> >ShortStringConst8b: TShortString8 = StringConst8;
> >ShortStringConst8c: TShortString8 = '123456789';   //did not get
> > expected compile error (bug?)
> >ShortStringConst8d: TShortString8 = StringConst9;  //did not get
> > expected compile error (bug?)
> >
> > var
> >CharArrayVar8:   TCharArray8;
> >CharArrayVar9:   TCharArray9;
> >ShortStringVar8: TShortString8;
> >
> > begin
> >CharArrayVar8 := '12345678';
> >CharArrayVar8 := StringConst8;
> >CharArrayVar8 := ['1','2','3','4','5','6','7','8'];  //requires fpc 3.3.1
> >CharArrayVar8 := CharArrayConst8c;
> >
> >//CharArrayVar8 := ['1','2','3','4','5','6','7','8','9']; //compile
> > error as expected
> >CharArrayVar8 := '123456789';   //did not get expected compile error 
> > (bug?)
> >CharArrayVar8 := StringConst9;  //did not get expected compile error 
> > (bug?)
> >
> >CharArrayVar8 := ShortStringVar8;
> >ShortStringVar8 := CharArrayVar

[fpc-devel] Incompatible assignments but no compile error (char array and shortstring)

[Wayne Sherman via fpc-devel]

Tested with fpc 3.3.1 trunk (as of 2022-Mar-12) and 3.2.2 stable.
Ubuntu 20.04 64-bit

Good compile time error checking is one of the wonderful things about
Pascal.  But some char array and shortstring assignments which are not
size compatible do not produce a compile error as expected.  Please
see below.

{$mode objfpc}{$H+}
type
  TCharArray8= packed array[0..7] of Char;
  TCharArray9= packed array[0..8] of Char;
  TShortString8  = string[8];

const
  StringConst8 = '12345678';
  StringConst9 = '123456789';

  CharArrayConst8a: TCharArray8 = '12345678';
  CharArrayConst8b: TCharArray8 = StringConst8;
  CharArrayConst8c: TCharArray8 = ('1','2','3','4','5','6','7','8');

  //CharArrayConst8d: TCharArray8 = '123456789';   //compile error as expected
  //CharArrayConst8e: TCharArray8 = StringConst9;  //compile error as expected
  //CharArrayConst8f: TCharArray8 =
('1','2','3','4','5','6','7','8','9'); //compile error as expected

  ShortStringConst8a: TShortString8 = '12345678';
  ShortStringConst8b: TShortString8 = StringConst8;
  ShortStringConst8c: TShortString8 = '123456789';   //did not get
expected compile error (bug?)
  ShortStringConst8d: TShortString8 = StringConst9;  //did not get
expected compile error (bug?)

var
  CharArrayVar8:   TCharArray8;
  CharArrayVar9:   TCharArray9;
  ShortStringVar8: TShortString8;

begin
  CharArrayVar8 := '12345678';
  CharArrayVar8 := StringConst8;
  CharArrayVar8 := ['1','2','3','4','5','6','7','8'];  //requires fpc 3.3.1
  CharArrayVar8 := CharArrayConst8c;

  //CharArrayVar8 := ['1','2','3','4','5','6','7','8','9']; //compile
error as expected
  CharArrayVar8 := '123456789';   //did not get expected compile error (bug?)
  CharArrayVar8 := StringConst9;  //did not get expected compile error (bug?)

  CharArrayVar8 := ShortStringVar8;
  ShortStringVar8 := CharArrayVar8;

  ShortStringVar8 := '12345678';
  ShortStringVar8 := CharArrayConst8a;
  ShortStringVar8 := TCharArray8(['1','2','3','4','5','6','7','8']);
//requires fpc 3.3.1

  ShortStringVar8 := '123456789';   //did not get expected compile error (bug?)
  ShortStringVar8 := CharArrayVar9; //did not get expected compile error (bug?)
  ShortStringVar8 :=
TCharArray9(['1','2','3','4','5','6','7','8','9']); //did not get
expected compile error (bug?) (requires fpc 3.3.1)

end;

Should I open a couple of bug reports?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Mon, May 10, 2021 at 7:08 PM Wayne Sherman wrote:
> The Google API supports the following signing algorithms:
> https://cloud.google.com/iot/docs/how-tos/credentials/jwts
>
>JWT RS256 (RSASSA-PKCS1-v1_5 using SHA-256 RFC 7518 sec 3.3). This
> is expressed as RS256 in the alg field in the JWT header.
>JWT ES256 (ECDSA using P-256 and SHA-256 RFC 7518 sec 3.4), defined
> in OpenSSL as the prime256v1 curve. This is expressed as ES256 in the
> alg field in the JWT header.

My statement above about ES256 does not appear to be correct.  After
looking over Google's API docs again, I discovered that JWTs signed
using ES256 are only supported on a few of their services (?).  (e.g.
Cloud IoT Core, Identity-Aware Proxy, and Cloud Security Token)

ES256 support was added to the google API python auth library here:
 https://github.com/googleapis/google-auth-library-python/pull/340

 See also:
 https://github.com/googleapis/google-auth-library-python/search?q=es256
 https://github.com/googleapis/google-api-dotnet-client/search?q=es256

But according to this page:

 *Computing the signature*
 
https://developers.google.com/identity/protocols/oauth2/service-account#httprest

 "The signing algorithm in the JWT header must be used when computing the
 signature. The *only signing algorithm supported* by the Google OAuth 2.0
 Authorization Server *is RSA using SHA-256 hashing* algorithm. This is
 expressed as RS256 in the alg field in the JWT header."
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Mon, Mar 14, 2022 at 3:45 AM Michael Van Canneyt
 wrote:
> Yes. I would still like to see if we can integrate the OAuth of the google 
> API and
> the JWT signing more closely, possibly through some extra classes.

The paint is still wet on the JWT signing code, so I figured there was
some more integration to do.  I started investigating the "googleapiconv"
program, the googleapi examples, and generated API files (which are
about 5-years old).  I discovered some issues.  I will send a report in
another email.

> There is more to come, but I am doing a round-robin on the many areas I am 
> actively working on
> in, so it may be some weeks before I come back to this... :-)

Understood.  Thanks for your work.

Wayne
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Tue, Jun 30, 2020 at 12:47 AM Michael Van Canneyt
 wrote:
>
> The problem with the service account is that you must create a JWT Token.
> FPC does not yet have a unit that can generate *and sign* a JWT Token.

I had a little time this weekend and started looking into this again.
I was happy to discover these recent developments:

 *Some new hash and elliptic curve crypto files*
 
https://gitlab.com/freepascal.org/fpc/source/-/tree/main/packages/fcl-hash/src

 *Commit which adds support for Java Web Token (JWT) signing*
 
https://gitlab.com/freepascal.org/fpc/source/-/commit/47610d5c651a232a2061a44e2358b147ba65ff1c
 (fpjwt.pp has new TJWTKey record, TJWTSigner class, and new
methods TJWT.Sign and TJWT.ValidateJWT)

 *Commit which adds ES256 JWT signing*
 
https://gitlab.com/freepascal.org/fpc/source/-/commit/5afaeaa3ac813e5682d0da9de7d17454a4771d2c
 (ES256 signing can be used for Google API service accounts)

Thank you Michael Van Canneyt.  :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Missing bootstrap compilers and feature request

[Wayne Sherman via fpc-devel]

Most stand alone compilers for bootstrapping a new system are missing
from recent releases:

   ftp://ftp.freepascal.org/pub/fpc/dist/3.0.4/bootstrap/  [has 7
bootstrap compilers]
   ftp://ftp.freepascal.org/pub/fpc/dist/3.2.0/bootstrap/  [has 2
bootstrap compilers]
   ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/bootstrap/  [emply]

I have a feature request. Instead of having a
"...dist/[fpc_version]/bootstrap" directory, use the cpu-os directory
and automatically create a bootstrap compiler archive during the build
process.  Like this:

ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/aarch64-linux/fpc-3.2.2.aarch64-linux_bootstrap.gz
ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/arm-android/fpc-3.2.2.arm-android_bootstrap.gz
ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/x86_64-linux/fpc-3.2.2.x86_64-linux_bootstrap.gz
...

Or maybe use the same archive name for each:
ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/aarch64-linux/bootstrap_compiler.gz
ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/arm-android/bootstrap_compiler.gz
ftp://ftp.freepascal.org/pub/fpc/dist/3.2.2/x86_64-linux/bootstrap_compiler.gz
...

Having a consistent location and naming convention would make it
easier to create automated build systems that download and bootstrap a
new fpc and lazarus build directly from git on a new system.  The
alternative is to download a large archive (e.g. ~83 MB for
fpc-3.2.2.x86_64-linux.tar) and extract a bootstrap compiler (e.g.
ppcx64 is only ~4 MB) from deep inside (three levels deep of
archives).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] How does FPC perform in the FASTEST Computer Language Race

[Wayne Sherman via fpc-devel]

If you are a speed geek, you will probably be interested in this.

What is the FASTEST Computer Language? 45 Languages Tested! (E01)
https://www.youtube.com/watch?v=tQtFdsEcK_s

Current standings at the time of this video
Iterations per sec:
Ada:  67
Pascal: 143
Fastest language: 7301
Slowest language: 1

"HD Episode 01: Retired Microsoft engineer Dave Plummer takes you on a
guided tour of 45 different computer languages and drag races each
against one another using a prime sieve benchmark.  From Ada to Zig
and everything in between, find out which is fastest and which is
slowest."

Github:  https://github.com/PlummersSoftwareLLC/Primes

Current Pascal code:
https://github.com/PlummersSoftwareLLC/Primes/tree/drag-race/PrimePascal/solution_1
Current Delphi code:
https://github.com/PlummersSoftwareLLC/Primes/tree/drag-race/PrimeDelphi/solution_1

Pull requests for faster/better implementations:
https://github.com/PlummersSoftwareLLC/Primes/pulls

Related:
https://components4developers.blog/2021/07/04/delphi-sieve-candidates-to-the-fastest-programming-language-contest/
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Tue, Jun 30, 2020 at 12:47 AM Michael Van Canneyt
 wrote:
>
> The problem with the service account is that you must create a JWT Token.
> FPC does not yet have a unit that can generate *and sign* a JWT Token.
>
> The good news is I have this weekend a reminder that I received some code
> that will allow me to complete the JWT Token support in FPC using at
> least the RSA256 signing algorithm (and some others as well).

Hi Michael.  I would like to assist with this if possible.

According to rfc7519:
https://datatracker.ietf.org/doc/html/rfc7519#section-8

The *required* JWT Signature and MAC algorithms are:
   HS256 (HMAC SHA-256)
   none

The *recommended* JWT Signature and MAC algorithms are:
   RS256 (RSASSA-PKCS1-v1_5 with the SHA-256 hash)
   ES256 (ECDSA using the P-256 curve and the SHA-256 hash)

The Google API supports the following signing algorithms:
https://cloud.google.com/iot/docs/how-tos/credentials/jwts

   JWT RS256 (RSASSA-PKCS1-v1_5 using SHA-256 RFC 7518 sec 3.3). This
is expressed as RS256 in the alg field in the JWT header.
   JWT ES256 (ECDSA using P-256 and SHA-256 RFC 7518 sec 3.4), defined
in OpenSSL as the prime256v1 curve. This is expressed as ES256 in the
alg field in the JWT header.

So for google api, we need at least RS256 (RSASSA-PKCS1-v1_5 using SHA-256).

Adding signing directly to fpjwt.pp would be the cleanest, but you
need to add native crypto code to fpc for that.
https://svn.freepascal.org/svn/fpc/trunk/packages/fcl-web/src/base/fpjwt.pp

There are libraries that have it:
https://github.com/fundamentalslib/fundamentals5
https://github.com/Xor-el/CryptoLib4Pascal

Adding JWT signing to google API units could be done with OpenSSL and
it is already a dependency.
OpenSSL can sign and verify a JWT using both RS256 and ES256. Examples:
https://stackoverflow.com/questions/58313106/create-rs256-jwt-in-bash
https://stackoverflow.com/questions/40559765/how-to-verify-json-web-tokens-with-openssl
https://learn.akamai.com/en-us/webhelp/iot/jwt-access-control/GUID-054028C7-1BF8-41A5-BD2E-A3E00F6CA550.html

What do you think?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Google API service acct and JWT

[Wayne Sherman via fpc-devel]

Michael Van Canneyt, there is a discussion about using fphttpclient
with Google API service account on the forums:
https://forum.lazarus.freepascal.org/index.php/topic,51478.0.html

The last post has a code example of signing a JWT:
https://forum.lazarus.freepascal.org/index.php/topic,51478.msg381426.html#msg381426

Is that basically what is needed to get it working?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] OpenSSL v1.1.x DLL loading / Bug 37137

[Wayne Sherman via fpc-devel]

On Mon, Sep 28, 2020 at 6:34 AM Michael Van Canneyt
 wrote:
> I made some time for this, it's checked in with revision 46987.
>
> Thank you for your patch, and I'm sorry that it took me so long to get to 
> it...

No worries, thank you so much for your work on this.  Much appreciated. :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] OpenSSL v1.1.x DLL loading / Bug 37137

[Wayne Sherman via fpc-devel]

On Tue, Jun 30, 2020 at 12:38 AM Michael Van Canneyt
 wrote:
>
> On Tue, 30 Jun 2020, Wayne Sherman via fpc-devel wrote:
>
> > OpenSSL library loading fixes. Patch file is attached.  Comments are 
> > welcome.
> >
> > Summary:
> >   1)  For Windows and OS2, remove DLLSSLNameX and DLLUtilNameX
> > variables.  Instead use matched arrays of DLL filenames (SSL_DLL_Names
> > and Crypto_DLL_Names)
> >
> >   2)  Due to refactoring, remove LoadLibHack or LoadLib functions.
> >
> >   3)  Remove LoadUtilLibrary function, UnloadSSLLib procedure, and
> > UnloadUtilLib procedure since these were not being called internally
> > and are unreachable externally.
> >
> >   4)  Add new function TryLoadLibPair which attempts to load a
> > matched pair of DLLs (SSL and Crypto).  If either fails to load,
> > UnloadLibraries is called to clean up.
> >
> >   5)  Refactored LoadLibraries to use arrays of names for Windows and
> > OS2, and to use the new TryLoadLibPair helper function for all
> > platforms.
>
> Thank you very much for the patch.
> I will look at this ASAP.

Hi Michael,

  The OpenSSL library loading code still has the potential to load
mismatched library versions.  Did you get a chance to review the patch
I sent?

Thanks
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Tue, Jun 30, 2020 at 1:36 AM Michael Van Canneyt
 wrote:
>
> I know. But we can't include Synopse in FPC.

Can just the JWT/crypto portions be incorporated if it makes sense?
It is released under these Open Source licenses:

   Mozilla Public License, version 1.1 or later (MPL);
   GNU General Public License, version 2.0 or later (GPL);
   GNU Lesser General Public License, version 2.1 or later (LGPL),
with linking exception of the FPC modified LGPL.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Tue, Jun 30, 2020 at 12:47 AM Michael Van Canneyt
 wrote:
> The problem with the service account is that you must create a JWT Token.
> FPC does not yet have a unit that can generate *and sign* a JWT Token.
>
> The good news is I have this weekend a reminder that I received some code
> that will allow me to complete the JWT Token support in FPC using at
> least the RSA256 signing algorithm (and some others as well).

Synopse mORMot has a JWT implementation that supports FPC:
http://blog.synopse.info/post/2016/12/19/JSON-Web-Tokens-(JWT)

The blog post refers to files SynCrypto.pas and SynEcc.pas
https://github.com/synopse/mORMot/

> Now all that is needed is to clone myself so I can actually do the work...
Thanks for any help you can provide. :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] OpenSSL v1.1.x DLL loading / Bug 37137

[Wayne Sherman via fpc-devel]

OpenSSL library loading fixes. Patch file is attached.  Comments are welcome.

Summary:
   1)  For Windows and OS2, remove DLLSSLNameX and DLLUtilNameX
variables.  Instead use matched arrays of DLL filenames (SSL_DLL_Names
and Crypto_DLL_Names)

   2)  Due to refactoring, remove LoadLibHack or LoadLib functions.

   3)  Remove LoadUtilLibrary function, UnloadSSLLib procedure, and
UnloadUtilLib procedure since these were not being called internally
and are unreachable externally.

   4)  Add new function TryLoadLibPair which attempts to load a
matched pair of DLLs (SSL and Crypto).  If either fails to load,
UnloadLibraries is called to clean up.

   5)  Refactored LoadLibraries to use arrays of names for Windows and
OS2, and to use the new TryLoadLibPair helper function for all
platforms.

On Fri, Jun 26, 2020 at 3:36 PM Michael Van Canneyt
 wrote:
>
>
>
> On Fri, 26 Jun 2020, Wayne Sherman wrote:
>
> > On Thu, Jun 25, 2020 at 11:43 PM Michael Van Canneyt
> >  wrote:
> >>>
> >>> 3) I recommend the newest libraries be tried first, working backwards
> >>> towards the oldest.
> >>
> >> This is already so ? The version numbers are added from new to old ?
> >
> > The libraries are currently loading in this order:
> > First attempt:  ssleay32.dll / libeay32.dll (older names, but not 
> > oldest)
> > Second attempt (util):  libcrypto-1_1-x64.dll (newest name)
> > Third attempt (ssl):  libssl32.dll (oldest name)
> > Fourth attempt (ssl):  libssl-1_1-x64.dll (newest name)
>
> Oh, you mean the DLL names.
> I thought you meant the version numbers. Yes, this can be changed too.
>
> >>
> >> This needs to be checked.
> >
> > I ran a test.  If a system has a stray file it can cause mismatched
> > DLLs to load (which could result in strange bugs)
> > Please see attached image "OpenSSL_dll_loading_screenshot.png"
>
> We'll adapt the mechanism.
>
> >> I have changed that to 4.0
> >
> > From:  https://www.openssl.org/policies/releasestrat.html
> >openssl versions 1.1.0, 1.0.2, 1.0.1, 1.0.0 and 0.9.8 are no
> > longer supported
>
> Maybe, but for backwards compatibility they must be supported in FPC.
>
> >
> > So can Windows openssl v1.1 library loading be included in the next
> > maintenance release (v3.2.1?) instead of waiting for v4.0?
>
> No worries, 4.0 means we can consider it for the next fix release.
> That will be 3.2.2 (even numbers only for releases, odd indicates
> development sources)
>
> >
> > Would you like me to have a go at a revised patch?
>
> Sure. If you create a patch, upload it to the bugtracker and assign it to me
> (or mail here so I can have a look)
>
> Michael.
> ___
> fpc-devel maillist  -  fpc-devel@lists.freepascal.org
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Index: packages/openssl/src/openssl.pas
===
--- packages/openssl/src/openssl.pas	(revision 45716)
+++ packages/openssl/src/openssl.pas	(working copy)
@@ -84,30 +84,24 @@
 {$ENDIF OS2}
   DynLibs, cTypes, SysUtils;
 
-var
-  {$IFDEF WINDOWS}
-  DLLSSLName: string = 'ssleay32.dll';
-  DLLSSLName2: string = 'libssl32.dll';
-  DLLSSLName3: string = {$IFDEF WIN64}'libssl-1_1-x64.dll'{$ELSE}'libssl-1_1.dll'{$ENDIF};
-  DLLUtilName: string = 'libeay32.dll';
-  DLLUtilName2: string = {$IFDEF WIN64}'libcrypto-1_1-x64.dll'{$ELSE}'libcrypto-1_1.dll'{$ENDIF};
-  {$ELSE}
-   {$IFDEF OS2}
-{$IFDEF OS2GCC}
-  DLLSSLName: string = 'kssl10.dll';
-  DLLUtilName: string = 'kcrypt10.dll';
-  DLLSSLName2: string = 'kssl.dll';
-  DLLUtilName2: string = 'kcrypto.dll';
-{$ELSE OS2GCC}
-  DLLSSLName: string = 'emssl10.dll';
-  DLLUtilName: string = 'emcrpt10.dll';
-  DLLSSLName2: string = 'ssl.dll';
-  DLLUtilName2: string = 'crypto.dll';
-{$ENDIF OS2GCC}
-   {$ELSE OS2}
-  DLLSSLName: string = 'libssl';
-  DLLUtilName: string = 'libcrypto';
-
+const
+// SSL and Crypto DLL arrays must have the same length and contain
+// matched pairs of DLL filenames. Place newer versions at the beginning.
+{$IFDEF WIN64}
+  SSL_DLL_Names:array[1..3] of string = ('libssl-1_1-x64.dll','ssleay32.dll', 'libssl32.dll');
+  Crypto_DLL_Names: array[1..3] of string = ('libcrypto-1_1-x64.dll', 'libeay32.dll', 'libeay32.dll');
+{$ELSEIF DEFINED(WINDOWS)}
+  SSL_DLL_Names:array[1..3] of string = ('libssl-1_1.dll','ssleay32.dll', 'libssl32.dll');
+  Crypto_DLL_Names: array[1..3] of string = ('libcrypto-1_1.dll', 'libeay32.dll', 'libeay32.dll');
+{$ELSEIF DEFINED(OS2GCC)}
+  SSL_DLL_Names:array[1..2] of string = ('kssl10.dll',   'kssl.dll');
+  Crypto_DLL_Names: array[1..2] of string = ('kcrypt10.dll', 'kcrypto.dll');
+{$ELSEIF DEFINED(OS2)}
+  SSL_DLL_Names:array[1..2] of string = ('emssl10.dll',  'ssl.dll');
+  Crypto_DLL_Names: array[1..2] of string = ('emcrpt10.dll', 'crypto.dll');
+{$ELSE}
+  BaseSSLName: string = 'libssl';
+  BaseCryptoName: string = 'libcrypto';
   { ADD NEW ONES WHEN THEY APPEAR!
 

Re: [fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

On Fri, Jun 26, 2020 at 6:27 PM Wayne Sherman  wrote:
>
> Is there support in the fpc google api units for authenticating as a
> service account?

Also for reference, here is a sample G Suite Service OAuth2 json file
(with made up keys and IDs).  After configuring a service account,
this file can be downloaded locally and used by applications to
authenticate as a service:

{
  "type": "service_account",
  "project_id": "g-suite-email-labels-work",
  "private_key_id": "9387991bed423dccc56b5dedc6676c5e81ed9b74",
  "private_key": "-BEGIN PRIVATE KEY-\\n-END PRIVATE KEY-\n",
  "client_email":
"our-domain-service-a...@g-suite-email-labels-work.iam.gserviceaccount.com",
  "client_id": "869930957598767961280",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth";,
  "token_uri": "https://oauth2.googleapis.com/token";,
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs";,
  "client_x509_cert_url":
"https://www.googleapis.com/robot/v1/metadata/x509/our-domain-service-acct%40g-suite-email-labels-work.iam.gserviceaccount.com";
}
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Google APIs - Authenticate using a service account?

[Wayne Sherman via fpc-devel]

Is there support in the fpc google api units for authenticating as a
service account?

Background:
Service accounts allow an application to interact with Google APIs
without authenticating as a specific user.  Instead a service account
is created with it's own credentials, ID, and key.  By authenticating
as a service account, applications can perform tasks on behalf of the
users in a G Suite domain without having to have their login
passwords.

Here is what it looks like in python:

   Create a Credentials object from the service account's
   credentials and the scopes your application needs access to.
   For example:

   [python code]
   from google.oauth2 import service_account

   SCOPES = ['https://www.googleapis.com/auth/gmail.labels']
   SERVICE_ACCOUNT_FILE = '/path/to/service.json'

   credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
   [/python code]

   Delegate domain-wide authority

   If you have delegated domain-wide access to the service account and
   you want to impersonate a user account, use the with_subject method
   of an existing ServiceAccountCredentials object.
   For example:

   [python code]
   delegated_credentials = credentials.with_subject('u...@example.org')
   [/python code]

   Use the Credentials object to call Google APIs in your application.

Reference Link:
https://developers.google.com/identity/protocols/oauth2/service-account

Background Links:
https://cloud.google.com/docs/authentication/
https://cloud.google.com/iam/docs/service-accounts

(btw, *thank you* Michael Van Canneyt et al. for your work in
providing the Google API support in fpc)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] OpenSSL v1.1.x DLL loading / Bug 37137

[Wayne Sherman via fpc-devel]

*Bug Reference:*
"OpenSSL v1.1.1 could not loaded on Windows platform"
https://bugs.freepascal.org/view.php?id=37137

1) I can confirm that the new v1.1.x DLL names are correct.  I also checked
some of the builds linked from the OpenSSL binary's page.  The OpenSSL
commit that discusses the name changes:
https://github.com/openssl/openssl/commit/520f434b42c83d63d8777075eb66967618551d5b

2) There is a problem with the loading.  The DLLs names have to be handled
in pairs.  If the first DLL fails to load (crypto/util DLL), then both
should be skipped and the next pair tried.  If the second DLL (SSL lib)
fails to load, the first one needs to be unloaded and the pair skipped to
try the next pair.  Otherwise it is possible to load two DLLs that are not
matched versions.

Reference master branch code:
https://github.com/graemeg/freepascal/blob/master/packages/openssl/src/openssl.pas#L5634

3) I recommend the newest libraries be tried first, working backwards
towards the oldest.

Both #2 and #3 are demonstrated by this patch on the synapse mailing list:
https://sourceforge.net/p/synalist/mailman/message/36933840/


4) The bug report says "Fixed in Version: 3.2.0", but that appears to be
incorrect.

Thanks,

Wayne
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel