Re: [fpc-pascal] FPC 3.3.x breaks the Firebird Project's Firebird.pas
On Sat, 4 May 2024, Tony Whyman via fpc-pascal wrote: Michael, I believe that the diplomatic answer to your response is that it is "disappointing". This is a serious problem and needs to be discussed. Thank you for being diplomatic. Interfaces between modules written in different programming languages or even built with different compilers are never easy and depend on having published and stable definitions for each data type and structure. This is a common problem for anyone using Pascal given how so many code libraries are written 'C' or C++. Change can imply breaking many user programs and should not be done lightly and for purely internal reasons. We guarantee backwards compatibility (as much as possible) on the language level and API level, but not on the internal structure of objects. The only structure that is guaranteed to work with C or C++ is the record. Hence my proposed solution using a record. Both object and class type data structures are published in the Free Pascal Documentation You may wish to re-read the documentation and specifically this: https://www.freepascal.org/docs-html/current/ref/refsu23.html#x73-970006.3.1 'As of version 3.0 of the compiler, the compiler can re-order the fields in memory if this leads to better alignment and smaller instances. That means that in an instance, the fields do not necessarily appear in the same order as in the declaration.' So your vTable also is in no way guaranteed to be correct, the above explicitly says that your assumptions are invalid. In the case of monitor data, you have said that Delphi have implemented this as a "hidden" field. I presume that you mean that this uses a technique such as a negative offset to the object instance pointer. No, actually behind all other fields. I don't intend to do work on this, since nowhere was there any guarantee that TObject has no fields. As I wrote, this assumption has no basis and this is even documented. It works purely by chance, not through design. I have given you a possible way to fix it, which is IMO the only correct and future proof way to implement what you need. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] FPC 3.3.x breaks the Firebird Project's Firebird.pas
Michael,
I believe that the diplomatic answer to your response is that it is
"disappointing". This is a serious problem and needs to be discussed.
Interfaces between modules written in different programming languages or
even built with different compilers are never easy and depend on having
published and stable definitions for each data type and structure. This
is a common problem for anyone using Pascal given how so many code
libraries are written 'C' or C++. Change can imply breaking many user
programs and should not be done lightly and for purely internal reasons.
Both object and class type data structures are published in the Free
Pascal Documentation and I do not believe that it is unreasonable to
expect that the FPC developers will do their best to maintain stability
for these structures as they would for any other data type. You seem to
think that assuming that the object data structure is a stable and well
known data structure that can be relied upon to be the same in each
successive compiler version is "an invalid assumption". If that is the
case then can we rely upon the stability of any data type and hence of
any interface between FPC and other language environments?
In the case of monitor data, you have said that Delphi have implemented
this as a "hidden" field. I presume that you mean that this uses a
technique such as a negative offset to the object instance pointer. This
would certainly be a good way to add system information to an object
without destabilising the data structure for the user. Surely the right
way to go.
The Firebird OO API may be the only interface library that uses the
existing object layout in order to implement the interface. That would
surprise me and there may well be more interfaces that are broken by
FPC 3.3.x.
In the case of the Firebird OO API a complete redesign will be needed to
work around your change - requiring a separation of the data blocks
exchanged via the interface, and the classes supporting them. Even
though a code generator is used, there are still about a hundred classes
to consider with some used as callbacks. The Firebird.pas unit itself
comprises about 14,000 lines of code and comments. You are asking
voluntary programmers to devote hundreds, perhaps thousands of hours of
time, to build and test a new interface design purely for the benefit of
FPC, when the existing design works with both Delphi and FPC and has
done so since the early part of the last decade. Until someone puts in
this effort Pascal users of Firebird will be restricted to using the old
legacy interface (which has not been updated since Firebird 2.5 and
misses many important features), keeping with the old version of the
compiler, or just moving to Delphi.
And, before going down this path, perhaps you might like to give a heads
up of what other changes are in the pipeline that may knowing break
interfaces to other programming environments.
Tony Whyman
On 02/05/2024 13:55, Michael Van Canneyt via fpc-pascal wrote:
On Thu, 2 May 2024, Tony Whyman via fpc-pascal wrote:
This is a problem reported to me by an IBX user. I have not yet
confirmed it (need to find the time to set up the latest development
branch of FPC), but the issue looks solid, and is due to the
following addition to TObject (copied from the GitLab master branch)
|TObject = class|{$IFDEF SYSTEM_HAS_FEATURE_MONITOR}
strictprivate
_MonitorData : Pointer;
private
functionSetMonitorData(aData,aCheckOld : Pointer):Pointer; inline;
functionGetMonitorData:Pointer; inline;
{$ENDIF}
...
Since Firebird 3.0, the Firebird project has provided a file called
"Firebird.pas" in order to provide an interface to Pascal (both
Delphi and FPC) from a C++ environment. There is an underlying
assumption in the interface. That is TObject contains no fields.
This assumption is not valid in Delphi either. It also has this field,
but
it is 'hidden' at a not-fixed location.
The idea is simple enough. There is a single entry point to the
Firebird client library with the Pascal signature
function fb_get_master_interface : IMaster; cdecl;
This returns a pointer to a structure that may contain fields but is
primarily a list of pointers to cdecl functions. These functions can
perform various tasks. In the case of IMaster, they can also return
similar structures representing other "interfaces".
On the Pascal side, IMaster is defined as a class with no virtual
methods and a single field called "vTable" and this is set to the
pointer to the structure returned by fb_get_master_interface. The
rest of the class comprises methods used to call functions from the
vTable. This makes IMaster look like a normal class and hides the use
of function pointers. The code is generated by a program know as
"cloop". All Firebird interfaces declared in Firebird.pas follow the
same approach.
Given the assumption that TObject defines no fields, it is possible
to coerce the
Re: [fpc-pascal] FPC 3.3.x breaks the Firebird Project's Firebird.pas
On Thu, 2 May 2024, Tony Whyman via fpc-pascal wrote:
This is a problem reported to me by an IBX user. I have not yet confirmed it
(need to find the time to set up the latest development branch of FPC), but
the issue looks solid, and is due to the following addition to TObject
(copied from the GitLab master branch)
|TObject = class|{$IFDEF SYSTEM_HAS_FEATURE_MONITOR}
strictprivate
_MonitorData : Pointer;
private
functionSetMonitorData(aData,aCheckOld : Pointer):Pointer; inline;
functionGetMonitorData:Pointer; inline;
{$ENDIF}
...
Since Firebird 3.0, the Firebird project has provided a file called
"Firebird.pas" in order to provide an interface to Pascal (both Delphi and
FPC) from a C++ environment. There is an underlying assumption in the
interface. That is TObject contains no fields.
This assumption is not valid in Delphi either. It also has this field, but
it is 'hidden' at a not-fixed location.
The idea is simple enough. There is a single entry point to the Firebird
client library with the Pascal signature
function fb_get_master_interface : IMaster; cdecl;
This returns a pointer to a structure that may contain fields but is
primarily a list of pointers to cdecl functions. These functions can perform
various tasks. In the case of IMaster, they can also return similar
structures representing other "interfaces".
On the Pascal side, IMaster is defined as a class with no virtual methods and
a single field called "vTable" and this is set to the pointer to the
structure returned by fb_get_master_interface. The rest of the class
comprises methods used to call functions from the vTable. This makes IMaster
look like a normal class and hides the use of function pointers. The code is
generated by a program know as "cloop". All Firebird interfaces declared in
Firebird.pas follow the same approach.
Given the assumption that TObject defines no fields, it is possible to coerce
the pointer returned by fb_get_master_interface to appear as an object
"instance" of the IMaster class, and similarly for every other Firebird
interface class. This coercion ceases to be valid as soon as a field, such as
_MonitorData is added to TObject.
It is probably a valid argument to say that this is Firebird's problem as the
TObject definition is an internal data structure. However, this ignores a
serious real world problem.
The real problem is the invalid assumption.
This assumption is invalid for Delphi as well. TObject is not empty there either.
The memory layout is simply different.
A fix or workaround is needed and that has to be
in Firebird.pas. It would be a logistic nightmare to try and change the C++
Firebird client DLL/SO.
It would be a logistic nightmare in FPC as well.
Also, does anyone know whether a similar problem will arise with Delphi, or
is thus just an FPC issue?
One of the following seems to be needed:
a) Rewriting Firebird.pas to set the vTable field explicitly rather than
implicitly through coercion - ideally without breaking any user code...
(note: no current need to free IMaster etc)
b) Taking a different approach to introducing _MonitorData e.g.
This field was introduced for Delphi compatibility:
The implementation of TMonitor requires this data.
Delphi has this field as well but "hides" the field behind all other actually
declared fields.
If you examine the actual memory size allocated for TObject, you'll see that an
extra field is actually present.
In FPC this turned out to be not so easy to do without modifying the compiler,
so it was decided that the cleanest approach is to actually introduce the field as part of TObject.
The firebase people use a trick to save you some work, based on an invalid
assumption.
I don't think this is sufficient reason to expect the FPC team to adapt their
compiler.
The correct solution is to change the cloop program to produce a
pascal record definition with procedural type fields at the correct offsets.
It should be perfectly usable without too many changes.
So something like (adapt to actual procedure types)
Type
RMaster = Record
init : procedure(xyz:sometype);
// etc
end;
IMaster = ^RMaster;
// optionally; although it seems this already exists?
TMaster = class
intf : IMaster;
procedure Init(xyz : sometype);
constructor create;
end;
function fb_get_master_interface : IMaster; cdecl;
constructor TMaster.Create;
begin
intf:=fb_get_master_interface;
end;
Procedure TMaster.Init(xyz : sometype);
begin
intf^.init(xyz);
end;
if you cannot change cloop, you can use fcl-passrc or some other tool
to actually parse the definition generated by cloop and change it to a record,
possibly with a wrapper class for easier access as in the above.
That should not be difficult to do, and it will work in Delphi as well.
The advantage is that it will be more robust against future changes as it
makes no assumptions about the layout of objects.
Michael.
[fpc-pascal] FPC 3.3.x breaks the Firebird Project's Firebird.pas
This is a problem reported to me by an IBX user. I have not yet
confirmed it (need to find the time to set up the latest development
branch of FPC), but the issue looks solid, and is due to the following
addition to TObject (copied from the GitLab master branch)
|TObject = class|{$IFDEF SYSTEM_HAS_FEATURE_MONITOR}
strictprivate
_MonitorData : Pointer;
private
functionSetMonitorData(aData,aCheckOld : Pointer):Pointer; inline;
functionGetMonitorData:Pointer; inline;
{$ENDIF}
...
Since Firebird 3.0, the Firebird project has provided a file called
"Firebird.pas" in order to provide an interface to Pascal (both Delphi
and FPC) from a C++ environment. There is an underlying assumption in
the interface. That is TObject contains no fields.
The idea is simple enough. There is a single entry point to the Firebird
client library with the Pascal signature
function fb_get_master_interface : IMaster; cdecl;
This returns a pointer to a structure that may contain fields but is
primarily a list of pointers to cdecl functions. These functions can
perform various tasks. In the case of IMaster, they can also return
similar structures representing other "interfaces".
On the Pascal side, IMaster is defined as a class with no virtual
methods and a single field called "vTable" and this is set to the
pointer to the structure returned by fb_get_master_interface. The rest
of the class comprises methods used to call functions from the vTable.
This makes IMaster look like a normal class and hides the use of
function pointers. The code is generated by a program know as "cloop".
All Firebird interfaces declared in Firebird.pas follow the same approach.
Given the assumption that TObject defines no fields, it is possible to
coerce the pointer returned by fb_get_master_interface to appear as an
object "instance" of the IMaster class, and similarly for every other
Firebird interface class. This coercion ceases to be valid as soon as a
field, such as _MonitorData is added to TObject.
It is probably a valid argument to say that this is Firebird's problem
as the TObject definition is an internal data structure. However, this
ignores a serious real world problem. A fix or workaround is needed and
that has to be in Firebird.pas. It would be a logistic nightmare to try
and change the C++ Firebird client DLL/SO.
Also, does anyone know whether a similar problem will arise with Delphi,
or is thus just an FPC issue?
One of the following seems to be needed:
a) Rewriting Firebird.pas to set the vTable field explicitly rather than
implicitly through coercion - ideally without breaking any user code...
(note: no current need to free IMaster etc)
b) Taking a different approach to introducing _MonitorData e.g.
i) Putting in a class helper (is this possible?)
ii) Having a version of TObject (e.g. TUnmonitoredObject) which is the
current TObject and making TObject a descendent class introducing
_MonitorData. The Firebird classes can then descend from
TUnMonitoredObject, either explicitly or as a result of a new compiler
directive setting the default class ancestor.
iii) Anything else that works.
I am primarily looking for a solution that will work with IBX, but also
one that I can suggest to the Firebird Devs to make generally available.
Tony Whyman
MWA Software
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
