Re: [fpc-devel] Adding a unidirectional dataset to sqldb
On Fri, 2006-10-06 at 07:51 +0200, Martin Schreiber wrote:
> On Thursday 05 October 2006 22.41, Joost van der Sluis wrote:
> > > > Now I'm thinking about using an interface, to avoid double code. But I
> > > > don't know what effect that has on run-time performance. I mean, the
> > > > idea was to make if faster ...
> > >
> > > A very good idea! It can then be implemented by a common worker class.
> > > TSQLConnection and TSQLTransaction should be independent of TSQLQuery and
> > > TSQLQuery should get its own unit.
> >
> > Can you explain to me how I should construct this worker-class? As
> > Michael said, my solution won't work...
> >
> If TSQLConnection or TSQLTransaction need callbacks into TSQLQuery and
> "TSQLQueryUni" do it by an corba interface (at the first glance I didn't see
> any).
> To implement the common functions of TSQLQuery and TSQLQueryUni use a "worker
> class" which does callbacks by an corba interface.
Ok, I tried to implement this. I still don't understand why I should
need an interface for this, though.
Attached is a patch that does this. It's work in progress, TSQLQuery
doesn't work with this patch, but the TSQLUnidirectionalQuery does. At
least, a basic select works...
But I wanted to make things faster. I'm not feeling comfortable with
creating more wrapper-class layers, more complexity, for nothing, if it
doesn't result in any speedup.
Good side-effect is, that I could make a parent for TSQLBasics 'worker-
class' which could be used by external sqldb-add-in's. As Michael, Bram
and I discussed before.
But for now, I'm curious what you guys find the work done in the
attachment, before I go on with this...
Regards,
Joost
Index: sqldb/sqldb.pp
===
--- sqldb/sqldb.pp (revision 4918)
+++ sqldb/sqldb.pp (working copy)
@@ -62,7 +62,6 @@
);
-{ TSQLConnection }
type
{ TSQLConnection }
@@ -165,25 +164,59 @@
property Params : TStringList read FParams write FParams;
end;
+ { TSQLBasics }
+
+ TSQLBasics = class (TObject)
+ private
+FDBObject: TComponent;
+
+FDatabase: TSQLConnection;
+FTransaction : TSQLTransaction;
+FCursor : TSQLCursor;
+FSQL : TStringList;
+FSQLBuf : String;
+FParams : TParams;
+FLoadingFieldDefs: boolean;
+
+procedure SetDatabase(const AValue : TSQLconnection);
+procedure SetTransaction(const AValue: TSQLtransaction);
+ public
+constructor Create(ADBObject : TComponent); virtual;
+destructor Destroy; override;
+procedure Prepare; virtual;
+Function IsPrepared : Boolean; virtual;
+procedure InternalOpen; virtual;
+procedure InternalInitFieldDefs; virtual;
+
+property Database : TSQLConnection read FDatabase write SetDatabase;
+property Transaction : TSQLtransaction read FTransaction write SetTransaction;
+property Params : TParams read FParams;
+property SQL : TStringlist read FSQL;
+property Cursor : TSQLCursor read FCursor write FCursor;
+ end;
+
+
{ TSQLQuery }
TSQLQuery = class (Tbufdataset)
private
-FCursor : TSQLCursor;
+FSQLBasics : TSQLBasics;
+
+FCursor : TSQLCursor; // r
FUpdateable : boolean;
FTableName : string;
-FSQL : TStringList;
+FSQL : TStringList;// r
FUpdateSQL,
FInsertSQL,
FDeleteSQL : TStringList;
FIsEOF : boolean;
-FLoadingFieldDefs: boolean;
+FLoadingFieldDefs: boolean; // r
FIndexDefs : TIndexDefs;
FReadOnly: boolean;
FUpdateMode : TUpdateMode;
-FParams : TParams;
+FParams : TParams; // r
FusePrimaryKeyAsKey : Boolean;
-FSQLBuf : String;
+FSQLBuf : String; // r
FFromPart: String;
FWhereStartPos : integer;
FWhereStopPos: integer;
@@ -196,12 +229,15 @@
FInsertQry : TSQLQuery;
procedure FreeFldBuffers;
+function GetParams: TParams;
procedure InitUpdates(ASQL : string);
function GetIndexDefs : TIndexDefs;
function GetStatementType : TStatementType;
procedure SetIndexDefs(AValue : TIndexDefs);
+procedure SetParams(const AValue: TParams);
procedure SetReadOnly(AValue : Boolean);
procedure SetParseSQL(AValue : Boolean);
+procedure SetSQL(const AValue: TStringlist);
procedure SetUsePrimaryKeyAsKey(AValue : Boolean);
procedure SetUpdateMode(AValue : TUpdateMode);
procedure OnChangeSQL(Sender : TObject);
@@ -210,6 +246,7 @@
Procedure SQLParser(var ASQL : string);
procedure ApplyFilter;
Function AddFilter(SQLstr : string) : string;
+function getSQL: TStringlist;
protected
// abstract & virtual methods of TBufDataset
Re: [fpc-devel] Threads and alot of crap
Op Tue, 17 Oct 2006, schreef Micha Nelissen: > Daniël Mantione wrote: > > no kernel call is necessary. If the lock starts spinning, on uniprocessor > > it won't be released until the kernel schedules the other thread. This > > exactly the idea behind kernel futexes, if the lock is not held, no > > kernel > > Aren't futexes 2.6+ only ? They are. For backward compatibility you use the semctl API. This is not problematic, pthreads uses semctl for <=2.4 as well. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On Tuesday 17 October 2006 10:03, Micha Nelissen wrote: > Windows events do not have this problem since they are stateful. To be more precise: Windows signals are persistent, not transient like Unix signals are. Vinzent. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Jonas Maebe wrote: By implication this means you need a mutex to protect against race conditions. Not necessarily, it is at least possible to implement an atomic linked list without requiring a mutex-style lock. That's irrelevant. If thread 1 lets thread 2 do something, and thread 2 would signal thread 1 after completion, then thread 1 might not have started to wait yet (descheduled just before cond_wait). Could happen for example with TThread.Synchronize. Windows events do not have this problem since they are stateful. In hindsight, a semaphore might have been a better emulation of an 'Event' (although I've not thought this through completely), but at the time condition variables seemed to be the logical way to go. Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On Tuesday 17 October 2006 09:46, Jonas Maebe wrote: > On 17 okt 2006, at 11:22, Vinzent Hoefler wrote: > >> The pthread_cond_wait() function atomically unlocks the > >> mutex argument > >> and waits on the cond argument. > >> > >> So the mutex should already be unlocked afterwards. > > > > If you would have read a couple of lines further you also would > > have > > > > found: > > |Before returning to the calling thread, pthread_cond_wait > > | re-acquires mutex (as per pthread_lock_mutex). > > That is not on the Mac OS X man page for pthread_cond_wait. See >http://www.hmug.org/man/3/pthread_cond_wait.php I see. > Not sure if this means it's not necessary in the Mac OS X (and > possibly FreeBSD) versions, or that the Mac OS X man pages are > incomplete. They are most definitely incomplete. Vinzent. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 17 okt 2006, at 11:58, Jonas Maebe wrote: Not necessarily, it is at least possible to implement an atomic linked list without requiring a mutex-style lock. ... using the atomic primitives of the PPC processor. Maybe it's not possible in general... Jonas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 17 okt 2006, at 11:50, Micha Nelissen wrote: Jonas Maebe wrote: Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete. It does say: The pthread_cond_signal() function unblocks one thread waiting for the condition variable cond. By implication this means you need a mutex to protect against race conditions. Not necessarily, it is at least possible to implement an atomic linked list without requiring a mutex-style lock. Those manual pages are incomplete/too concise I would say. That indeed seems to be the case. Jonas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
[ Charset ISO-8859-1 unsupported, converting... ] > Jonas Maebe wrote: > > Not sure if this means it's not necessary in the Mac OS X (and possibly > > FreeBSD) versions, or that the Mac OS X man pages are incomplete. > > It does say: The pthread_cond_signal() function unblocks one thread > waiting for the condition variable cond. > > By implication this means you need a mutex to protect against race > conditions. Those manual pages are incomplete/too concise I would say. FreeBSD text for cond_wait: DESCRIPTION The pthread_cond_wait() function atomically blocks the current thread waiting on the condition variable specified by cond, and unblocks the mutex specified by mutex. The waiting thread unblocks only after another thread calls pthread_cond_signal(3), or pthread_cond_broadcast(3) with the same condition variable, and the current thread reacquires the lock on mutex. The last line seems to say the same as Linux. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Jonas Maebe wrote: Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete. It does say: The pthread_cond_signal() function unblocks one thread waiting for the condition variable cond. By implication this means you need a mutex to protect against race conditions. Those manual pages are incomplete/too concise I would say. Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 17 okt 2006, at 11:22, Vinzent Hoefler wrote: The pthread_cond_wait() function atomically unlocks the mutex argument and waits on the cond argument. So the mutex should already be unlocked afterwards. If you would have read a couple of lines further you also would have found: |Before returning to the calling thread, pthread_cond_wait re-acquires |mutex (as per pthread_lock_mutex). That is not on the Mac OS X man page for pthread_cond_wait. See http://www.hmug.org/man/3/pthread_cond_wait.php Further, what is the lock/unlock for in the intRTLEventSetEvent? pthread_cond_signal can perfectly deal with multiple threads simultaneously signalling the same condition variable (pthreads is a multithreading api, not a random non-reentrant piece of code). Oh, is it? Read the f*cking man page: Again, this is not on the Mac OS X man page: http://www.hmug.org/man/3/pthread_cond_signal.php Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete. Jonas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On Tuesday 17 October 2006 09:03, Jonas Maebe wrote: > On 17 okt 2006, at 10:44, Daniël Mantione wrote: > > procedure intRTLEventSetEvent(AEvent: PRTLEvent); > > var p:pintrtlevent; > > > > begin > > p:=pintrtlevent(aevent); > > pthread_mutex_lock(@p^.mutex); > > pthread_cond_signal(@p^.condvar); > > pthread_mutex_unlock(@p^.mutex); > > end; > > > > procedure intRTLEventStartWait(AEvent: PRTLEvent); > > var p:pintrtlevent; > > > > begin > > p:=pintrtlevent(aevent); > > pthread_mutex_lock(@p^.mutex); > > end; > > > > procedure intRTLEventWaitFor(AEvent: PRTLEvent); > > var p:pintrtlevent; > > > > begin > > p:=pintrtlevent(aevent); > > pthread_cond_wait(@p^.condvar, @p^.mutex); > > pthread_mutex_unlock(@p^.mutex); > > end; > > This last pthread_mutex_unlock does not make sense to me. From the > pthread_cond_wait man page: > > The pthread_cond_wait() function atomically unlocks the mutex > argument > and waits on the cond argument. > > So the mutex should already be unlocked afterwards. If you would have read a couple of lines further you also would have found: |Before returning to the calling thread, pthread_cond_wait re-acquires |mutex (as per pthread_lock_mutex). > Further, what is > the lock/unlock for in the intRTLEventSetEvent? pthread_cond_signal > can perfectly deal with multiple threads simultaneously signalling > the same condition variable (pthreads is a multithreading api, not a > random non-reentrant piece of code). Oh, is it? Read the f*cking man page: |A condition variable must always be associated with a mutex, to avoid |the race condition where a thread prepares to wait on a condition |variable and another thread signals the condition just before the first |thread actually waits on it. Vinzent. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Jonas Maebe wrote: This last pthread_mutex_unlock does not make sense to me. From the pthread_cond_wait man page: Read the man page *completely*. A condition variable must always be associated with a mutex, to avoid the race condition where a thread prepares to wait on a condition variable and another thread signals the condition just before the first thread actually waits on it. and: Before returning to the calling thread, !pthread_cond_wait! re-acquires |mutex| (as per !pthread_lock_mutex!). Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 17 okt 2006, at 10:44, Daniël Mantione wrote: procedure intRTLEventSetEvent(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_mutex_lock(@p^.mutex); pthread_cond_signal(@p^.condvar); pthread_mutex_unlock(@p^.mutex); end; procedure intRTLEventStartWait(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_mutex_lock(@p^.mutex); end; procedure intRTLEventWaitFor(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_cond_wait(@p^.condvar, @p^.mutex); pthread_mutex_unlock(@p^.mutex); end; This last pthread_mutex_unlock does not make sense to me. From the pthread_cond_wait man page: The pthread_cond_wait() function atomically unlocks the mutex argument and waits on the cond argument. So the mutex should already be unlocked afterwards. Further, what is the lock/unlock for in the intRTLEventSetEvent? pthread_cond_signal can perfectly deal with multiple threads simultaneously signalling the same condition variable (pthreads is a multithreading api, not a random non-reentrant piece of code). Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Daniël Mantione wrote: no kernel call is necessary. If the lock starts spinning, on uniprocessor it won't be released until the kernel schedules the other thread. This exactly the idea behind kernel futexes, if the lock is not held, no kernel Aren't futexes 2.6+ only ? Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Micha Nelissen wrote: Marc Weustink wrote: This was exactly the reason why I choose to use pthreads directly. For the given situation one single semaphore call could replace this. Well, events <> semaphores :-). thats why I wrote: for the given situation ;) It was for the cheap-concurrency benchmark. Later it turned out that a semaphore was more suitable that I was used to with (windows) events. We need semaphore abstraction as well for "proper" RTL, besides events. Eh, yes, that would be nice. Marc ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Op Tue, 17 Oct 2006, schreef Micha Nelissen: > Marc Weustink wrote: > > This was exactly the reason why I choose to use pthreads directly. For > > the given situation one single semaphore call could replace this. > > Well, events <> semaphores :-). We need semaphore abstraction as well for > "proper" RTL, besides events. Yes, but Tthread makes a lot use of both semaphores and events, so we need fast implementations for both. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Marc Weustink wrote: This was exactly the reason why I choose to use pthreads directly. For the given situation one single semaphore call could replace this. Well, events <> semaphores :-). We need semaphore abstraction as well for "proper" RTL, besides events. Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Daniël Mantione wrote: Op Tue, 17 Oct 2006, schreef Jonas Maebe: On 17 okt 2006, at 09:25, Daniël Mantione wrote: If I compare my implementation of the Chameneos benchmark with the one from Marc (which uses Pthreads directly), mine is about two times slower. This is propably caused that our thread functions often require multiple Pthread calls, Where? I only see this in starting/initialising/stopping/destructing things. The actual locking etc use one pthread call in all cases afaics. Of course, starting a Pascal thread requires initialising all the threadvars etc, but that is not going to change if you use something else than pthreads. For example: procedure intRTLEventSetEvent(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_mutex_lock(@p^.mutex); pthread_cond_signal(@p^.condvar); pthread_mutex_unlock(@p^.mutex); end; This was exactly the reason why I choose to use pthreads directly. For the given situation one single semaphore call could replace this. Marc ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Op Tue, 17 Oct 2006, schreef Jonas Maebe: > > On 17 okt 2006, at 09:25, Daniël Mantione wrote: > > > If I compare my implementation of the Chameneos benchmark with the one > > from Marc (which uses Pthreads directly), mine is about two times slower. > > This is propably caused that our thread functions often require multiple > > Pthread calls, > > Where? I only see this in starting/initialising/stopping/destructing things. > The actual locking etc use one pthread call in all cases afaics. Of course, > starting a Pascal thread requires initialising all the threadvars etc, but > that is not going to change if you use something else than pthreads. For example: procedure intRTLEventSetEvent(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_mutex_lock(@p^.mutex); pthread_cond_signal(@p^.condvar); pthread_mutex_unlock(@p^.mutex); end; procedure intRTLEventStartWait(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_mutex_lock(@p^.mutex); end; procedure intRTLEventWaitFor(AEvent: PRTLEvent); var p:pintrtlevent; begin p:=pintrtlevent(aevent); pthread_cond_wait(@p^.condvar, @p^.mutex); pthread_mutex_unlock(@p^.mutex); end; I'm not 100% sure yet, but I believe you can implement a set-event with a FUTEX_WAKE, and a waitfor with an unconditional FUTEX_WAIT, which would mean a lot less calls. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 17 okt 2006, at 09:25, Daniël Mantione wrote: If I compare my implementation of the Chameneos benchmark with the one from Marc (which uses Pthreads directly), mine is about two times slower. This is propably caused that our thread functions often require multiple Pthread calls, Where? I only see this in starting/initialising/stopping/destructing things. The actual locking etc use one pthread call in all cases afaics. Of course, starting a Pascal thread requires initialising all the threadvars etc, but that is not going to change if you use something else than pthreads. Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
Op Tue, 17 Oct 2006, schreef Jonas Maebe: > > On 16 okt 2006, at 22:49, Daniël Mantione wrote: > > > In other works, pthreads results > > in subpar performance, > > Is the overhead of a few user level routines really that big? Once the threads > are setup, they automatically become kernel threads anyway. Having a user > level layer in between might even help in some cases (like e.g. if the > threading subsystem decides that a spinlock is more appropriate in a > particular situation based on the semaphore's properties). If I compare my implementation of the Chameneos benchmark with the one from Marc (which uses Pthreads directly), mine is about two times slower. This is propably caused that our thread functions often require multiple Pthread calls, which might result in multiple kernel calls. So it is mainly the synchronisation that is the issue. Of course, if the threads are running independend code with no need for accessing shared resources, it all depends on the kernel scheduling and there should be no significant overhead. In practise, this is seldom the case. A spinlock is mainly fast in the case the lock is not held, in such case no kernel call is necessary. If the lock starts spinning, on uniprocessor it won't be released until the kernel schedules the other thread. This exactly the idea behind kernel futexes, if the lock is not held, no kernel call is done. If the lock is held, a schedule is necessary anyway, so the kernel is called. The kernel is in a much better position in this case, because it exactly knows which threads it should not schedule. On SMP it might be the case both threads are scheduled at the same time, and here a spinlock might prevent a kernel call altogether. However, even here nobody has ever beaten the futex system in performance. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Threads and alot of crap
On 16 okt 2006, at 22:49, Daniël Mantione wrote: In other works, pthreads results in subpar performance, Is the overhead of a few user level routines really that big? Once the threads are setup, they automatically become kernel threads anyway. Having a user level layer in between might even help in some cases (like e.g. if the threading subsystem decides that a spinlock is more appropriate in a particular situation based on the semaphore's properties). Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
