Let me try to explain locking once more (since there's confusion on that and
the db pool issue).
The MailRepository has a few methods...
void store(Mail)
Iterator list()
Mail retrieve(String key)
void remove(Mail)
void remove(key)
What each method does should be pretty obvious. The MailRepository doesn't
do any locking, but is the basis for the SpoolRepository.
The SpoolRepository extends MailRepository and (basically) has one new
method...
String accept()
A thread calls this method, and then this method will block until there is a
message that needs processing. When a message is available, the method will
acquire/place a lock on that message, and return the key (the String).
Multiple threads can then call this method, all blocking, and when a message
arrives, only one thread will get a lock on that message. Again, you're
trying to prevent threads in the same JVM from processing the same message,
so you can keep track of this locking in memory.
Everything with this approach is very scalable (locks work great, multiple
threads can work on multiple messages simultaneously, use multiple db
connections, etc...) except for how TownSpoolRepository implements accept().
Right now what it does is list() of all the keys, and then walks through the
list to try to find an unlocked message. This is unscalable because as the
list of messages grows, this gets slower and slower...
What's even worse is how it implements accept(long delay). If a message is
put back in the queue for a later retry, a thread calls accept(long delay),
which means the same thing as accept but wait up to "delay" milliseconds
before retrying a failed message. Not only does it lock a message, but it
then retrieves the message to check the time. This makes it geometrically
slower as the number of messages needing a retry increases.
Anyway, pure JDBC access will help me build much better implementations of
this (not that it's impossible without it). I have to look over the db pool
code from excalibur, but with that, hopefully I can get it working this
weekend.
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
----- Original Message -----
From: "David Doucette" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Saturday, June 16, 2001 2:14 PM
Subject: Re: RDBMS support
> First off, I'm very new to James and since I haven't been able to get the
> last stable release out of CVS yet, I'm forced to sit on the sidelines and
> watch james-dev and james-user. However, I have learned some things and
> have thought about how I would have to make changes if I wanted to
increase
> performance.
>
> >1. James's database structure is very simple. The core of James has the
> >potential to use 2 unrelated tables. One for a user repository and one
for
> >a message spool repository. The structure of these tables are very
simple.
> This is what struck me originally when I heard of all the abstraction
going
> on. It doesn't seem like a very complex system is needed, since James
> doesn't really have complex database requirements!
>
> >2. Users are already shielded from the database code by the
UserRepository
> >and SpoolRepository API. This ease of use would really only help the one
or
> >two developers who write the DatabaseUserRepository and
> >DatabaseSpoolRepository class.
> Very good point.
>
> >3. We need a lot of control over how data is returned for performance
> >reasons. Two big limitations we are experiencing with Town (that I
believe
> >we would have with Turbine and other abstraction layers) is the inability
to
> >return parts of a ResultSet and to get streamed access to binary data. I
> >believe both of these are critical to increasing scalability and
> >performance.
> I'm still not 100% clear on how the locking of records works (it almost
> sounds like it uses Java's thread locking mechanisms rather than locking
by
> setting a flag in the DB). However, If the engine has to periodically
> re-read all the messages in the spool just to send some of them out, then
> I'm all for anything that will turn that around. It just won't scale
> without doing something about this. If someone could explain again how
> this works, I'd appreciate it.
>
> >Then the only thing
> >we need is a JDBC connection pool code, and I happen to have one I can
add
> >pretty easily.
> Again, I'm just not clear on how database connections work now. It seems
> to me that there can only be one connection retrieving spool messages,
> because otherwise locking in memory wouldn't work and each thread that has
> a connection would retrieve the same messages and send them again.
>
> >Any thoughts? I can throw the JDBC spool repository implementation
together
> >pretty quickly using the old table structure.
> Go for it!
>
> I know that I would really like not having to redesign this part of James
> so that it scales and I'm sure others feel the same way.
>
> David
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
