Hi Jim,
Sounds to me as if your design could use Drizzle as a component of the
Nimbus node.
Have you considered this?
On Sep 29, 2008, at 4:59 PM, Jim Starkey wrote:
Paul McCullagh wrote:
On Sep 27, 2008, at 1:45 PM, Jim Starkey wrote:
Let me propose an alternative solution for sharding and load
balancing (oh, I hate the very idea of this being user visible!).
Suppose you added a mechanism to associate an arbitrary ordered
set of attribute/value pairs with a connection that would be sent
along as part of the protocol. Your proxy could use these to
decide where to direct a connection. After, after making this
weighty decision, it could pre-pend its own attribute/value pair
indicating where it sent the connection so subsequent statements
would go the same place. This has many advantages:
* It doesn't require standardization of SQL entrails
* It isn't dependent on client winks and proxy nudges -- it's all
above board
* It provides a consistent mechanism to direct a series of
transactional statements to a particular server
* It is consistent with compiled statements (so is existing
proposal, incidentally)
* It is consistent with aggregating interfaces
How does this handle the fact that the data is partitioned and so
not all data is available to a particular node/server?
It doesn't handle it at all. Neither does Brian's approach of
tokening SQL on the client so the proxy can pick through the
entrails to deduce the shard to which the query must be be
delivered. It just makes a difficult and unpleasant technique a
little easier for the client and the proxy.
I've been redesigning the connection/service/protocol handling for
Nimbus over the last couple of days. If anyone's interested, I'd
be happy to sketch out the evolving design.
I would be most interested!
Let me start with the problem. A Nimbus node needs to support an
open ended set of services/protocols/relationships with other
nodes. First, there are an intra-Nimbus connections for
replication. Second, there is a straightforward JDBC based protocol
for handling database access by Nimbus clients. Next, there are
cloud management connection/services to monitor (and to a lessor
degree control) a Nimbus cloud. Finally, there is an open ended set
of aggregating interfaces to enable execution of a number of SQL
statements under control of server side logic to reduce the number
of round trips between a client and Nimbus node. It's safe to
assume that these services use a variety of different line
protocols, and trying to bring them into one mega-protocol is a
fool's errand.
One way to handle this is for each server to have a separate port
for each service. But this is a lot of IP address and ports to
track. And given a requirement to support any number of server
processes on a single node for development, a more difficult problem.
So each Nimbus node uses a single port for all incoming connections
with a single very simple connect request packet, containing an
encoded message type (OK, these leaves a lot of run for future
abuse) and a snippet of XML. The XML to join a cloud, for example,
is:
<connect Service="platform" NodeType="1" Password="in your dreams"/>
Internally, services register with a ConnectionManager. When a
connection shows up, the ConnectionManager reads a message from the
connection, verifies the integrity of the packet, parses the XML,
finds a service corresponding to the Service attribute, and passes
the socket off to the service, at which point the service owns the
socket and can do anything it wishes. Each service will extend the
XML to handling any identification or authentication information,
connection specific data, etc.
I can hear hundreds of you saying to yourself, "yuck! XML!". It
doesn't have to be actual XML. Anything homeomophic will do. The
requirements are nested named entities each with a set of attribute/
value pair. Binary, character, who cares? I use XML because it's
well defined and you can read it. If it offends you, use something
else.
This isn't a rocket science, but a straightforward solution to a
messy problem that doesn't require that fragments of my brain come
to a unified protocol. The more complex the project, the bigger the
payback.
--
Jim Starkey
President, NimbusDB, Inc.
978 526-1376
--
Paul McCullagh
PrimeBase Technologies
www.primebase.org
www.blobstreaming.org
pbxt.blogspot.com
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
