Brian Aker wrote:
> Hi!
>
> On Aug 3, 2008, at 12:32 PM, Mats Kindahl wrote:
>
>> Long-term, I would like to see the table definitions stored in
>> (transactional) tables. That would mean that:
>>
>> - DDL operations can be replicated as tables changes, even in row format
>> - DDL operations can be transactional (provided the tables holding the
>> transactional data is transactional)
>
> All agreed :)
>
> If we use discovery as our primary means of asking about tables, will
> this work for the above?
>
> (I believe it does)
>
>
>> Since we are aiming for a micro-server that can be a component of a
>> cloud, I think that we, long-term, have to move to a raw data format for
>> replication: the current replication system lacks in performance and it
>> would be beneficial to be able to treat all changes that need to
>> replicate in the same manner. This simplifies the logic, hence shrinks
>> the code base and eliminates the number of code paths, hence improves
>> performance of the replication (and of the system in general).
>
> Can you expand on this? I have some thoughts on how we can begin to
> rework replication but have only begun working through them. There are a
> couple of natural split points:
>
> 1) Message format
> 2) Message store
> 3) Message transport
>
> What else do you have?
I think the single most important thing to get right is the high-level
architecture. By making a number of basic assumptions about the
execution environment, we allow ourselves some wiggle room to improve
performance:
Let's assume:
* Modern processors (pipeline, cache, multiple cores)
* High-speed high-latency networks
* Transactional engines
* Cheap hardware
This suggests the following:
* Asynchronous message dispatch
* This also suggests using asynchronous I/O to disk
* Reduce size of working memory
* Caches will be used more efficiently
* Linear code paths
* Avoid branching or decisions in the code or keep them early
* Speculative replication
* We can always roll back if a problem occurs
* Send changes as soon as they are scheduled
* Row-based/data-based replication
* Interleaved execution
* Micro-packets
* Keep the state small
* Recovering the state on restarts will be easier
* Data-driven execution
* Frequent changes of topology when nodes fail
* Ad-hoc replication
* Replication stream is a DAG, not linear
I see four items that need to be designed:
* Message propagation mechanism ("dump thread" + "I/O thread")
* Replication control and handshake
* Replication log ("binary log")
* Change application ("SQL thread")
Just my few cents,
Mats Kindahl
>
> Cheers,
> -Brian
>
>>
>>
>> Just my few cents,
>> Mats Kindahl
>>
>> Brian Aker wrote:
>>> Hi!
>>>
>>> 1) Engines should be asked about tables, and tables belong to engines.
>>> 2) We still need some sort of FRM for Engines which have no ability to
>>> handle their own data dictionaries.
>>>
>>> The current Unireg format is a real pain in the butt and has not been
>>> scaling well with recent additions. It pre-set binary type.
>>>
>>> Three methods for storage:
>>> 1) XML (slow)
>>> 2) SQL (tamper-able/less slow)
>>> 3) Binary Stream
>>>
>>> Personally I am for types 2 or 3 for a serial method. In fact I am for
>>> both being added to the table class.
>>>
>>> For the binary type though I see a few choices:
>>> 1) Google Protocols (not in distributions, beta)
>>> 2) Science Types/ASN (also not in distributions)
>>> 3) Envelope style (you have to roll your own, but cake to do (think
>>> ID3)).
>>>
>>> Thoughts? Additional ones?
>>>
>>> I lean toward #3 only because I fear the hassle factor in the above ones
>>> (it is also simple and easy to understand). I am hoping we can get our
>>> hands into Innodb and just use its for tables it has.
>>>
>>> Oh, and why do engines need to know? If an engine is not available we do
>>> not want to know about its tables, and if an engine is network based
>>> things get a lot more complicated.
>>>
>>> BTW for methods think:
>>>
>>> tbl = new Table(binary/char array);
>>>
>>> and:
>>>
>>> char *ptr= table->serialze();
>>>
>>> or
>>> char *ptr= table->create_table_string();
>>>
>>> Cheers,
>>> -Brian
>>>
>>> --
>>> _______________________________________________________
>>> Brian "Krow" Aker, brian at tangent.org
>>> Seattle, Washington
>>> http://krow.net/ <-- Me
>>> http://tangent.org/ <-- Software
>>> _______________________________________________________
>>> You can't grep a dead tree.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~drizzle-discuss
>>> Post to : [email protected]
>>> Unsubscribe : https://launchpad.net/~drizzle-discuss
>>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> --
>> Mats Kindahl
>> Lead Software Developer
>> Replication Team
>> MySQL AB, www.mysql.com
>> <mats.vcf>
>
> --
> _______________________________________________________
> Brian "Krow" Aker, brian at tangent.org
> Seattle, Washington
> http://krow.net/ <-- Me
> http://tangent.org/ <-- Software
> _______________________________________________________
> You can't grep a dead tree.
>
>
>
--
Mats Kindahl
Lead Software Developer
Replication Team
MySQL AB, www.mysql.com
begin:vcard
fn:Mats Kindahl
n:Kindahl;Mats
org:Sun Microsystems
adr;quoted-printable:;;Tegv=C3=A4gen 3;Storvreta;SE;74334;Sweden
email;internet:[EMAIL PROTECTED]
title:Lead Replication Software Developer
x-mozilla-html:FALSE
version:2.1
end:vcard
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
