Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Kristian Nielsen]

andrei.el...@pp.inet.fi writes:

> where master maintains an "immediate" (without binlog proxy)
> constant connection to slaves (when it breaks the slave would have to
> take snapshot, or find a binlog service e.g on some other slave).
> [In such a case the master connection would still collect transaction 
> statement till
> prepare or commit to send them to slaves at those points.]
> So here replicated transactions need identification on the binlog-less
> master to display its execution status (which would matter when
> configuration changes and a new master is reelected).

Aha, so delegate the binlogging task to a slave. I guess there needs to be a
binlog *somewhere* for GTID to make sense.

I wonder if it would be possible to avoid introducing a new table (for
example updating the mysql.gtid_slave_pos table on the master). Having two
different places (binlog and mysql.gtid_slave_pos) is already a source of
considerable complexity in the MariaDB GTID, something I would have liked
to avoid. But I felt that it is was really important to allow a slave to use
GTID without --log-slave-updates (hence the need for the table), and also
that adding a mandatory extra table operation to every transaction on the
master was unacceptable (hence the binlog).

Another fundamental design detail of MariaDB GTID is that it supports
out-of-order sequence numbers in the binlog. Again, this is a source of
considerable complexity. This was needed so that 10.0 could enable GTID by
default on the master, so that a simple CHANGE MASTER is sufficient to start
using GTID after an update. For example, an existing master-master ring that
is upgraded to 10.0 will not have gtid_domain_id set and thus will generate
out-of-order GTIDs in the binlog. The implementation is very carefully
designed to always be able to find the correct binlog position from a GTID
even in case of out-of-order sequence numbers.

A consequence is that in MariaDB, it is not in general possible to compare
GTIDs without reference to the binlog. This can also easily affect things
like extending gtid_current_pos or this idea of binlogless master. I wonder
how well understood this generally is by users and developers - and if not,
it is a good sign that it is too complicated...

My point is - maybe it would be an idea to introduce a fully-strict GTID
mode now. In fully-strict mode, GTIDs are guaranteed to be in-order (within
each domain) in every binlog in the system. Ie. there would be a marker at
the start of every binlog file (GTID_LIST perhaps) that this server is
fully-strict and any attempt to disable fully-strict (without CHANGE MASTER)
or connect a fully-strict slave would give an error.

Then in fully-strict mode sequence numbers are strictly monotonic (within a
domain), and GTIDs can be compared simply on sequence numbers. And then a
lot of more advanced GTID features can be implemented in a simpler and more
intuitive (for the user) way. For example, having GTIDs stored in multiple
places is less of a problem, since unification is trivial by just taking the
max sequencenumber within each domain.

 - Kristian.

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[andrei . elkin]

>> I would also raise another but relevant topic of maintaining
>>
>>gtid_"executed"_pos
>>
>> which is an union of all GTID executed regardless of their arrival
>> method. E.g some of foreign (to the recipient server) domains gtid:s may
>
>> The master potentially could be (made) interested in such table
>> should create a replication mode without necessary binlogging on the
>> server.
>
> Sorry, I don't follow.

My fault, sorry!  I should've referred straight to synchronous
replication ...

where master maintains an "immediate" (without binlog proxy)
constant connection to slaves (when it breaks the slave would have to
take snapshot, or find a binlog service e.g on some other slave).
[In such a case the master connection would still collect transaction statement 
till
prepare or commit to send them to slaves at those points.]
So here replicated transactions need identification on the binlog-less
master to display its execution status (which would matter when
configuration changes and a new master is reelected).

I am not dwelling into specifics too deep now, it's just to exemplify
such use case.

>
> Do you mean to create a new table of GTID positions, which will be updated
> by all transactions, whether originating locally or replicated by a slave
> thread?

This one.

>
> Or do you mean to have the mysql.gtid_slave_pos table updated also by
> locally originated transactions?
>
> Or do you mean to have a new system variable @@gtid_executed_pos which is
> constructed from the existing binlog and mysql.gtid_slave_pos_table, similar
> to @@gtid_current_pos, but maybe including more GTIDs somehow?
>
>> This is not really something new as it's exactly how mysql implemented
>> gtid bookkeeping.
>
> I think MySQL originally kept track of GTIDs only in the binlog,
> right?
> And
> binlog was required to be enabled for GTID to work?

True.

>I do remember seeing
> something that relaxed this requirement, but I haven't followed the details.

Like we update `gtid_slave_pos` on slave by the slave applier on the
binlog-less server, so they keep updating a general `gtid executed` table
by any incl the user connection and slave applier.


Cheers,

Andrei

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Kristian Nielsen]

Andrei Elkin  writes:

> I would also raise another but relevant topic of maintaining
>
>gtid_"executed"_pos
>
> which is an union of all GTID executed regardless of their arrival
> method. E.g some of foreign (to the recipient server) domains gtid:s may

> The master potentially could be (made) interested in such table
> should create a replication mode without necessary binlogging on the
> server.

Sorry, I don't follow.

Do you mean to create a new table of GTID positions, which will be updated
by all transactions, whether originating locally or replicated by a slave
thread?

Or do you mean to have the mysql.gtid_slave_pos table updated also by
locally originated transactions?

Or do you mean to have a new system variable @@gtid_executed_pos which is
constructed from the existing binlog and mysql.gtid_slave_pos_table, similar
to @@gtid_current_pos, but maybe including more GTIDs somehow?

> This is not really something new as it's exactly how mysql implemented
> gtid bookkeeping.

I think MySQL originally kept track of GTIDs only in the binlog, right? And
binlog was required to be enabled for GTID to work? I do remember seeing
something that relaxed this requirement, but I haven't followed the details.

 - Kristian.

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Kristian Nielsen]

Hi Andrei!

The @@gtid_current_pos exists for one sole purpose. This is to let the user
promote a slave as the new master and attach the old master as a slave to
the new master.

By using master_use_gtid=current_pos, the exact same command can be used to
attach a slave to the new master, regardless of whether that slave was
previously a slave or a master:

  CHANGE MASTER TO master_host=new_promoted_master

If not using gtid_current_pos (ie master_use_gtid=slave_pos), then to let
the old master become a slave of the new master, the old master's position
must explicitly be set:

  SET GLOBAL gtid_slave_pos=@@gtid_binlog_pos

This is because for efficiency reasons, the master doesn't update the
mysql.gtid_slave_pos in each commit.

So now we can see why only GTIDs with the servers own server_id should
contribute to @@gtid_current_pos. If a GTID was replicated from another
server, that GTID will appear in the @@gtid_slave_pos. If the GTID
originated on this server, it will appear in @@gtid_binlog_pos. The
@@gtid_current_pos is the @@gtid_slave_pos extended with GTIDs originating
on this server, hence only GTIDs with our own server id.

Normally, every GTID in the binlog with a different server id than our own
will already be in the @@gtid_slave_pos as well - since it originated on
another server and was replicated to this server.

Thus, in the normal case, where user did not play tricks with the binlog and
slave state, extending the @@gtid_current_pos as suggested in MDEV-18404 has
no effect - the GTIDs are already in the @@gtid_slave_pos, so
@@gtid_current_pos is unaffected.

And in case the user deliberately modified the state, it should be up to the
user to decide what goes into @@gtid_slave_state and @@gtid_binlog_state.
For example, the MDEV-18404 change would make it impossible on a server to
remove a replicated GTID from the @@gtid_current_pos if --log-slave-updates
(without the drastic RESET MASTER).

Another problem is that the server cannot reliably compare GTIDs with
distinct server ids to decide which one is the most recent. There is no
guarantee that sequence numbers are monotonic across different server ids.
Thus the MDEV-18404 method could create completely invalid positions in
some setups where @@gtid_strict_mode=0 and replication domains are not
strictly maintained.

I don't see the value in MDEV-18404. If the user is updating
@@gtid_binlog_state (itself a very drastic operation), and wants a specific
GTID to go into the slave position - just update the @@gtid_slave_pos with
the desired GTID, don't leave the server with an inconsistent replication
state.

And finally, let me reiterate: I consider the @@gtid_current_pos a design
mistake. Better to just transfer the @@gtid_binlog_pos to the
@@gtid_slave_pos _only_ at the point where an old master is turned into a
slave. This can be done manually already, and it would be simple to
implement automatic support for this with an extra option for CHANGE MASTER.

Hope this helps,

 - Kristian.

Andrei Elkin  writes:

> Kristian, howdy!
>
> I thought it's good to refer to a previous discussion that touched
> 'current_pos' feature now when we need to resolve MDEV-18404 which I
> look at under a specific angle, narrated here.
>
>> Finally, experience has shown that a _lot_ of users get problems when
>> locally done transactions on a slave influence the slave's GTID position. In
>> retrospect, I have realised that CHANGE MASTER TO
>> master_use_gtid=current_pos was a mistake, only slave_pos should be used.
>> Similarly, if a local transaction in a slave's binlog can cause transactions
>> from the master to be silently ignored, it will cause a lot of grief for
>> users.
>
>
> To the definition of
>
>   gtid_current_pos
>
> the docs https://mariadb.com/kb/en/library/gtid/#gtid_current_pos first 
> introduce an intuition
> with
>
>   This variable is the GTID of the last change to the database for
>   each replication domain. Such changes can either be master events
>   (ie. local changes made by user or application),
>   or replicated events originating from another master server.
>
> and then define the variable essentially as an union of two sets:
>
>   For each replication domain, if the server ID of the corresponding
>   GTID in @@gtid_binlog_pos is equal to the servers own server_id, and
>   the sequence number is higher than the corresponding GTID in
>   @@gtid_slave_pos, then the GTID from @@gtid_binlog_pos will be
>   used. Otherwise the GTID from @@gtid_slave_pos will be used for that
>   domain.
>
> The union operation is on a *subset* of @@gtid_binlog_pos (call it B')
> and the whole @@gtid_slave_pos (S):
>
> C = B' \cup S
>
> where C @@gtid_current_pos.
> The @@gtid_binlog_pos subset
> consists of GTID:s originated by @@global.server_id (that is foreign
> ones excluded).
>
> Notice that makes the union non symmetric as binary operator. A foreign
> GTID feels perfectly fine in S and gets into the union result.
>
> The following use case
>
>   

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Kristian Nielsen]

I am sure you can find some who would want something that ignores replicated
GTIDs that duplicate GTIDs originating locally.

I can only say that my experience is that this can cause unexpected
problems, and requires a lot of thought to get a well-defined semantics that
users can understand and will not bring surprises.

A central design decision for MariaDB GTID is _not_ to try to remember the
whole history of GTIDs applied, unlike MySQL GTID. Because of this there are
limitations to what can be done in terms of avoiding duplicate GTIDs - the
server lacks the required information.

Another decision was to allow and handle correctly out-of-order sequence
numbers (eg. gtid_strict_mode=0). This was necessary to be able to generate
GTIDs by default in 10.0. But it again means that detecting duplicates is
harder, and in fact only the master has the required information to do this,
the slave does not (in the general case).

Finally, experience has shown that a _lot_ of users get problems when
locally done transactions on a slave influence the slave's GTID position. In
retrospect, I have realised that CHANGE MASTER TO
master_use_gtid=current_pos was a mistake, only slave_pos should be used.
Similarly, if a local transaction in a slave's binlog can cause transactions
from the master to be silently ignored, it will cause a lot of grief for
users.

Hope this helps,

 - Kristian.

andrei.el...@pp.inet.fi writes:

> Kristian, howdy.
>
> Thanks for a simple CHANGE MASTER ... IGNORE_SERVER_IDS
> that you remind us about! (This time evaded myself alone :-))
> It perfectly covers a cluster circular case.
>
> What motivated me to consider this option for looking for duplicates
> also in gtid_binlog_pos was the following observation.
>
> A duplicate gtid (transaction) can also arrive from a separate session
> of the same server but in this case the gtid_ignore_duplicates rules
> do not apply. Such gtid would silently override an existing.
>
> On the other hand gtid_strict_mode applies to either the ordinary server
> and the slave (by the docs).
>
>MariaDB [test]> show global variables like 'gtid_binlog_pos';
>+-++
>| Variable_name   | Value  |
>+-++
>| gtid_binlog_pos | 0-1-12 |
>+-++
>1 row in set (0.00 sec)
>
>MariaDB [test]> set @@session.gtid_seq_no=11;
>ERROR 1950 (HY000): An attempt was made to binlog GTID 0-1-11 which
>would create an out-of-order sequence number with existing GTID 0-1-12,
>
> Maybe it would not a bad idea to generalize the gtid_ignore_duplicates to
> cover any source duplicate which would become effectively a "soft" mode to 
> silently
> ... reject.
>
> In other words how about extending a gtid (operational) mode as a set to
>
> "gtid_mode" \in {
>   on (override by dups),
>   strict (error out dups)
> +   , soft   (ignore dups)
> }
>
> To other subjects,
>
>> Sachin Setiya  writes:
>>
>>> I have some question related to rpl_slave_state. Suppose A circular
>>> async replication between A < -- > B (gtid_ignore_duplicates on)
>>
>> Why do you set gtid_ignore_duplicates? This option is for multi-source
>> replication:
>>
>>   https://mariadb.com/kb/en/library/gtid/#gtid_ignore_duplicates
>>
>>   "When set, different master connections in multi-source replication are
>>   allowed to receive and process event groups with the same GTID"
>>
>> But you are not using multi-source connection here, there is only one master
>> connection (eg. connection to B on slave A).
>> Thus, the option will do nothing in this case.
>>
>>> Now, we set some temp server_id on server A , lets say `X`. Now the
>>> problem is each event group which
>>> originates from A is executed 2 times. For example we insert into
>>> table t1 and gtid is 0-X-2. The event goes to slave B
>>> B applies it, And send it back to A, Since its server_is different
>>
>> I think here you mean that A has server_id=1 (eg), B has server_id=2, but on
>> A you do
>>
>>   SET server_id=3;
>>   INSERT INTO t1 VALUES (1);
>>
>> But there is no server with server_id=3 anywhere. In this case, you need to
>> break the circle yourself somewhere. For example by CHANGE MASTER ...
>> IGNORE_SERVER_IDS=3 on A.
>>
>> To my knowledge, this has always been so for ring replication.
>>
>>> Andrei suggested a solution of checking rpl_binlog_state in
>>> check_duplicate_gtid, This solution solves some problem but creates
>>
>> It seems you think that --gtid-ignore-duplicates should magically ignore any
>> apply of duplicate GTID. But that is not the case, as the documentation
>> states (though admittedly rather briefly). --gtid-ignore-duplicates is
>> _only_ for multi-source replication (so perhaps unfortunately named).
>>
>> In this case, the conflict is not between GTIDs replicated from different
>> master connections. It is a conflict between a transaction originated 

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[andrei . elkin]

Kristian, howdy.

Thanks for a simple CHANGE MASTER ... IGNORE_SERVER_IDS
that you remind us about! (This time evaded myself alone :-))
It perfectly covers a cluster circular case.

What motivated me to consider this option for looking for duplicates
also in gtid_binlog_pos was the following observation.

A duplicate gtid (transaction) can also arrive from a separate session
of the same server but in this case the gtid_ignore_duplicates rules
do not apply. Such gtid would silently override an existing.

On the other hand gtid_strict_mode applies to either the ordinary server
and the slave (by the docs).

   MariaDB [test]> show global variables like 'gtid_binlog_pos';
   +-++
   | Variable_name   | Value  |
   +-++
   | gtid_binlog_pos | 0-1-12 |
   +-++
   1 row in set (0.00 sec)

   MariaDB [test]> set @@session.gtid_seq_no=11;
   ERROR 1950 (HY000): An attempt was made to binlog GTID 0-1-11 which
   would create an out-of-order sequence number with existing GTID 0-1-12,

Maybe it would not a bad idea to generalize the gtid_ignore_duplicates to
cover any source duplicate which would become effectively a "soft" mode to 
silently
... reject.

In other words how about extending a gtid (operational) mode as a set to

"gtid_mode" \in {
  on (override by dups),
  strict (error out dups)
+   , soft   (ignore dups)
}

To other subjects,

> Sachin Setiya  writes:
>
>> I have some question related to rpl_slave_state. Suppose A circular
>> async replication between A < -- > B (gtid_ignore_duplicates on)
>
> Why do you set gtid_ignore_duplicates? This option is for multi-source
> replication:
>
>   https://mariadb.com/kb/en/library/gtid/#gtid_ignore_duplicates
>
>   "When set, different master connections in multi-source replication are
>   allowed to receive and process event groups with the same GTID"
>
> But you are not using multi-source connection here, there is only one master
> connection (eg. connection to B on slave A).
> Thus, the option will do nothing in this case.
>
>> Now, we set some temp server_id on server A , lets say `X`. Now the
>> problem is each event group which
>> originates from A is executed 2 times. For example we insert into
>> table t1 and gtid is 0-X-2. The event goes to slave B
>> B applies it, And send it back to A, Since its server_is different
>
> I think here you mean that A has server_id=1 (eg), B has server_id=2, but on
> A you do
>
>   SET server_id=3;
>   INSERT INTO t1 VALUES (1);
>
> But there is no server with server_id=3 anywhere. In this case, you need to
> break the circle yourself somewhere. For example by CHANGE MASTER ...
> IGNORE_SERVER_IDS=3 on A.
>
> To my knowledge, this has always been so for ring replication.
>
>> Andrei suggested a solution of checking rpl_binlog_state in
>> check_duplicate_gtid, This solution solves some problem but creates
>
> It seems you think that --gtid-ignore-duplicates should magically ignore any
> apply of duplicate GTID. But that is not the case, as the documentation
> states (though admittedly rather briefly). --gtid-ignore-duplicates is
> _only_ for multi-source replication (so perhaps unfortunately named).
>
> In this case, the conflict is not between GTIDs replicated from different
> master connections. It is a conflict between a transaction originated on a
> master with a transaction replicated from another master.
>
>> write gtid_event in log. But this does not make sense. rpl_slave_state
>> should be used for slave replication usage.
>
> Agree. rpl_binlog_state should not be involved in slave GTID processing.
> There should be a clear separation: rpl_slave_state is what a slave has
> applied from another master. rpl_binlog state is what a master has
> originated.
>
> The gtid_ignore_duplicates option is already very difficult for users to
> understand and use correctly. It would be a mistake to make it even more
> complicated.
>
> Also, this seems to originate from some Galera issue. It is well known that
> Galera was merged prematurely into MariaDB with a broken design, and this
> was never fixed. Galera issues must never influence how non-galera
> replication (which at least attempts to have a proper design) works.

I would support this.

Cheers,

Andrei


___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Kristian Nielsen]

Sachin Setiya  writes:

> I have some question related to rpl_slave_state. Suppose A circular
> async replication between A < -- > B (gtid_ignore_duplicates on)

Why do you set gtid_ignore_duplicates? This option is for multi-source
replication:

  https://mariadb.com/kb/en/library/gtid/#gtid_ignore_duplicates

  "When set, different master connections in multi-source replication are
  allowed to receive and process event groups with the same GTID"

But you are not using multi-source connection here, there is only one master
connection (eg. connection to B on slave A).
Thus, the option will do nothing in this case.

> Now, we set some temp server_id on server A , lets say `X`. Now the
> problem is each event group which
> originates from A is executed 2 times. For example we insert into
> table t1 and gtid is 0-X-2. The event goes to slave B
> B applies it, And send it back to A, Since its server_is different

I think here you mean that A has server_id=1 (eg), B has server_id=2, but on
A you do

  SET server_id=3;
  INSERT INTO t1 VALUES (1);

But there is no server with server_id=3 anywhere. In this case, you need to
break the circle yourself somewhere. For example by CHANGE MASTER ...
IGNORE_SERVER_IDS=3 on A.

To my knowledge, this has always been so for ring replication.

> Andrei suggested a solution of checking rpl_binlog_state in
> check_duplicate_gtid, This solution solves some problem but creates

It seems you think that --gtid-ignore-duplicates should magically ignore any
apply of duplicate GTID. But that is not the case, as the documentation
states (though admittedly rather briefly). --gtid-ignore-duplicates is
_only_ for multi-source replication (so perhaps unfortunately named).

In this case, the conflict is not between GTIDs replicated from different
master connections. It is a conflict between a transaction originated on a
master with a transaction replicated from another master.

> write gtid_event in log. But this does not make sense. rpl_slave_state
> should be used for slave replication usage.

Agree. rpl_binlog_state should not be involved in slave GTID processing.
There should be a clear separation: rpl_slave_state is what a slave has
applied from another master. rpl_binlog state is what a master has
originated.

The gtid_ignore_duplicates option is already very difficult for users to
understand and use correctly. It would be a mistake to make it even more
complicated.

Also, this seems to originate from some Galera issue. It is well known that
Galera was merged prematurely into MariaDB with a broken design, and this
was never fixed. Galera issues must never influence how non-galera
replication (which at least attempts to have a proper design) works.

Hope this helps,

 - Kristian.

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state

[Sachin Setiya]

Hi All,

On Tue, Nov 28, 2017 at 3:03 PM, Sachin Setiya
 wrote:
> Hi Kristian, Andrei
>
> I have some question related to rpl_slave_state. Suppose A circular
> async replication between A < -- > B (gtid_ignore_duplicates on)
> Now, we set some temp server_id on server A , lets say `X`. Now the
> problem is each event group which
> originates from A is executed 2 times. For example we insert into
> table t1 and gtid is 0-X-2. The event goes to slave B
> B applies it, And send it back to A, Since its server_is different
> from A global server_id , this event is not filtered in queue_event().
> Now event goes to handle_sql thread , And check_duplicate_gtid
> function is called. Since rpl_slave_state::hash does not have a
> element
> whose sequence_no is >= 2, So this event is applied again.
>
> Andrei suggested a solution of checking rpl_binlog_state in
> check_duplicate_gtid, This solution solves some problem but creates
> some
> problem. (multi_source.gtid_ignore_duplicates fails in master_sync,
> test for mdev_10715 also fails ). Reason for this is sync_with_master
> calls master_gtid_wait , which internally calls
> gtid_waiting::wait_for_gtid , and this checks rpl_salve_state::hash.
> Although this method can
> also be patched for to look for rpl_gtid_global_binlog_state. But I am
> not sure whether this will solve all problem or create new problem.
Actually this wont work , because I can just look at maximum gtid
>
> Another solution might be somehow update rpl_slave_state::hash when we
> write gtid_event in log. But this does not make sense. rpl_slave_state
> should be used for slave replication usage.
>
> I think we need a more better solution for this.
>
> --
> Regards
> sachin

Experimental code is on this branch
http://buildbot.askmonty.org/buildbot/grid?category=main=bb-10.1-10715


-- 
Regards
Sachin Setiya
Software Engineer at  MariaDB

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp