Re: [Maria-developers] GSoC projects related on data replication.

2013-05-09 Thread Kristian Nielsen
QIU Shuang writes: > A quick question: are the codes related to replication in the path > `sql/rpl_*' and `sql/repl_*'? Much of the replication code is in: sql/log_event.* sql/slave.* sql/log.* sql/sql_repl.* sql/rpl_* sql/repl* Hope this helps, - Kristian.

Re: [Maria-developers] MariaDB GTID first impression

2013-05-12 Thread Kristian Nielsen
Pavel Ivanov writes: > I was thinking about your words above and they make me wonder: if we > have a master and it doesn't write its current state into > rpl_slave_state, then we shut down the master, and while it's out we > failover to another server. Then the first server comes back and > shoul

Re: [Maria-developers] MariaDB GTID first impression

2013-05-13 Thread Kristian Nielsen
Pavel Ivanov writes: > Sorry, I didn't quite understand. Sorry for being unclear, let me elaborate: > You didn't mention .state here. Is it intentional? Does it > try to look into .state after rpl_slave_state and binlogs? Or > e.g. in the situation when binlogs are not found? There is just a s

Re: [Maria-developers] MariaDB GTID first impression

2013-05-13 Thread Kristian Nielsen
Pavel Ivanov writes: > How does it detect if binlogs were closed properly? This is the standard binlog crash recovery going back to 5.1 (or possibly 5.0): When a new binlog file is created, the first Format_description event has a flag LOG_EVENT_BINLOG_IN_USE_F set. At normal shutdown, everythin

Re: [Maria-developers] MariaDB GTID first impression

2013-05-14 Thread Kristian Nielsen
Pavel Ivanov writes: >> My intention for this was to use the existing facilities for such >> backup/restore, >> like mysqldump and XtraDB. > > I didn't check with the 10.0.2, but in 10.0.0 XtraDB didn't compile. > Does it compile now? I do not know, I do not think anyone tried. It may need some

Re: [Maria-developers] MariaDB build logs

2013-05-14 Thread Kristian Nielsen
Marcin Kulisz writes: > as promised I'm attaching build log for maria DB, it's much smaller then > anticipated. Thanks! > From what I can tell it was just one test which failed and rendered whole > build as a > failure. > My build env is based on cowbuilder on fully patched SID chroot. Ah, I

Re: [Maria-developers] MariaDB GTID first impression

2013-05-14 Thread Kristian Nielsen
Pavel Ivanov writes: > We reaching consistency in the backup by shutting down the server. I > would think InnoDB state shouldn't be a problem in such situation. Am > I right? Oh, I see, right then there is certainly no issue. - Kristian. ___ Mailing

Re: [Maria-developers] MariaDB GTID first impression

2013-05-15 Thread Kristian Nielsen
Pavel Ivanov writes: > We reaching consistency in the backup by shutting down the server. I So thinking about this again ... So if we shut down the server gracefully, copy over everything to a new server, delete all binlogs (or omit binlogs from the copy), restart the server - then it could act

[Maria-developers] Proposed changes to GTID syntax

2013-05-17 Thread Kristian Nielsen
I am planning some changes to the user interface of Global Transaction ID. Testing by Elena has shown usability issues that I want to address with these changes, but I wanted to give everyone a change to voice their opinion on the changes. The problem: The issue concerns when a slave server init

Re: [Maria-developers] Proposed changes to GTID syntax

2013-05-17 Thread Kristian Nielsen
Kristian Nielsen writes: > I am planning some changes to the user interface of Global Transaction > ID. Some small simplication/improvement to naming after feedback from Elena: CHANGE MASTER TO master_use_gtid = SLAVE_POS CHANGE MASTER TO master_use_gtid = CURRENT_POS CHANGE

Re: [Maria-developers] Getting community to write missing test cases

2013-05-20 Thread Kristian Nielsen
Elena Stepanova writes: > If we start doing it, Oracle will soon realize that this activity is > based at least as much on commit comments as on the code. As a result, > they will probably start making much vaguer comments than they do now, > which in turn might make merge and related activities

Re: [Maria-developers] MariaDB and MySQL replication

2013-05-20 Thread Kristian Nielsen
Jan Lindström writes: > My question is how compatible replication between > > 1) MySQL (master) -> MariaDB (slave) > 2) MariaDB (master) -> MySQL (slave) In general, it should work, though there are some limitations. It also depends on versions. I always try to make MariaDB (master) -> MySQL (s

Re: [Maria-developers] GTID replication and relay_log.info

2013-05-21 Thread Kristian Nielsen
Pavel Ivanov writes: > There's a well-known problem in MySQL that relay_log.info is not > always in sync with the database state, and killing and restarting > server at an unfortunate time will cause it to re-execute last > statement and potentially break replication. Did you fix this > situation

Re: [Maria-developers] GTID replication and relay_log.info

2013-05-21 Thread Kristian Nielsen
Pavel Ivanov writes: > Could you please elaborate what kind of bug is that? The slave connects to master with GTID. The binlog dump thread on the master starts scanning the last binlog file to find the start position. It finds the position for one replication domain, but aother domain is still s

Re: [Maria-developers] Proposed changes to GTID syntax

2013-05-23 Thread Kristian Nielsen
Kristian Nielsen writes: > I am planning some changes to the user interface of Global Transaction > ID. Testing by Elena has shown usability issues that I want to address with > these changes, but I wanted to give everyone a change to voice their opinion > on the changes. I have now

Re: [Maria-developers] GTID replication and relay_log.info

2013-05-25 Thread Kristian Nielsen
Pavel Ivanov writes: > yet). But from time to time we see failures of rpl.rpl_gtid_crash test > with the output like this: Hm, strange, it does not fail in our Buildbot, which runs hundreds of tests on different hosts... > Can you suggest what debugging information should I gather to > understa

Re: [Maria-developers] GTID replication and relay_log.info

2013-05-27 Thread Kristian Nielsen
Pavel Ivanov writes: > See logs in attachment. It looks clear to me that relay logs get replayed > after crash and don't get deleted. I'm not sure though if the reason of > that can be seen in the logs. No, I agree it's not clear from the logs one way or the other, so good that you mention this.

Re: [Maria-developers] GTID replication and relay_log.info

2013-05-27 Thread Kristian Nielsen
Pavel Ivanov writes: > Thanks for the pointers, Kristian. It turned out that it was an incorrect > merge of GTID-related changes into 10.0.1 tree. In particular I didn't > quite understand the meaning of LINE_FOR_FIRST_MYSQL_5_6 in sql/rpl_mi.cc > and thus left it at the value of 19. Because of t

Re: [Maria-developers] GTID work priorities

2013-05-28 Thread Kristian Nielsen
Pavel Ivanov writes: > we would really love to see "strict gtid mode" and related stuff next. I've now pushed strict mode to 10.0-base, and documented it in the knowledgebase. However, one thing was not clear from your description in MDEV-4478: > 4) Probably with a separate flag we want to pro

Re: [Maria-developers] Buildbot question

2013-06-11 Thread Kristian Nielsen
Sergey Vojtovich writes: > I'm observing pretty odd buildbot behavior with some of my pushes (probably > merges). Bar and Daniel were unable to answer my questions and suggested to > ask you instead. The problem seems to be that Buildbot has become confused about revisions. Look at the top of th

Re: [Maria-developers] Group commit comparison between mysql and mariadb

2013-06-11 Thread Kristian Nielsen
Sergei Petrunia writes: > Hi Kristian, > > I'm wondering, do we have any comparison of MariaDB's binlog group commit > implementation vs MySQL's binlog's group commit? I recall Mats writing a blog This is the best I have: http://kristiannielsen.livejournal.com/16382.html You can also try a

Re: [Maria-developers] Process suggestion for minor issues

2013-06-14 Thread Kristian Nielsen
Rasmus Johansson writes: > This is a suggestion on how to deal with minor or lower priority bugs > and tasks in JIRA, http://mariadb.org/jira . > Please provide your thoughts. Do you think it's an OK way to deal with > the minors or do you prefer some other way? The whole Jira thing is a farce.

Re: [Maria-developers] a bug of ding-qing parallel replication

2013-07-10 Thread Kristian Nielsen
"nanyi607rao" writes: > I find a bug of ding-qing parallel replication. > > let's consider a table like this: > > create table t1( > a int(11) NOT NULL DEFAULT '0', > b varchar(10), > PRIMARY KEY (a) > )ENGINE=InnoDB > > if we do transactions on master(binlog_format=ROW) like this: > tran

[Maria-developers] Status on MDEV-4506, parallel replication

2013-07-12 Thread Kristian Nielsen
So I have been working for some weeks now on implementing MDEV-4506. This task is about making the slave apply (some) events in parallel threads, to speed up replication and reduce the risk of slave not being able to keep up with a busy master. Events are applied in parallel on the slave if they w

Re: [Maria-developers] Structure of MariaDB 10.0 binary logs

2013-07-20 Thread Kristian Nielsen
Giuseppe Maxia writes: > Specifically, I would like to know > * what are the differences between binary logs in MariaDB 5.5 and 10.0 > * what are the differences between binary logs in MariaDB 10.0 and MySQL 5.6. I think the difference is mainly the addition of new events. At least, I cannot t

Re: [Maria-developers] [Commits] Rev 3797: MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg in file:///data0/psergey/dev2/10.0-serg/

2013-08-07 Thread Kristian Nielsen
Sergey Petrunya writes: > revision-id: pser...@askmonty.org-20130802141209-4dqfvx2db8acxwbl > message: > MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg > Temorary fix for a number of replication tests (rpl.rpl_temp_table_mix_row > rpl.rpl_trunc_temp rpl.rpl_current_user rpl.rpl_gtid_mast

Re: [Maria-developers] GTID and removing binary logs

2013-08-08 Thread Kristian Nielsen
Elena Stepanova writes: > Reading recent Pavel Ivanov's reports (the last one is MDEV-4820, but > I think there were some others which you've already closed as fixed), > I started wondering if I conceptually misunderstood something > before. From our previous discussions I made a conclusion that

[Maria-developers] Review of patch for MDEV-4820

2013-08-12 Thread Kristian Nielsen
I took a close look at your patch for MDEV-4820. I think there is a fundamental disconnect. In MariaDB GTID, I do not require or rely on monotonically increasing seqeunce numbers (monoticity is requred per-server-id, but not between different servers). Nor do I enforce or rely on absence of holes

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-12 Thread Kristian Nielsen
Jan Lindström writes: > I found this disturbing and not fully follow what kind of holes are > possible. These GTIDS can be used by human users to start slaves on For example, if you use some of the many filtering options, like --replicate-ignore-*. Then there will be GTIDs on the master that are

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-13 Thread Kristian Nielsen
Pavel Ivanov writes: > Note that the patch I've attached have test case that should reproduce the > problems. Thanks, I've now gone through the testcases also. Let me number the individual tests as follows: 1. Check that gap in seq_no without binlogs doesn't allow to replicate 2. The same test

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-13 Thread Kristian Nielsen
Jan Lindström writes: > Hi, > > On 08/13/2013 09:49 AM, Kristian Nielsen wrote: >> You can always use the contents of the binlogs to know this. You can >> search the binlogs for your GTID and determine if it was a) logged >> in an earlier binlog that was purged, b

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-13 Thread Kristian Nielsen
Vilho Raatikka writes: > What exactly does it require for the user to enable strict or 'relaxed' > (=disasterous imho) mode? Technically, one needs to SET GLOBAL gtid_strict_mode=1 with root privileges (config file option works as well of course). That's it. The practical issue is that old-styl

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-13 Thread Kristian Nielsen
Kristian Nielsen writes: > BTW, for the normal non-advanced user, I think silly stuff like running random > manual transactions on slaves that get into the binlog is a far more common > mistake. Gtid strict mode will cause errors on this, so it needs to be > explictly enabled by

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-14 Thread Kristian Nielsen
>> SHOW VARIABLES or SELECT @@GLOBAL.gtid_strict_mode will show the current >> value. But there is no way to check if it was enabled eg. two weeks ago, if >> that is what you are asking (the value is not stored in the binlog). > That is really pity. Is it really so useful to give the user opportun

Re: [Maria-developers] ALTER ONLINE TABLE syntax

2013-08-14 Thread Kristian Nielsen
Sergei Golubchik writes: > Unless we redefine what "online" mean. If not, there's no choice but > stop claming an "online alter" support and remove ONLINE and LOCK=NONE > from the syntax. > > Opinions? I think "ONLINE" should mean: no exclusive lock is taken while data is migrated from the old t

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-15 Thread Kristian Nielsen
Pavel Ivanov writes: > Have I misunderstood what you said? Yes, totally :-( > By that you are basically saying: I need to support those > who set up lousy multi-master replication, thus I won't treat > domain_id as a single replication stream, I'll treat a pair > (domain_id, server_id) as a sin

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-15 Thread Kristian Nielsen
Vilho Raatikka writes: > Is this documented profoundly already? Should be because probably we'll see > at least some users as ignorant as I am asking the same questions soon. I wrote up some comprehensive documentation in the knowledgebase, and also wrote a series of articles about in on my blog

Re: [Maria-developers] Fwd: [Commits] Rev 3674: MDEV-4645: Incorrect reads of frozen binlog events; FDE corrupted in relay log in file:///home/bell/maria/bzr/work-maria-10.0-base-MDEV-4645/

2013-08-15 Thread Kristian Nielsen
Oleksandr Byelkin writes: > Could you review this? > > (it is https://mariadb.atlassian.net/browse/MDEV-4645 ) Ok, but it will be at least a couple more days before I will have time to start. If urgent, you can also ask Sergei Golubchik, IIRC this is the patch from Jeremy Cole that he asked me

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-16 Thread Kristian Nielsen
Ok, I've pushed to 10.0-base a patch for MDEV-4820. revid:kniel...@knielsen-hq.org-20130816131025-etjrvmfvupsjzq83 As far as I can determine (and I checked quite carefully), this fixes all the problems you mentioned in the bug description and in your test cases. But I could have misunderstood som

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-19 Thread Kristian Nielsen
Pavel Ivanov writes: > I think to fix this bug we should stop using gtid_slave_pos as > indication of the current db state. We should make it possible to Agree. > change gtid_binlog_pos when there's no events in binlogs. And when Ok. Actually, I think we should expose the real binlog state (wh

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-22 Thread Kristian Nielsen
Pavel Ivanov writes: > Could you say are you working on these? Is there an ETA? > This is blocking us from pushing MariaDB into testing in the > near-production environment, and I'm hesitant to implement fixes > myself because I'd think you'll do it completely differently. If you have time to te

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-22 Thread Kristian Nielsen
Kristian Nielsen writes: > If you have time to test this then that would be a nice help. I'll see if I > can come up with a quick patch (ie. later today or tomorrow). Please try this patch and let me know if you find any issues. I still need to implement test cases, but it seems t

Re: [Maria-developers] Buildbot failure in 10.0-base

2013-08-23 Thread Kristian Nielsen
Sergei Petrunia writes: > 10.0-base started to get compile failures on [nearly] all hosts after your > yesterday push: Thanks for the notice! I've pushed a fix and it seems to have solved it. - Kristian. ___ Mailing list: https://launchpad.net/~mari

Re: [Maria-developers] Fwd: [Commits] Rev 3674: MDEV-4645: Incorrect reads of frozen binlog events; FDE corrupted in relay log in file:///home/bell/maria/bzr/work-maria-10.0-base-MDEV-4645/

2013-08-23 Thread Kristian Nielsen
Oleksandr Byelkin writes: > Could you review this? > > (it is https://mariadb.atlassian.net/browse/MDEV-4645 ) Ok to push. - Kristian. ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsub

[Maria-developers] Additional documentation for GTID

2013-08-23 Thread Kristian Nielsen
Hi Daniel, Ian, I have implemented a new system variable for global transaction ID and pushed it into 10.0-base (so it should appear in 10.0.5 I suppose). I have written some text for documenting the new variable, appended below. Can one of you help me get this text into the Knowledgebase with p

Re: [Maria-developers] Review of patch for MDEV-4820

2013-08-24 Thread Kristian Nielsen
Pavel Ivanov writes: > I took 10.0-base r3685. Started new just bootstrapped server with > server_id = 1. It has @@global.gtid_binlog_pos, > @@global.gtid_slave_pos and @@global.gtid_current_pos empty. Then I > execute > > set global gtid_binlog_state = '0-10-10' > > After that @@global.gtid_binl

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-08-28 Thread Kristian Nielsen
Pavel Ivanov writes: > You have the following comment in the queue_event() in sql/slave.cc: > > /* > Do not queue any format description event that we receive after a > reconnect where we are skipping over a partial event group received > before the reconnect. > > (If

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-08-29 Thread Kristian Nielsen
Pavel Ivanov writes: >> But this must be the same problem with normal replication? Whenever the slave >> decides to rotate the relay log, it will write a format description event >> created by itself with no following format description created on the >> master. So it seems this must work somehow

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-08-30 Thread Kristian Nielsen
Pavel Ivanov writes: > The attached patch doesn't work properly because the format > description event it saves is different from what master has sent. It > has different number_of_event_types and potentially it has garbage (or > incomplete data) in post_header_len. Aha, so it's because of this,

Re: [Maria-developers] FWD: Status on MDEV-4506, parallel replication

2013-09-01 Thread Kristian Nielsen
Michael Widenius writes: >> On the master, I implemented --binlog-commit-wait-count=N and >> --binlog-commit-wait-usec=T. A transaction will wait at most T microseconds >> for at least N transactions to queue up and be ready for group commit. This >> allows to deliberately delay transactions on t

Re: [Maria-developers] Replicating same server_id problem

2013-09-03 Thread Kristian Nielsen
Pavel Ivanov writes: > --replicate-same-server-id flag which as I understand (when set to 0) > controls two things: > 1) It doesn't allow slave to connect to a master with the same server_id. > 2) Slave ignores all binlog events in the replication stream that have > the same server_id as slave. >

Re: [Maria-developers] Replicating same server_id problem

2013-09-03 Thread Kristian Nielsen
Kristian Nielsen writes: > The main problem I see is what should be the default? I suppose we cannot > safely change the default for --replicate-same-server-id. On the other hand, > if users explicitly set --replicate-same-server-id=0, then it really does not > seem correct that some

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-09-03 Thread Kristian Nielsen
Pavel Ivanov writes: > Why not change Format_description_log_event::write()? It seems > actually that it will be more correct if we change it. I made the > following change (line numbers may be off) and it worked: Agree, that sounds like a good idea, thanks! >> If you are interested in helping

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-09-04 Thread Kristian Nielsen
Pavel Ivanov writes: > Thanks. This patch works for me. Ok, I'll push it then. > Note that there's also Query_log_event::peek() that should have the > same dependency on common_header_len. It's hard (if at all possible) > for me to experience that code path in my tests, but for the sake of > ge

Re: [Maria-developers] Rollback causes assertion with Galera 10.0

2013-09-09 Thread Kristian Nielsen
Jan Lindström writes: > #3 0x7f1ff9877eb2 in __GI___assert_fail (assertion=0x109b4ff "empty()", > file=0x109b4d8 "/home/jan/mysql/galera-10.0/sql/log.cc", line=294, function= > 0x10a01e0 "void > binlog_cache_data::reset()") at assert.c:101 > #4 0x009497d4 in binlog_cache_data::rese

Re: [Maria-developers] Rollback causes assertion with Galera 10.0

2013-09-10 Thread Kristian Nielsen
Jan Lindström writes: > (gdb) p cache_log > $1 = {pos_in_file = 11936128518282651045, end_of_file = 18446744073709547520, > (gdb) p *cache_log.pos_in_file > Cannot access memory at address 0xa5a5a5a5a5a5a5a5 Ouch. That looks like memory corruption :-( Seems something has overwritten the start

Re: [Maria-developers] Replicating same server_id problem

2013-09-11 Thread Kristian Nielsen
Pavel Ivanov writes: > Kristian Nielsen writes: >> So perhaps it is better to just say that --replicate-same-server-id does not >> apply to GTID mode at all. Instead, in GTID mode, if we receive an event with >> our own server_id and smaller seq_no than what we alread

Re: [Maria-developers] [Commits] Rev 3810: MDEV-4994: Crash in dynamic column API in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/

2013-09-13 Thread Kristian Nielsen
sa...@askmonty.org writes: > At file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/ > > > revno: 3810 > revision-id: sa...@askmonty.org-20130911151436-rjz6pw6fh6rtj83e > parent: s...@mariadb.org-20130904072837-u3zj89fq4xvb4cnp > commi

Re: [Maria-developers] [Commits] Rev 3810: MDEV-4994: Crash in dynamic column API in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/

2013-09-13 Thread Kristian Nielsen
Sorry about the empty reply. Slip of my fingers, Emacs provides very nice facilities for working quickly, and just a bit too quickly in this case ;-) - Kristian. ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@l

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-13 Thread Kristian Nielsen
Monty, Thanks for the comments so far. I have pushed fixes to 10.0-knielsen, and some specific replies are below. - Kristian. Michael Widenius writes: > + --slave-parallel-threads=# > + If non-zero, number of threads to spawn to apply in > + parallel events on the slave that were group-comm

Re: [Maria-developers] Review of patch for MDEV-4820

2013-09-15 Thread Kristian Nielsen
Pavel Ivanov writes: > 3. Set gtid_binlog_state and gtid_slave_pos to '0-3-10' on S1 and to > '0-1-1' on S2. Try to start slave on S2. Now I get error "slave has > diverged". What gives? It's not diverged, it's just behind. Yes, it is diverged. S1 has binlog state '0-3-10'. This means that the

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-17 Thread Kristian Nielsen
Michael Widenius writes: > + if (list->wakeup_subsequent_commits_running) > + { > +mysql_mutex_lock(&list->LOCK_wait_commit); > +list->wakeup_subsequent_commits_running= false; > +mysql_mutex_unlock(&list->LOCK_wait_commit); > > Why do we need the mutex above? I

Re: [Maria-developers] [All] MariaDB's best (download) month ever

2013-09-17 Thread Kristian Nielsen
Colin Charles writes: > It's 17.09 today, and we've hit 50,160 total downloads (this number is > considered company-internal, we don't share this number externally). "company-internal"? What have you been smoking? Colin, you are supposed to be the community guy for crying out lout :-( This is *

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-19 Thread Kristian Nielsen
Jan Lindström writes: > I need help. After merge galera-10.0 with 10.0, rollback asserts. I seem not > to > be able to find the actual reason. I added some extra output on log.cc to dump > out the cache_log data from both trx_cache and stmt_cache, but to me these > values do not really mean anyt

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-20 Thread Kristian Nielsen
Jan Lindström writes: > Yes, I did run using valgrind, while it reported some uninitialized variables, > not really any memory corruptions.  Attached, the log from node0. Similarly, Well, uninitialised variables are exactly what I would expect could lead to things looking like you have shown so

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-20 Thread Kristian Nielsen
Jan Lindström writes: > Correct log file attached. > ==00:00:02:33.708 25289== Use of uninitialised value of size 8 > ==00:00:02:33.708 25289==at 0x5EF865B: _itoa_word (_itoa.c:179) > ==00:00:02:33.708 25289==by 0x5EFCB91: vfprintf (vfprintf.c:1654) > ==00:00:02:33.708 25289==by 0x5F

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-23 Thread Kristian Nielsen
Michael Widenius writes: > Kristian> We _do_ need a full memory barrier here (memory barrier is implied > in taking a > Kristian> mutex). Otherwise the compiler or CPU could re-order the setting of > the > Kristian> wakeup_subsequent_commits_running flag with the reads and writes > done in the

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-23 Thread Kristian Nielsen
Michael Widenius writes: > What should happen if you kill a replication thread is that > replication should stop for that master. > > Kristian> This needs more thought, I think ... certainly something looks not > right. > > After looking at the full code, I think that the logical way things > sh

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-24 Thread Kristian Nielsen
Jan Lindström writes: > Full unedited log using --valgrind and --mysqld=--debug=+d at location: > https:/ > /www.dropbox.com/s/dcufp7l1cch8dfs/mysql.err.gz Ok, thanks. Hm, this log looks different from what I am used to seeing when I use mysql-test-run.pl --debug. From looking at the mysql-tes

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-24 Thread Kristian Nielsen
tuff for that thread id only. Hope this helps, - Kristian. Kristian Nielsen writes: > Jan Lindström writes: > >> Full unedited log using --valgrind and --mysqld=--debug=+d at location: >> https:/ >> /www.dropbox.com/s/dcufp7l1cch8dfs/mysql.err.gz > > Ok, thank

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Jan Lindström writes: > static int binlog_savepoint_set(handlerton *hton, THD *thd, void *sv) > { > #ifdef WITH_WSREP >   if (wsrep_emulate_bin_log) DBUG_RETURN(0); > #endif /* WITH_WSREP */ > > wsrep_emulate_bin_log = 1, thus no savepoint is written to bin log Ah, I missed that. Right, then I

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Jan Lindström writes: > The point I do not understant is at ha_savepoint > > we have again: > > if ((err= ht->savepoint_set(ht, thd, (uchar *)(sv+1)+ht->savepoint_offset))) > > (gdb) p *sv > $3 = {prev = 0xa5a5a5a5a5a5a5a5, name = 0x7f505c007390 "A", length = 1, > ha_list > = 0xa5a5a5a5a5a5a5a5,

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-25 Thread Kristian Nielsen
Hi Monty, So as promised, I took a look at the existing code for STOP SLAVE, and came up with some ideas for how to extend this to handle parallel replication. In existing code, STOP SLAVE ends up in terminate_slave_threads(). The interesting part here is the SQL thread; stopping the IO thread sh

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Kristian Nielsen writes: >> if ((err= ht->savepoint_set(ht, thd, (uchar *)(sv+1)+ht->savepoint_offset))) > Note that it is not "sv+1". It is ((uchar *)sv+1) + ht->savepoint_offset. Sorry, I messed up that explanation. The point is: sv points to a SAVEPOINT object.

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-25 Thread Kristian Nielsen
Kristian Nielsen writes: > So we should introduce >rgi->abort_slave, and set it during STOP SLAVE for all queued rgi entries. Actually, we do not need to introduce another flag. We can just check rgi->rli->abort_slave. - Kristian.

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-27 Thread Kristian Nielsen
seppo.jaak...@codership.com writes: > Did I understand it right that the problem is with "ROLLBACK TO > SAVEPOINT...", where the designated savepoint does not exist? Did you No. It is a very simple issue. The beginning of binlog_savepoint_set() reads: #ifdef WITH_WSREP if (wsrep_emula

[Maria-developers] Comments for parallel replication patch (MDEV-4506)

2013-10-07 Thread Kristian Nielsen
Hi Monty, I read through your latest patch for the MDEV-4506 task, parallel replication. Just to re-cap, this is a work-in-progress patch, so these are just some comments along the way that I hope will be useful. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_s

Re: [Maria-developers] Wrong SQL thread position reporting after IO thread restart

2013-10-11 Thread Kristian Nielsen
Pavel Ivanov writes: > Here's my approach to fixing both of these problems if you are interested. Thanks. My patch does basically the similar thing, though I changed some details (MDEV-5130). - Kristian. ___ Mailing list: https://launchpad.net/~mari

Re: [Maria-developers] Comments for parallel replication patch (MDEV-4506)

2013-10-18 Thread Kristian Nielsen
Michael Widenius writes: >> What about Rotate_log_event::do_update_pos() and >> Stop_log_event::do_update_pos()? >> >> They currently access the flag in rli. I am not sure that is right, as > this >> could be far ahead of the event they are actually executing? > > Aren't the above events handled

[Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release

2013-11-01 Thread Kristian Nielsen
Hi Monty, I was asked to prepare MDEV-4506, parallel replication, for release of 10.0.5 on Monday. For this, I did merge to 10.0-base and then to 10.0. I have pushed the results here: lp:~maria-captains/maria/mdev-4506-base # for 10.0-base merge lp:~maria-captains/maria/mdev-4506

Re: [Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release

2013-11-03 Thread Kristian Nielsen
Michael Widenius writes: > Did you forget to push the latest changes into mdev-4506-base ? Yes, sorry about that. > If all tests passes in your 10.0-base, please push the changes into > 10.0-base tree. Done. > I have now one failing test in suite/rpl in mdev-4506 that I am trying > to fix bef

Re: [Maria-developers] Semisync plugin incompatibility

2013-11-11 Thread Kristian Nielsen
Pavel Ivanov writes: > We've noticed recently that semisync_master plugin in MariaDB (which > apparently was fully inherited from MySQL) is seriously incompatible > with our understanding of the purpose of semi-sync replication. This > incompatibility was apparently introduced as a fix for > http

Re: [Maria-developers] Semisync plugin incompatibility

2013-11-14 Thread Kristian Nielsen
Pavel Ivanov writes: > So basically my question is: if I prepare a patch that will restore > the original behavior of semi-sync replication (and remove the tests > added for Bug#45672) will that be acceptable for MariaDB? I don't have anything against it, as I said I do not have much opinion on

Re: [Maria-developers] Slave can take a very long time to start replication

2013-11-14 Thread Kristian Nielsen
Pavel Ivanov writes: > start replicating it passes GTID to start from, master finds binlog > file where the earliest GTID is located and then scans through that > file to find the exact binlog position to start sending binlog events > from. If this binlog file is pretty big then scanning can take

Re: [Maria-developers] parallel replication monitoring

2013-11-14 Thread Kristian Nielsen
Giuseppe Maxia writes: > I am looking for: > * how many workers are used; The number of workers created is determined by the value of @@slave_parallel_threads. That many worker threads are in the replication worker thread pool, shared among all multi-source master connections. Workers are sched

Re: [Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server

2013-11-15 Thread Kristian Nielsen
Sergey Vojtovich writes: > * binlog-max-flush-queue-time This is not needed (it is about MySQL 5.6 group commit, MariaDB uses a different implementation. I suppose one could say it is always set to 0). > * binlog-row-image This is not merged to 10.0 yet (as far as I know). It looks like it wou

Re: [Maria-developers] parallel replication monitoring

2013-11-15 Thread Kristian Nielsen
Giuseppe Maxia writes: > * which database they are running is important even if the parallel > replication is not split by schema. As a DBA, I need to know at a glance > where the action is occurring. SHOW PROCESSLIST gives me that information, > but it doesn't tell me the GTID, which is the info

Re: [Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server

2013-11-15 Thread Kristian Nielsen
Sergey Vojtovich writes: >> > * enforce-gtid-consistency >> > * gtid-mode >> >> This is for MySQL 5.6 GTID, which will not be in MariaDB (we have a >> completely >> different GTID implementation). So they should not be implemented, nor should >> we have dummy options for compatibility. > You me

Re: [Maria-developers] Accessing rpl_global_gtid_binlog_state is not consistently protected

2013-11-18 Thread Kristian Nielsen
Pavel Ivanov writes: > sql/log.cc has highly inconsistent pattern of accessing > rpl_global_gtid_binlog_state. In some places it's protected by mutex > rpl_global_gtid_binlog_state.LOCK_binlog_state, in some places it's > protected by global mutex LOCK_rpl_gtid_state, and in some places it's > no

Re: [Maria-developers] MariaDB-10.0-beta sysbench results

2013-11-22 Thread Kristian Nielsen
MARK CALLAGHAN writes: > Resources are finite, but it would be great if MariaDB also did something > about single-thread performance regressions. MySQL has not done/said much Do you have some pointers about where to start with this? I am thinking about which types of queries / benchmarks to chec

[Maria-developers] MDEV-5115 review

2013-12-04 Thread Kristian Nielsen
Hi Serg, Here is my review of your MDEV-5115 patch. First, a general comment: Another way to fix this would be a smaller patch. It would not attempt to implement anything related to the new event types. Instead, it would just have a small bit of code that allows to read the binary format of the

[Maria-developers] Need Review: MDEV-5509: Seconds_behind_master incorrect in parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-01-08 Thread Kristian Nielsen
Could someone review this patch for MDEV-5509? I am in particular worried that this might change some subtle behaviour of Seconds_Behind_Master in unexpected ways. Of course I tried my best to make sure such could not occur, but the semantics is really tricky and poorly documented, and I do not th

Re: [Maria-developers] benchmarking next steps

2014-01-20 Thread Kristian Nielsen
MARK CALLAGHAN writes: > Are there other goals for single-thread performance regressions? There are > many that have nothing to do with replication code although almost every > single-threaded replication hurts replication. I am planning to do a general in-depth analysis of single-threaded perf

Re: [Maria-developers] [Commits] Rev 3964: MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg in file:///home/psergey/dev2/10.0-fix1/

2014-01-20 Thread Kristian Nielsen
Sergey Petrunya writes: > > revno: 3964 > revision-id: pser...@askmonty.org-20140120123657-g1z4g11xmsdiw2dc > committer: Sergey Petrunya > message: > MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg > Undo the previous band-aid fix

Re: [Maria-developers] New message from Yoshinori Matsunobu

2014-01-20 Thread Kristian Nielsen
"Facebook" writes: > I read your latest blog post about profile-guided optisation, which is really > interesting. > > I have one question.. What perf options/arguments did you use to get > INST_RETIRED.ANY and ICACHE.MISSES ? perf stat -e cache-misses? I'm not sure > what options give these nu

Re: [Maria-developers] uint6korr optimization

2014-01-23 Thread Kristian Nielsen
Alexey Botchkov writes: > Only question i have with this is if there's a possibility to make the > hf_mi_uint6korr(dest,src) macro a function. > So we can write as usual > dest= hf_mi_uint6korr(src) > > Didn't find how that can nicely be done with the assembler code. Do it like this: static

Re: [Maria-developers] uint6korr optimization

2014-01-23 Thread Kristian Nielsen
Kristian Nielsen writes: > Do it like this: > static inline ulonglong > mi_uint6korr(const void *p) > { > uint32 a= *(uint32 *)p; > uint16 b= *(uint16 *)(4+(char *)p); > ulonglong v= ((ulonglong)a | ((ulonglong)b << 32)) << 16; > asm ("bswap

Re: [Maria-developers] [Commits] Rev 3965: MDEV-5403 - Reduce usage of LOCK_open: tc_count in lp:maria/10.0

2014-01-24 Thread Kristian Nielsen
Sergey Vojtovich writes: > +#ifdef __cplusplus > +template class Atomic_type_triat; > + > + > +template class Atomic_type_triat > +template class Atomic_type_triat > +template > class > Atomic > + T operator++() { return add(1) + 1; } [lots and lots of more template code snipped] I am

Re: [Maria-developers] [Commits] Rev 3965: MDEV-5403 - Reduce usage of LOCK_open: tc_count in lp:maria/10.0

2014-01-24 Thread Kristian Nielsen
I really do not like hiding the type. If you did not think carefully about the type of the underlying variable, you _really_ should not be using something as complex and subtle as an atomic operation. And I _really_ do not like that we would now have two _different_ ways to do atomic operations, o

[Maria-developers] Next steps in improving single-threaded performance

2014-01-24 Thread Kristian Nielsen
I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejour

<    1   2   3   4   5   6   7   8   9   10   >