Re: [Maria-developers] Comments/thoughts on patch

2015-03-04 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: I wrote attached patch (so that i would understand what code actually does) and found myself while refactoring also doing a change, namely that i removed the check for server_id in a couple of places. it still passes all tests... could you comment on

Re: [Maria-developers] Comments/thoughts on patch

2015-03-04 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: hmm...i'm not sure I get it... is it a bug or a feature that the rouge transactions is skipped by Slave2 in statement based replication, skipping 0-2-3 and 0-2-4 can cause arbitrary data drift, right ? They are not skipped. The bug is in your patch (I

Re: [Maria-developers] Contributing in Support for GTID in mysqlbinlog for GSoC'15

2015-02-24 Thread Kristian Nielsen
Rohit Kashyap rkgud...@gmail.com writes: I am having a doubt, GTID assignss unique identifier to every transaction. And gtid_mode variable is not dynamic.Should the constraint of matching the master and replica be dropped to implement this. I am sorry, I do not understand what you are

Re: [Maria-developers] Contributing in Support for GTID in mysqlbinlog for GSoC'15

2015-02-23 Thread Kristian Nielsen
Rohit Kashyap rkgud...@gmail.com writes: I am Rohit Kashyap, I am currently working on MDEV-7569 and recently came over to your post on Support for GTID in mysqlbinlog [MDEV-4989] I would like to ask you for guidance and leads on this so that I can take it up as my GSoC'15 Project under

Re: [Maria-developers] Row-based replication of old DECIMAL to new DECIMAL

2015-02-13 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: Make it print an error: The table 'db.table' has deprecated columns types incompatible with Row-based replication. (the same text with #b, but this time an error instead of a note). I leave it to Kristian... First, note

Re: [Maria-developers] Interest in Contributing to MariaDB and participation in GSoC 2015

2015-02-09 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: Fyi: we have already fixed 4991 (after having quickly browsed the MDEV)... i'll try to contribute it shortly Really? Cool ... looking forward to seeing it. - Kristian. ___ Mailing list:

Re: [Maria-developers] Interest in Contributing to MariaDB and participation in GSoC 2015

2015-02-08 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: On Jan 24, Tekang Check wrote: the 2015 round of the Google Summer of Code. While looking at the projects. I'm interested in Database replication and would love to do some work in Take a look at

Re: [Maria-developers] [Commits] 101d7eb: MDEV-4987: Sort by domain_id when list of GTIDs are output

2015-02-03 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: I originally thought of pushing it to 10.1 (10.1.4). Do you want me to push it to 10.0 as well? Hm. I guess it's best to push only to 10.1? - Kristian. ___ Mailing list:

Re: [Maria-developers] [Commits] 101d7eb: MDEV-4987: Sort by domain_id when list of GTIDs are output

2015-02-02 Thread Kristian Nielsen
nirb...@mariadb.com writes: revision-id: 101d7eb963816514362da8a98fc7db7135181910 committer: Nirbhay Choubey timestamp: 2015-01-31 21:48:14 -0500 message: MDEV-4987: Sort by domain_id when list of GTIDs are output Added logic to sort gtid list based on domain_id before populating them in

Re: [Maria-developers] [Commits] Rev 4562: MDEV-7402 'reset master' hangs, waits for signalled COND_xid_list in lp:~maria-captains/maria/10.0

2015-01-19 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: revno: 4562 revision-id: ser...@pisem.net-20150118130747-rs67yzzk8a7ebwdd parent: ser...@pisem.net-20150118002113-snztyzwvzc500c2m fixes bug: https://mariadb.atlassian.net/browse/MDEV-7402

[Maria-developers] Buildbot slow upload

2015-01-12 Thread Kristian Nielsen
Hi Serg, Daniel, Some time ago there was discussion of slow Buildbot upload speed (sorry, don't have the original mail thread to reply to). I found some new information on this. Yesterday I had to do a custom build, but was not able to because my tarbake upload kept getting stuck. So I had to

[Maria-developers] Buildslave terrier2 disabled due to KVM problems, needs fixing to be re-enabled

2015-01-05 Thread Kristian Nielsen
Jani, Daniel, I discovered that our buildslave terrier2 seems to have problems with its KVM. Builds are incredibly slow, and it appears that it is not using KVM at all. So it is apparently using just QEMU, and soft-emulating the CPU. This is too slow to be useful, I think. So for now, I've

Re: [Maria-developers] Row-based replication of old DECIMAL to new DECIMAL

2014-12-23 Thread Kristian Nielsen
On Dec 22, Alexander Barkov wrote: Hi Sergei, Kristian, So, there are a lot of details in that bug and in the discussion, and I did not get a full overview of the problem, or what you were asking from me, but I'll try to answer as best I can. Do I understand correctly that row-based

Re: [Maria-developers] Review of MDEV-162: Enhanced semisync replication

2014-12-23 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: uploaded dump-thread-enhancements.changes-v2-v3.patch which releases LOCK_binlog_end_pos while sending hb. and...to repeat, the dump-thread-enhancements is a prerequisite for this. Thanks! I've pushed the dump-thread enhancements v3 and the MDEV-162

Re: [Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-17 Thread Kristian Nielsen
Pavel Ivanov piva...@google.com writes: So the slave coordinator (or I don't remember how you call it) reads relay log ahead of the last executing transaction? I.e. it will read and assign to threads T1.1, T1.2, then it will read T1.3, detect that there are no threads available for execution,

[Maria-developers] Review of MDEV-162: Enhanced semisync replication

2014-12-12 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: Thanks for the MDEV-162 patch! Here is my review. I think the patch looks fine, and we should take it. Just a few questions below, and one minor comment: === modified file 'mysql-test/suite/rpl/t/rpl_semi_sync.test' ---

Re: [Maria-developers] Review of MDEV-162: Enhanced semisync replication

2014-12-12 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: I think it might be easier you just took and applied it to your favorite branch (hence i'll won't send a new version of the patch). Ok. however there is one think that I discovered that needs to be fixed with the interaction with the Dump Thread

Re: [Maria-developers] JIRA ping

2014-12-12 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: i addressed your comments and I uploaded 2 new patches to JIRA. 1) a new complete patch 2) a patch that is changes from v1 to v2. the problem was that I didn't update info-last_pos and linfo-pos correctly in send_format_descriptor_event so the

Re: [Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-10 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: i didn't read the fully reply, sorry for that, but still wanted to say that even if @@replicate_expect_conflicts might only be useful to a very small minority of users, i think it might be worth to implement...if it's not too hard/time-consuming... It

Re: [Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-09 Thread Kristian Nielsen
Pavel Ivanov piva...@google.com writes: This is not entirely true, right? Let's say master binlog has transactions T1.1, T1.2, T1.3, T1.4, T2.1, T1.5, T2.2 (where T1.* have domain_id = 1 and T2.* have domain_id = 2) and slave has 3 parallel threads. Then as I understand threads will be

Re: [Maria-developers] JIRA ping

2014-12-08 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: I light of recent I did see any of your comments until by accident just now, I now mail you directly that I updated both MDEV-162 and MDEV-7257. Sorry for the late answer, I do not follow _all_ Jira activity, so I did not see your comments on those

Re: [Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-08 Thread Kristian Nielsen
Pavel Ivanov piva...@google.com writes: Does this mean that group commit will be possible if slave is able to execute several transactions consecutively while previous transaction commits/fsyncs? Yes. Because the following transaction is in all cases allowed to start as soon as the prior

[Maria-developers] Detailed explanation of the operation of optimistic parallel replication

2014-12-05 Thread Kristian Nielsen
i don't understand what only_commits is, how can it prepare transaction queue up for group commit if only 1 is prepared in parallel ? Right, so this is an important optimisation, let me try to explain that, and then I'll answer the other points in a different (shorter) mail for clarity.

Re: [Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-05 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: A slightly off topic question that struck me last night: won't all parallel transactions conflict when updating the slave_gtid_pos table ? They would, if the GTID was not carefully designed in anticipation of this issue. So the GTID position is updated

[Maria-developers] More suggestions for changing option names for optimistic parallel replication

2014-12-04 Thread Kristian Nielsen
I discussed with Monty, and we came up with some more suggested changes for the options used to configure the optimistic parallel replication feature (MDEV-6676, https://mariadb.atlassian.net/browse/MDEV-6676). The biggest change is to split up the --slave-parallel-mode into multiple options. I

Re: [Maria-developers] [Commits] Rev 4369: MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock-recursive in lp:maria/5.5

2014-11-27 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: I failed to understand why the patch would fix the assertion mentioned in the bug report. But the patch itself looks correct. That's a bit tricky... I'll explain just one of the simplest side effects of this bug, which is half way to this assertion

Re: [Maria-developers] add a rotate event at begin of relay log make greate help

2014-11-27 Thread Kristian Nielsen
nanyi607rao nanyi607...@gmail.com writes: currently, a relaylog is immediately purged after it be completely replayed (relay_log_purge=1). and if IO thread crashes, we should delete all relay logs and fetch master binlog again from exec_master_log_pos/file (relay_log_purge=ON). I think that

Re: [Maria-developers] [Commits] Rev 3902: MDEV-7123: MariaDB 10.0.14 Galera node shutdown with signal 11 in lp:~maria-captains/maria/maria-10.0-galera

2014-11-26 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: revno: 3902 revision-id: nirb...@mariadb.com-20141123163533-9ov1yirj26cur301 parent: nirb...@mariadb.com-20141119183337-n5spskf54beqbhif committer: Nirbhay Choubey nirb...@mariadb.com branch nick: maria-10.0-galera timestamp: Sun 2014-11-23

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera

2014-11-24 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: Suppose we have this transaction: BEGIN GTID 2-1-100 INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); COMMIT; What happens in the following scenario? CHANGE MASTER TO master_use_gtid=current_pos, ignore_domain_ids=(2); START

Re: [Maria-developers] [Commits] Rev 4507: MDEV-7157 plugins.server_audit fails sporadically in buildbot. in file:///home/hf/wmar/mdev-7157/

2014-11-24 Thread Kristian Nielsen
holyf...@askmonty.org writes: revno: 4507 revision-id: holyf...@askmonty.org-20141123225345-p2xd7yrklzee10u1 parent: ser...@pisem.net-20141121192039-d0lv6cj96kg5pw02 committer: Alexey Botchkov holyf...@askmonty.org branch nick:

Re: [Maria-developers] [Commits] Rev 4369: MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock-recursive in lp:maria/5.5

2014-11-24 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: revno: 4369 revision-id: s...@mariadb.org-20141120094823-fozdsrm1kzv46kxi parent: jplin...@mariadb.org-20141119182734-cwbaed0ka90ocj5e committer: Sergey Vojtovich s...@mariadb.org branch nick: 5.5 timestamp: Thu 2014-11-20 13:48:23 +0400 message:

Re: [Maria-developers] [Commits] Rev 4367: MDEV-7026: Race in InnoDB/XtraDB mutex implementation can stall or hang the server. in http://bazaar.launchpad.net/~maria-captains/maria/5.5

2014-11-20 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: +/* + os_atomic_test_and_set_byte_acquire() is a full memory barrier on x86. But + in general, just an aquire barrier should be sufficient. */ +# define os_atomic_test_and_set_byte_acquire(ptr, new_val) \ __sync_lock_test_and_set(ptr,

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5

2014-11-19 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: thanks for your feedback! Since we're short on time and you have an idea of a proper fix, what if we do it the other way around: you'll implement a fix and I'll review it from PPC64 perspective? Hm, ok then. You can find the patch here:

Re: [Maria-developers] [Commits] Rev 4484: MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails in lp:~maria-captains/maria/10.0

2014-11-18 Thread Kristian Nielsen
By the way, it would be a lot better if we could do it so that the statement is only marked unsafe if it modifies more than one row. I would guess that UPDATE of a unique key are by far most common with single-row updates, and such updates should be safe. So it would be great if they did not get

Re: [Maria-developers] MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails in lp:~maria-captains/maria/10.0

2014-11-18 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: Anyway, it seems that we agree not to push it into 10.0. What do you think - shall it be pushed into 10.1 or not at all? Well, as I said, I wouldn't have prioritised fixing that bug. However, now that you've made the patch, I think it's ok to push it

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5

2014-11-18 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: revno: 4346 revision-id: s...@mariadb.org-20141118080742-sdf93kgsgxulenvb parent: kniel...@knielsen-hq.org-20141112101013-t63ayylsk08ncgs3 committer: Sergey Vojtovich s...@mariadb.org branch nick: 5.5 timestamp: Tue 2014-11-18 12:07:42 +0400

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5

2014-11-17 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: Look at the cset comment: every mutex_exit() has to issue full memory barrier unconditionally! Oh, you're right. I mixed up the code paths between mutex_exit() and the other side (in mutex_spin_wait()). me Strange... Monty should have fixed this.

Re: [Maria-developers] [Commits] Rev 4484: MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails in lp:~maria-captains/maria/10.0

2014-11-17 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: Kristian Nielsen, could you please review this patch? I guess my main question here is, did you think about why you want to) fix this 1) in a post-GA release, 2) at all? One problem with adding new statements as unsafe for statement-based binlogging

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera

2014-11-14 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: # Case 7: Stop slave into the middle of a transaction being filtered and # start it back with filtering disabled. --echo # On master connection master; SET @@session.gtid_domain_id=1; BEGIN; INSERT INTO t2 VALUES(3);

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5

2014-11-14 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: revno: 4346 revision-id: s...@mariadb.org-20141112114907-thnro6e3kg2ofdsw committer: Sergey Vojtovich s...@mariadb.org timestamp: Wed 2014-11-12 15:49:07 +0400 message: MDEV-7026 - Occasional hang during startup on Power8 === modified file

Re: [Maria-developers] Syntax for parallel replication

2014-11-14 Thread Kristian Nielsen
Pavel Ivanov piva...@google.com writes: I'd say following_master will be very confusing, because degree of parallelization on slave won't match degree of parallelization on master in this case. Just because some commits are in different commit Right, that was my original concern as well. With

Re: [Maria-developers] Syntax for parallel replication

2014-11-13 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: I didn't know that it was possible to use --connection_name.XXX to configure things, which is why I thought I needed to use CHANGE MASTER instead. Well, possible can have different meanings. I meant that from the user point view

[Maria-developers] Memory barrier problem in InnoDB/XtraDB mutex implementation causing hangs

2014-11-06 Thread Kristian Nielsen
I have been debugging some test failures seen in Buildbot: https://mariadb.atlassian.net/browse/MDEV-7026 After some debugging, this is what I think is going on. I will refer to MariaDB 5.5 sources, though the problem is also in 10.0. InnoDB/XtraDB mutexes are taken in mutex_enter_func()

Re: [Maria-developers] Memory barrier problem in InnoDB/XtraDB mutex implementation causing hangs

2014-11-06 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: # First spin wait, hoping to get the mutex without yielding. os_compare_and_swap_ulint(mutex-waiters, 0, 1); Where did you see os_compare_and_swap_ulint(mutex-waiters, 0, 1)? FWICS waiters of InnoDB mutex are non-atomic loads/stores.

Re: [Maria-developers] Memory barrier problem in InnoDB/XtraDB mutex implementation causing hangs

2014-11-06 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: Never seen this before. InnoDB of 5.5 and InnoDB/XtraDB of 10.0 both have: UNIV_INTERN void mutex_set_waiters( /*==*/ ib_mutex_t* mutex, /*! in: mutex */ ulint n) /*! in: value to set */ {

[Maria-developers] Valgrind failure in Buildbot

2014-10-30 Thread Kristian Nielsen
Hi Sergey, We briefly discussed the new valgrind error in buildbot after your push on IRC: https://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/6282/steps/test/logs/stdio ==13161== Conditional jump or move depends on uninitialised value(s) ==13161==at 0x4C280AA: memcpy

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera

2014-10-27 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: [NC] You are right, setting both DO_ and IGNORE_ at the same time would not make any sense. But I actually followed the pattern of the existing replication, where both _do_ and _ignore_ filters are allowed to hold values at the same time and _do_

[Maria-developers] Problem with KVM version in Buildbot?

2014-10-20 Thread Kristian Nielsen
Hi Daniel, We have a problem with a build in Buildbot: https://internal.askmonty.org/buildbot/builders/kvm-rpm-centos7_0-x86_64/builds/127/steps/compile/logs/stdio + qemu-img create -b /kvm/vms/vm-centos7_0-x86_64-build.qcow2 -f qcow2 /dev/shm/vm-tmp-2302.qcow2 qemu-img: '' uses a qcow2

Re: [Maria-developers] Problem with KVM version in Buildbot?

2014-10-20 Thread Kristian Nielsen
Otto Kekäläinen writes: There is no need for this image to be so big. The OS is maybe that 3.1 GB and the MariaDB build output is about 3 GB, so something like 9 GB disk size should be well enough by a fair margin. Right. On the other hand, there is a lot of work involved in re-doing the VM

Re: [Maria-developers] [Commits] Rev 3900: Fix for test failures on 64-bit platform. in lp:~maria-captains/maria/10.0-galera-6593

2014-10-17 Thread Kristian Nielsen
Nirbhay Choubey nirb...@mariadb.com writes: message: Fix for test failures on 64-bit platform. === modified file 'sql/rpl_mi.cc' --- a/sql/rpl_mi.cc 2014-10-10 23:06:40 + +++ b/sql/rpl_mi.cc 2014-10-17 02:08:55 + @@ -1466,7 +1466,7 @@ { for (int i= DO_DOMAIN_IDS; i =

Re: [Maria-developers] Syntax for parallel replication

2014-10-16 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: In the view of 4) above, did you consider using system variables? Like --slave-parallel-mode={domain|groupcommit|transactional|waiting} and, the usual, --connection_name.slave-parallel-mode=... for multi-source. This variable can be of SET or

[Maria-developers] Syntax for parallel replication

2014-10-06 Thread Kristian Nielsen
Hi Serg, Can you help me with suggestions/comments for the following proposal for how to do configuration for MDEV-6676, speculative parallel replication? I am not too happy about how things are in 10.0. There, there is a single option --slave-parallel-threads=N. If N0, then parallel replication

[Maria-developers] MDEV-6429 (storage engine API for parallel replication) possibly a requirement for MDEV-6676 (speculative parallel replication)

2014-10-06 Thread Kristian Nielsen
Hi Serg, Earlier this year we discussed MDEV-6429, which is to implement a proper storage engine API for a feature used in parallel replication. This is where InnoDB reports to the replication layer whenever transaction T1 needs to wait for T2; parallel replication uses this to detect and resolve

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-22 Thread Kristian Nielsen
I have done some more patches on this, and basic functionality should be there now in this tree: lp:~maria-captains/maria/10.0-knielsen Let me know if you want to experiment with it, and I can provide more details... - Kristian. ___ Mailing

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-16 Thread Kristian Nielsen
Robert Hodges robert.hod...@continuent.com writes: In that case it gets complex for outsiders to figure out how to restart if there's a failure. Let's say my transaction fails after T1 commits but before T1 commits. Then on restart I have to regenerate T2 and rerun it. That could be hard if

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-16 Thread Kristian Nielsen
Robert Hodges robert.hod...@continuent.com writes: Right. I thought about that problem a lot in the Tungsten parallel apply design and ended up with an approach that allows workers to diverge by several minutes or longer. This enables Tungsten to maintain good throughput even in the face of

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera

2014-09-15 Thread Kristian Nielsen
Nirbhay Choubey nirb...@skysql.com writes: revno: 3879 revision-id: nirb...@skysql.com-20140912181622-s33e7r0vtgpju95q parent: nirb...@skysql.com-20140814224304-rtkeal7p9lk06li1 committer: Nirbhay Choubey nirb...@skysql.com branch

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera

2014-09-15 Thread Kristian Nielsen
Kristian Nielsen kniel...@knielsen-hq.org writes: 4. Also see detailed comments for some possible problems with the implementation. The most serious is probably to ensure that events are not skipped after the end of the group, we need a couple of tests for this, see Hm, actually I thought

Re: [Maria-developers] master.info and relay-log.info location

2014-09-15 Thread Kristian Nielsen
Tim Callaghan tmcallag...@gmail.com writes: Has any work been done in MariaDB 10.0 or 10.1 to move them from their file-based location to within XtraDB (and thus potentially TokuDB)? The In MariaDB 10.0, it is recommended to use MariaDB Global Transaction ID (GTID). With GTID, the position is

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-11 Thread Kristian Nielsen
Robert Hodges robert.hod...@continuent.com writes: Thanks Kristian. That's a nice optimization that is difficult to do outside the DBMS engine. Indeed. We might be able to expose it to you, though. Like SET SESSION wait_for_other_commit=processid, or something. - Kristian.

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-11 Thread Kristian Nielsen
Robert Hodges robert.hod...@continuent.com writes: What would be cool is something like a group transaction that other threads can join so that the commit becomes atomic. Most of the parallel load use cases I can think of don't require you to coordinate things like isolation because they are

[Maria-developers] Total breakdown of development procedures :-(

2014-09-10 Thread Kristian Nielsen
Today, I learned that a large change (the galera merge) has been pushed into 10.1 on August 27. At that point, there was not a _single_ builder in buildbot that was green! As a result, we now have an extremely serious regression in the server caused by this push, easily seen in the testsuite.

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-10 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: Hi Jonas, I actually was planning to discuss this with you, as it is based on some of the ideas you mentioned earlier on parallel replication... Intermediate commit. Patch is far from complete, but this small patch was nevertheless sufficient to be

Re: [Maria-developers] [Commits] Rev 4376: MDEV-6676: Speculative parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-09-10 Thread Kristian Nielsen
Robert Hodges robert.hod...@continuent.com writes: There is one thing I have never understood about your parallel apply algorithm. How do you handle the case where the server crashes when some threads have committed but others have not? It seems as if you could have a problem with recovery.

Re: [Maria-developers] a question about group commit

2014-08-07 Thread Kristian Nielsen
nanyi607rao nanyi607...@gmail.com writes: yeah, I got it. InnoDB/xtradb group commit indeed reduce fsync() called times in prepare step, but it could do more than one fsync() for a group of transactions in binlog group commit to be durable in prepare step. Ah, yes, now I see, I had forgotten

Re: [Maria-developers] a question about group commit

2014-08-04 Thread Kristian Nielsen
nanyi607rao nanyi607...@gmail.com writes: as we know there are 3 steps in XA transaction committing 1, prepare step 2, write binary log 3, commit step in engines all these steps need a fsync(). Group commit strategy can make a group of transactions durable with one fsync() at step 2 and

Re: [Maria-developers] apt-get upgrade on buildbot

2014-07-11 Thread Kristian Nielsen
Otto Kekäläinen o...@seravo.fi writes: Could you do a small favour and run 'apt-get upgrade' and then trigger manually a rebuild in the virtual machine Debian unstable the buildbot uses at http://buildbot.askmonty.org/buildbot/builders/debpkg-sid Done. As usual, there might be some delay

Re: [Maria-developers] [JIRA] (MDEV-6336) NEED REVIEW: mysqldump --master-data does not work with GTID setups

2014-07-09 Thread Kristian Nielsen
Michael Widenius (JIRA) j...@mariadb.atlassian.net writes: Here is the review: revno: 4245 revision-id: knielsen at knielsen-hq.org-20140616093803-gub60pda6t77ch3k committer: knielsen at knielsen-hq.org timestamp: Mon 2014-06-16

Re: [Maria-developers] [JIRA] (MDEV-4937) NEED REVIEW: sql_slave_skip_counter does not work with GTID

2014-07-09 Thread Kristian Nielsen
: knielsen at knielsen-hq.org-20140620104939-xdvn985cgmwy3odd committer: Kristian Nielsen knielsen at knielsen-hq.org timestamp: Fri 2014-06-20 12:49:39 +0200 message: MDEV-4937: sql_slave_skip_counter does not work with GTID === modified file 'sql/slave.cc' --- a/sql/slave.cc2014-06-19 12

Re: [Maria-developers] apt-get upgrade on buildbot

2014-07-09 Thread Kristian Nielsen
Otto Kekäläinen o...@seravo.fi writes: Hello Kristian! Could you do a small favour and run 'apt-get upgrade' and then trigger manually a rebuild in the virtual machine Debian unstable the buildbot uses at http://buildbot.askmonty.org/buildbot/builders/debpkg-sid Thanks! Hm, I tried this,

Re: [Maria-developers] comments on parallel applying

2014-07-04 Thread Kristian Nielsen
Jonas Oreland jon...@google.com writes: quick thoughts on implementation for row-based replication this seems quite easy. for statement-based replication i image that you would have to add hooks into the real code after parsing has been performed, but before the actual execution is started

Re: [Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262

2014-07-04 Thread Kristian Nielsen
Jan Lindström jan.lindst...@skysql.com writes: 1) In innodb deadlock detection it is not certain that there is actually a deadlock, deadlock detection is done when a lock request is issued. At that point yes, we can see from lock manager that there is a deadlock, however if waiting

Re: [Maria-developers] comments to the plugin.h

2014-07-04 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: As you suggested on irc, it would make sense to make a smaller innodb/xtradb only fix in 10.0 and a more engine-friendly, with the new api, in 10.1 Hmm, okay... When you put it this way, it does sound simpler. Allright, let's keep

Re: [Maria-developers] comments to the plugin.h

2014-07-03 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: I'm kind of lost here. I really don't like requiring engines to do more obligatory things, but I don't see a good way around it either. Agree, that's how I felt while trying to think of _any_ way to solve these problems. I ended up with current

Re: [Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262

2014-07-03 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: Here it is, below. Thanks for review! I'll give comments inline where applicable, other points I will just implement as you suggested, once the overall API is decided. * MySQL has a variable binlog_order_commits. we can introduce it too - it

Re: [Maria-developers] comments to the plugin.h

2014-07-01 Thread Kristian Nielsen
Sergei Golubchik vuv...@gmail.com writes: This is what I've written in the patch: I've tried to answer all points in detail, also to help myself better understand all the issues: + The storage engine can report such conflicting locks using this call. This + will allow parallel replication

Re: [Maria-developers] [Commits] Rev 4245: MDEV-6336: mysqldump --master-data does not work with GTID setups in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-06-25 Thread Kristian Nielsen
Sergey Vojtovich s...@mariadb.org writes: MDEV-6344 looks fine. But I don't feel confident reviewing MDEV-6336. Ok, thanks. Hopefully we can find someone else to review it. - Kristian. ___ Mailing list: https://launchpad.net/~maria-developers Post

Re: [Maria-developers] Doubt in bin logging for CREATE VIEW

2014-06-24 Thread Kristian Nielsen
sriram patil spsrirampa...@gmail.com writes: I have not yet figured out how to write automated test cases for replication, but will do that soon. Here is one existing test case that should be a good simple example as a starting point for this, when you get to it:

[Maria-developers] Review of MDEV-5730: enhance security using special compilation options

2014-06-24 Thread Kristian Nielsen
commit feab48657528b9bb40fb035d51bee28d93710c1e Author: Sergei Golubchik s...@mariadb.org Date: Mon Jun 16 21:39:09 2014 +0200 MDEV-5730 enhance security using special compilation options -Wl,-z,relro,-z,now -pie -fstack-protector --param=ssp-buffer-size=4

Re: [Maria-developers] Doubt in bin logging for CREATE VIEW

2014-06-19 Thread Kristian Nielsen
sriram patil spsrirampa...@gmail.com writes: In case of CREATE VIEW (method: mysql_create_view file: sql/sql_view.cc), while the bin logging is skipped for most of the errors by jumping to err label, it is not skipped in following code snippet (line number: 616) res= mysql_register_view(thd,

[Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262

2014-06-10 Thread Kristian Nielsen
Hi Serg, We discussed on IRC a review of the $subj bugs related to deadlocks in parallel replication and handling of automatic retry of transactions in case of deadlock. The patches are in my tree: lp:~maria-captains/maria/10.0-knielsen The complete diff can be obtained eg. with: bzr

Re: [Maria-developers] Question about GTIDs in MariaBD

2014-05-12 Thread Kristian Nielsen
Peter Laursen peter_laur...@webyog.com writes: MyISAM is/used to be completely 'transaction-agnosstic'. But GTIDs seem to confict with non-transactional storage engines (as far as I can understand). MySQL/Oracle have solved (!) this by not allowing updates to tables using transactional

Re: [Maria-developers] Question about GTIDs in MariaBD

2014-05-12 Thread Kristian Nielsen
AL13N al...@rmail.be writes: Peter Laursen peter_laur...@webyog.com writes: So yes, there are others that consider Oracle's design of GTID 'a dirty hack', but MariaDB 10.0 GTID is not affected by that design. What about mariadb being a slave of a Oracle MySQL GTID? The restrictions are on

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-04-30 Thread Kristian Nielsen
Vangelis Katsikaros vkatsika...@yahoo.gr writes: What do you mean with the phrase [code|server] falls over? I am refering to a common phenomenon in high-concurrency benchmarks. See for example the graph in this email: https://lists.launchpad.net/maria-developers/msg06799.html

Re: [Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP

2014-04-30 Thread Kristian Nielsen
Laurynas Biveinis laurynas.bivei...@gmail.com writes: Did you test InnoDB or XtraDB? It should be XtraDB, which is the default in MariaDB 10 now. This is the version info from univ.i: #define INNODB_VERSION_MAJOR5 #define INNODB_VERSION_MINOR6 #define INNODB_VERSION_BUGFIX

Re: [Maria-developers] 10.1 planning

2014-04-29 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: There is also https://mariadb.atlassian.net/browse/MDEV-4487 Fine. But the question was whether you think it should be a bug for 10.0.11 or it's big enough to consider it a task to do in 10.1? 10.0.11? You mean whatever 10.0 version will be due

[Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP

2014-04-29 Thread Kristian Nielsen
At the Barcelona meeting in January, I promised to take a look at the high-concurrency sysbench OLTP benchmarks, and now I finally had the time do do this. There was a lot of work on LOCK_open by Svoj and Serg. If I have understood correctly, the basic problem was that at high concurrency (like,

Re: [Maria-developers] 10.1 planning

2014-04-28 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: See below. What do you think about it? On Apr 23, Elena Stepanova wrote: There is also https://mariadb.atlassian.net/browse/MDEV-4487 (Replication from 5.6 with enabled GTID) to be considered. It's currently Minor and has 10.0.x as a target

Re: [Maria-developers] [GSoC] Accepted student ready to work : )

2014-04-28 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: note, that two *different* revisions got the same revno! And the changes from the first revision are completely and totally lost, there is no way to retrieve from from anywhere. Indeed. But note that in main trees (5.1, 5.2, 5.3, 5.5, and 10.0), this

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-25 Thread Kristian Nielsen
nanyi607rao nanyi607...@gmail.com writes: It seems most reasonable to put cached_charset into the THD, and your patch is very smart and clean. You are very kind :-) Thanks for checking this second patch. I've pushed it to 10.0. - Kristian. ___

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-24 Thread Kristian Nielsen
nanyi607rao nanyi607...@gmail.com writes: I have filed a bug for this that I will fix: https://mariadb.atlassian.net/browse/MDEV-6156 I'm afraid there is still some cases would lead mistake if move cached_charset into rpl_group_info struct. Yes, you are absolutely right, the patch I

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-24 Thread Kristian Nielsen
nanyi607rao, Kristian Nielsen kniel...@knielsen-hq.org writes: Yes, you are absolutely right, the patch I pushed for this is completely wrong, just as you explained :-( Sorry about this, I will try to come up with a better fix. What do you think about the below patch? It puts

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-23 Thread Kristian Nielsen
Hi nanyi607rao, sorry for the delayed response, in part due to the Easter holidays. nanyi607rao nanyi607...@gmail.com writes: If character_set in different Query_log_events changed, worker threads may apply them with wrong character_set. the codes leading this problem is in

Re: [Maria-developers] bb01.mariadb.net

2014-04-23 Thread Kristian Nielsen
Daniel Bartholomew db...@mariadb.com writes: Serg and Knielsen: I've got the box mostly configured I think, but could be (very) wrong, so I wanted to give you a chance to make sure everything looks good on it before I go ahead and add it to the It looks like it already mostly works. I

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-23 Thread Kristian Nielsen
Kristian Nielsen kniel...@knielsen-hq.org writes: I have filed a bug for this that I will fix: https://mariadb.atlassian.net/browse/MDEV-6156 I've now pushed a fix for this bug to 10.0. Thanks again for your help! - Kristian. ___ Mailing

Re: [Maria-developers] [GSoC] Accepted student ready to work : )

2014-04-22 Thread Kristian Nielsen
Sergei Golubchik s...@mariadb.org writes: as for (1) - I think the tables where test failures are recorded is our own addition, not part of the buildbot. if that's true, then buildbot documentation won't help much. It is our own addition, but it has been included in upstream Buildbot,

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-04-22 Thread Kristian Nielsen
Kristian Nielsen kniel...@knielsen-hq.org writes: Axel Schwenke a...@askmonty.org writes: Benchmark 1 is good old sysbench OLTP. I tested 10.0.7 vs. 10.0.7-pgo. With low concurrency there is about 10% win by PGO; however this is completely reversed at higher concurrency by mutex contention

Re: [Maria-developers] MariaDB 10.x GTID

2014-04-11 Thread Kristian Nielsen
Jan Lindström jplin...@mariadb.org writes: some background, lets assume we have two galera clusters that are geographically distributed (lets say for example that cluster A is on Europe and cluster B on USA). Inside a cluster all nodes are set up so that server_id and domain_id are the same

Re: [Maria-developers] Buildbot machines

2014-04-09 Thread Kristian Nielsen
Daniel Bartholomew db...@mariadb.com writes: Kristian (or others): Any preferences on Linux distro for the box? Should I go with OpenSuse, which is what is on the terriers, or does it matter? Heh, I'd prefer anything _but_ Suse, I think we've had only grief from that. But in the end I don't

Re: [Maria-developers] Buildbot machines

2014-04-07 Thread Kristian Nielsen
Rasmus Johansson ras...@mariadb.com writes: To sum it up, there seems to be two aspects to it; 1) more machines, i.e. slaves could/should be added immediately and 2) some development work Right. We have a setup where we can more or less plug in an empty machine with Linux, Buildbot and KVM,

<    1   2   3   4   5   6   7   8   >