Re: [Wikitech-l] [Labs-l] Maria DB
For most projects, I recommend using the official packages available via
the MariaDB projects own apt repo.
The official packages are based on the Debian mysql packaging where
installing the server package also installs a default database created
around generic config defaults, a debian mysql maintenance user with a
randomly generated password, and scripts (including init) that assume
privileged access via that user. That is, installing the packages provides
you with a fresh running working database with generic defaults suitable
for a small server, and certain admin tasks automated. I think that's what
the average labs and general users wants and expects.
The packages I've built for production use at wmf strips out all of the
debianisms, the debian project script rewrites, the pre/post install
actions. They also leave debug symbols in the binaries and have compiler
flag tweaks, but do not at this stage contain any source
patches. Installing the server package doesn't create a default db, or
provide an environment where you can even start the server on a fresh sever
install without further work. Probably not a good choice for most labs
users.
On Wednesday, February 13, 2013, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and
the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:_e({}, 'cvml', 'afeld...@wikimedia.org');
wrote:
The production migration to MariaDB was paused for a time by the EQIAD
datacenter migration and issues involving other projects that took up my
time, but the trial production roll-out will resume this month. All signs
still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the
course
of more than a month before the first production deployment. Major
version
migrations with mysql and derivatives are not to be taken lightly in
production environments. At a minimum, one must be concerned about query
optimizer changes making one particular query type significantly slower.
In the case of the switch to 5.5, there are several default behavior
changes over 5.1 that can break applications or change results. Hence,
some serious work over a plodding time frame before that first production
slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query
generated by what seems to be a very rare edge case from that AFTv4
extension that violated stricter enforcement of unsigned integer types in
5.5, breaking replication and requiring one off rewriting and execution of
the query locally to ensure data consistency before skipping over it. I
opened a bug, Mathias fixed the extension, and I haven't seen any other
compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my
testing that has gone into enwiki cannot be assumed to fully cover
everything else. Because of that, and because I want to continue
proceeding with caution for all of our projects, this will continue to be
a
slow and methodical process at this stage. Bugs in extensions that aren't
used by English Wikipedia may be found and require fixing along the way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best,
Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena
benap...@gmail.comjavascript:_e({}, 'cvml', 'benap...@gmail.com');
wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad
innocentkil...@gmail.comjavascript:_e({}, 'cvml',
'innocentkil...@gmail.com');
wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff
bawolff...@gmail.comjavascript:_e({}, 'cvml',
'bawolff%2...@gmail.com');
wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for mariadb 5.5 in production for english
wikipedia
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Labs-l mailing list
lab...@lists.wikimedia.org javascript:_e({}, 'cvml',
'lab...@lists.wikimedia.org');
https://lists.wikimedia.org/mailman/listinfo/labs-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
Re: [Wikitech-l] [Labs-l] Maria DB
Er, no it shouldn't. Initial execution might take microseconds longer due
to larger binary sizes and the elf loader having to skip over the symbols
but that's about it.
On Thursday, February 14, 2013, Petr Bena wrote:
Keeping debug symbols in binaries will result in poor performance, or it
should
On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:_e({}, 'cvml', 'afeld...@wikimedia.org');
wrote:
For most projects, I recommend using the official packages available via
the MariaDB projects own apt repo.
The official packages are based on the Debian mysql packaging where
installing the server package also installs a default database created
around generic config defaults, a debian mysql maintenance user with a
randomly generated password, and scripts (including init) that assume
privileged access via that user. That is, installing the packages provides
you with a fresh running working database with generic defaults suitable
for a small server, and certain admin tasks automated. I think that's what
the average labs and general users wants and expects.
The packages I've built for production use at wmf strips out all of the
debianisms, the debian project script rewrites, the pre/post install
actions. They also leave debug symbols in the binaries and have compiler
flag tweaks, but do not at this stage contain any source
patches. Installing the server package doesn't create a default db, or
provide an environment where you can even start the server on a fresh
sever
install without further work. Probably not a good choice for most labs
users.
On Wednesday, February 13, 2013, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and
the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:_e({}, 'cvml',
'afeld...@wikimedia.org');javascript:_e({},
'cvml', 'afeld...@wikimedia.org javascript:_e({}, 'cvml',
'afeld...@wikimedia.org');');
wrote:
The production migration to MariaDB was paused for a time by the EQIAD
datacenter migration and issues involving other projects that took up
my
time, but the trial production roll-out will resume this month. All
signs
still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the
course
of more than a month before the first production deployment. Major
version
migrations with mysql and derivatives are not to be taken lightly in
production environments. At a minimum, one must be concerned about
query
optimizer changes making one particular query type significantly
slower.
In the case of the switch to 5.5, there are several default behavior
changes over 5.1 that can break applications or change results. Hence,
some serious work over a plodding time frame before that first
production
slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query
generated by what seems to be a very rare edge case from that AFTv4
extension that violated stricter enforcement of unsigned integer types
in
5.5, breaking replication and requiring one off rewriting and
execution of
the query locally to ensure data consistency before skipping over it.
I
opened a bug, Mathias fixed the extension, and I haven't seen any other
compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my
testing that has gone into enwiki cannot be assumed to fully cover
everything else. Because of that, and because I want to continue
proceeding with caution for all of our projects, this will continue to
be
a
slow and methodical process at this stage. Bugs in extensions that
aren't
used by English Wikipedia may be found and require fixing along the
way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best,
Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena
benap...@gmail.comjavascript:_e({}, 'cvml',
'benap...@gmail.com');javascript:_e({},
'cvml', 'benap...@gmail.com javascript:_e({}, 'cvml',
'benap...@gmail.com');');
wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we
going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad
innocentkil...@gmail.comjavascript:_e({}, 'cvml',
'innocentkil...@gmail.com');javascript:_e({},
'cvml', 'innocentkil...@gmail.com javascript:_e({}, 'cvml',
'innocentkil...@gmail.com');');
wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff
bawolff...@gmail.comjavascript:_e({}, 'cvml',
'bawolff%2...@gmail.com');javascript:_e({},
'cvml', 'bawolff%2...@gmail.com javascript:_e({}, 'cvml',
'bawolff%252...@gmail.com');');
wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Re: [Wikitech-l] [Labs-l] Maria DB
On Feb 14, 2013, at 5:02 PM, Petr Bena benap...@gmail.com wrote: Keeping debug symbols in binaries will result in poor performance, or it should That's bollocks. It results in a larger binary _on disk_. The symbol table isn't even loaded into memory and doesn't affect performance. Debug information is *highly useful* in a production setup, and we try to run all our core applications with it so we have a chance to debug issues when they occur. I think the only reason distributions omit debug information is to save disk space. -- Mark Bergsma m...@wikimedia.org Lead Operations Architect Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
On Thu, Feb 14, 2013 at 05:14:31PM +0100, Mark Bergsma wrote: Debug information is *highly useful* in a production setup, and we try to run all our core applications with it so we have a chance to debug issues when they occur. I think the only reason distributions omit debug information is to save disk space. A lot of Debian packages ship -dbg alongside the main package, that contains the stripped-out debug symbols in /usr/lib/debug (gdb loads those automatically, either based on the filename or build-id). The toolchain handles this more or less automatically, but it still needs maintainer action to define this separate package and upload it with every package upload. Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a ddebs (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors. This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :) For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg package by default. That's a shame, we can ask for that. As for the rest of Asher's changes, I'd love to find a way to make stock packages work in our production setup, but I'm not sure if the maintainer would welcome the extra complexity of conditionally switching behaviors. We can try if you're willing to, Asher :) Regards, Faidon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
I would much rather abandon using debs than use what the debian project has done to mysql packaging in any production environment. If the discussion has come down to this, I did WMF a disservice by drifting away from Domas' optimized make ; make install ; rsync unstripped binaries to prod workflow. In general, I find environments that don't individually package according to distro standards every part of their core application stack that gets built in-house to be more productive, and more responsive to the needs of developers and ultimately the application. When an ops team claims that building a recent version of libmemcached for a stable OS is almost impossibly hard and will take weeks because it requires backporting a debian maintainers packaging of it for an experimental distro with that distros unrelated library version dependencies and reliance on a newer incompatible dpkg tool chain, there's probably something wrong with that workflow. I like to rely on Linux distros for the lowest common denominator layer of the stack and related security updates. The approach that goes into building and maintaining such a beast are rather different than the concerns that go into operating a continually developed and deployed distributed application used by half a billion people. I don't see a win in trying to force the two together. On Thursday, February 14, 2013, Faidon Liambotis wrote: For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg package by default. That's a shame, we can ask for that. As for the rest of Asher's changes, I'd love to find a way to make stock packages work in our production setup, but I'm not sure if the maintainer would welcome the extra complexity of conditionally switching behaviors. We can try if you're willing to, Asher :) Regards, Faidon ___ Labs-l mailing list lab...@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/labs-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
On 14/02/13 18:26, Faidon Liambotis wrote: Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a ddebs (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors. This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :) Good question. There are a few bugs and blueprints about it, and they show as *implemented* https://blueprints.launchpad.net/ubuntu/+spec/apt-get-debug-symbols https://bugs.launchpad.net/ubuntu/+bug/14484 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2006-September/000195.html What happened, then? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production. I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch. Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki. That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way. As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. Best, Asher On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena benap...@gmail.com wrote: Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production? On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkil...@gmail.com wrote: On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff...@gmail.com wrote: Umm there was a thread several months ago about how it is used on several of the slave dbs, if I recall. Indeed, you're looking for mariadb 5.5 in production for english wikipedia http://www.gossamer-threads.com/lists/wiki/wikitech/319925 -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Labs-l mailing list lab...@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
thanks for updates. Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu? On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman afeld...@wikimedia.orgwrote: The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production. I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch. Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki. That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way. As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. Best, Asher On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena benap...@gmail.com wrote: Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production? On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkil...@gmail.com wrote: On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff...@gmail.com wrote: Umm there was a thread several months ago about how it is used on several of the slave dbs, if I recall. Indeed, you're looking for mariadb 5.5 in production for english wikipedia http://www.gossamer-threads.com/lists/wiki/wikitech/319925 -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Labs-l mailing list lab...@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
