Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Sergei Golubchik 
Hi, Reindl!

On Aug 24, Reindl Harald wrote:
> 
> Am 24.08.20 um 14:23 schrieb Sergei Golubchik:
> > Sorry, I couldn't understand, was 10.3 returning 3..2..1 and 10.4
> > started to return 1..2..3? Or was it the other way around?
> 
> that at a bottom is a unified iff between expected output and what
> happens when trying 10.4.14
> 
> 10.2 and 10.3 returning 1-2-3 for years
> 10.4.14 in the identical environemnt returns 3-2-1
> 
> * they are created 1-2-3
> * timestamps identical because created in same second
> * 'desc' ordering by timestamp gave 1-2-3
> 
> my issue is not that the result as such could be called wrong but that
> it changes after years, the autotests which is part of a bug testsuite
> has the expected HTML output base64 decoded, fires a diff against the
> now created html-output and alerts if there is a byte changed

Yes, I understand that.
But, as I just wrote:

> > I tried your table structure and your test data on both 10.3 and 10.5
> > and in both cases I've got 3..2..1

What happens in my tests - decides to use the comment_key_1 index.
It reads the index backwards, so it naturally gets 3-2-1.

If I delete this index from the CREATE TABLE statement, I get 1-2-3.

For me this happens identically both in 10.3 and 10.4
May be some changes to server defaults, like optimizer_switch or
use_stat_tables caused the optimizer to prefer a different plan in your
case.

I suspect for a large table with lots of rows the plan will be the same
in 10.3 and 10.4, but for a small table when all plans have very similar
(and small) costs you can, probably, expect this kind of instability.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Reindl Harald 



Am 24.08.20 um 14:23 schrieb Sergei Golubchik:
> Sorry, I couldn't understand, was 10.3 returning 3..2..1 and 10.4
> started to return 1..2..3? Or was it the other way around?

that at a bottom is a unified iff between expected output and what
happens when trying 10.4.14

10.2 and 10.3 returning 1-2-3 for years
10.4.14 in the identical environemnt returns 3-2-1

* they are created 1-2-3
* timestamps identical because created in same second
* 'desc' ordering by timestamp gave 1-2-3

my issue is not that the result as such could be called wrong but that
it changes after years, the autotests which is part of a bug testsuite
has the expected HTML output base64 decoded, fires a diff against the
now created html-output and alerts if there is a byte changed

> I tried your table structure and your test data on both 10.3 and 10.5
> and in both cases I've got 3..2..1
> 
> On Aug 21, Reindl Harald wrote:
>>
>> in all previous versions this test was stable, so as the timestamps are
>> identical and ordering is 'desc' the result was ordered by the creation
>> time of the the records
> ... 
>> diff of the expected application output:
>>
>> -Autotest> name="commentpart">> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
>> 1Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 2Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 3Test Test Test Test Test Test
>> Test Test Test
>> Test
>>
>> +Autotest> name="commentpart">> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
>> 3Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 2Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 1Test Test Test Test Test Test
>> Test Test Test
>> Test

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Sergei Golubchik 
Hi, Reindl!

Sorry, I couldn't understand, was 10.3 returning 3..2..1 and 10.4
started to return 1..2..3? Or was it the other way around?

I tried your table structure and your test data on both 10.3 and 10.5
and in both cases I've got 3..2..1

On Aug 21, Reindl Harald wrote:
> 
> in all previous versions this test was stable, so as the timestamps are
> identical and ordering is 'desc' the result was ordered by the creation
> time of the the records
... 
> diff of the expected application output:
> 
> -Autotest name="commentpart"> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
> 1Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 2Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 3Test Test Test Test Test Test
> Test Test Test
> Test
> 
> +Autotest name="commentpart"> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
> 3Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 2Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 1Test Test Test Test Test Test
> Test Test Test
> Test
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Reindl Harald 



Am 24.08.20 um 12:23 schrieb Reindl Harald:
> Am 24.08.20 um 09:46 schrieb Sergei Golubchik:
>> Hi, Reindl!
>>
>> 1. Is it 10.3 to 10.4 difference? There were some optimizations that
>> could have such an effect.
>> https://mariadb.com/kb/en/changes-improvements-in-mariadb-104/#optimizer
>> Can you share the table structure?
> 
> it is a 10.3 to 10.4 interface, hence the 10.4 in the subject
> 
>> 2. Technically, MyISAM only returns rows in the order of insertion if
>> you insert into an empty table. This was always the case, see this
>> simple test:
>>
>>   create table t1 (a int) engine=myisam;
>>   insert t1 values (1),(2),(3),(4),(5); select * from t1;
>>   delete from t1 where a<10;select * from t1;
>>   insert t1 values (1),(2),(3),(4),(5); select * from t1;
>>   drop table t1;
>>
>> I don't think this is your case though.
> 
> given that tjis is an autotest that table is *always* empty at the begin
> of the autotest suite

forgot the table structure

cms1_comments | CREATE TABLE `cms1_comments` (
  `kid` mediumint(7) unsigned NOT NULL AUTO_INCREMENT,
  `ksid` mediumint(7) unsigned NOT NULL DEFAULT 0,
  `ks2id` mediumint(7) unsigned NOT NULL DEFAULT 0,
  `ktimestamp` int(10) unsigned NOT NULL DEFAULT 0,
  `kip` varchar(50) COLLATE latin1_german1_ci NOT NULL,
  `kname` varchar(100) COLLATE latin1_german1_ci NOT NULL,
  `kherkunft` varchar(100) COLLATE latin1_german1_ci NOT NULL,
  `kemail` varchar(80) COLLATE latin1_german1_ci NOT NULL,
  `khomepage` varchar(255) COLLATE latin1_german1_ci NOT NULL,
  `kkommentar` text COLLATE latin1_german1_ci NOT NULL,
  `kaktiv` tinyint(1) unsigned NOT NULL DEFAULT 0,
  PRIMARY KEY (`kid`),
  KEY `comment_key_1` (`ksid`,`kaktiv`,`ktimestamp`),
  KEY `comment_key_2` (`ks2id`,`kaktiv`,`ktimestamp`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci
PACK_KEYS=0 DELAY_KEY_WRITE=1

and thats how the test records are generated

$data_template =
[
 'ksid'   => 0,
 'ks2id'  => 0,
 'kname'  => 'CMS-Autotest [count]',
 'kemail' => 'server-adm...@thelounge.net',
 'kkommentar' => trim(str_repeat('Test ', 10)),
 'kherkunft'  => 'Wien',
 'khomepage'  => 'www.thelounge.net',
 'kativ'  => 1,
];

for($count=1; $count<=3; $count++)
{
 $data = str_replace('[count]', $count, $data_template);
 $data['ksid'] = 1;
 $insert_id = $this->add($data);
 if(empty($insert_id))
 {
  $cl_autotests->trigger_error('check failed');
 }
}

>> On Aug 21, Reindl Harald wrote:
>>> MyISAM:
>>>
>>> in all previous versions this test was stable, so as the timestamps are
>>> identical and ordering is 'desc' the result was ordered by the creation
>>> time of the the records
>>>
>>> now it's reverse which can break all sort of expectations in subtle ways
>>>
>>> while we could discuss what is the expected output when order by
>>> 'ktimestamp' and all are identical it still worries me that something
>>> which didn't change over years and major versions now comes with the
>>> reverse ordering
>>>
>>> unless someone tells me that's the result of some real peformance
>>> optimization i would perfer the known result given that it's hard to
>>> know how much other code depends implicit on the previous behavior
>>>
>>> -
>>>
>>> select SQL_NO_CACHE SQL_CALC_FOUND_ROWS * from `cl_autotest_comments`
>>> where ksid=1 and kaktiv=1  order by ktimestamp desc limit 0,10
>>>
>>> -
>>>
>>> Array
>>> (
>>> [0] => Array
>>> (
>>> [kid] => 1
>>> [kaktiv] => 1
>>> [ksid] => 1
>>> [ks2id] => 0
>>> [ktimestamp] => 1598022502
>>> [kip] => 127.0.0.1
>>> [kname] => CMS-Autotest 1
>>> [kherkunft] => Wien
>>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>>> Test
>>> )
>>>
>>> [1] => Array
>>> (
>>> [kid] => 2
>>> [kaktiv] => 1
>>> [ksid] => 1
>>> [ks2id] => 0
>>> [ktimestamp] => 1598022502
>>> [kip] => 127.0.0.1
>>> [kname] => CMS-Autotest 2
>>> [kherkunft] => Wien
>>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>>> Test
>>> )
>>>
>>> [2] => Array
>>> (
>>> [kid] => 3
>>> [kaktiv] => 1
>>> [ksid] => 1
>>> [ks2id] => 0
>>> [ktimestamp] => 1598022502
>>> [kip] => 127.0.0.1
>>> [kname] => CMS-Autotest 3
>>> [kherkunft] => Wien
>>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>>> Test
>>> )
>>>
>>> )
>>>
>>> -
>>>
>>> diff of the expected application output:
>>>
>>> -Autotest>> name="commentpart">>> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
>>> 1Test Test Test Test Test Test
>>> Test Test Test TestCMS-Autotest
>>> 2Test Test Test

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Reindl Harald 



Am 24.08.20 um 09:46 schrieb Sergei Golubchik:
> Hi, Reindl!
> 
> 1. Is it 10.3 to 10.4 difference? There were some optimizations that
> could have such an effect.
> https://mariadb.com/kb/en/changes-improvements-in-mariadb-104/#optimizer
> Can you share the table structure?

it is a 10.3 to 10.4 interface, hence the 10.4 in the subject

> 2. Technically, MyISAM only returns rows in the order of insertion if
> you insert into an empty table. This was always the case, see this
> simple test:
> 
>   create table t1 (a int) engine=myisam;
>   insert t1 values (1),(2),(3),(4),(5); select * from t1;
>   delete from t1 where a<10;select * from t1;
>   insert t1 values (1),(2),(3),(4),(5); select * from t1;
>   drop table t1;
> 
> I don't think this is your case though.

given that tjis is an autotest that table is *always* empty at the begin
of the autotest suite

> On Aug 21, Reindl Harald wrote:
>> MyISAM:
>>
>> in all previous versions this test was stable, so as the timestamps are
>> identical and ordering is 'desc' the result was ordered by the creation
>> time of the the records
>>
>> now it's reverse which can break all sort of expectations in subtle ways
>>
>> while we could discuss what is the expected output when order by
>> 'ktimestamp' and all are identical it still worries me that something
>> which didn't change over years and major versions now comes with the
>> reverse ordering
>>
>> unless someone tells me that's the result of some real peformance
>> optimization i would perfer the known result given that it's hard to
>> know how much other code depends implicit on the previous behavior
>>
>> -
>>
>> select SQL_NO_CACHE SQL_CALC_FOUND_ROWS * from `cl_autotest_comments`
>> where ksid=1 and kaktiv=1  order by ktimestamp desc limit 0,10
>>
>> -
>>
>> Array
>> (
>> [0] => Array
>> (
>> [kid] => 1
>> [kaktiv] => 1
>> [ksid] => 1
>> [ks2id] => 0
>> [ktimestamp] => 1598022502
>> [kip] => 127.0.0.1
>> [kname] => CMS-Autotest 1
>> [kherkunft] => Wien
>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>> Test
>> )
>>
>> [1] => Array
>> (
>> [kid] => 2
>> [kaktiv] => 1
>> [ksid] => 1
>> [ks2id] => 0
>> [ktimestamp] => 1598022502
>> [kip] => 127.0.0.1
>> [kname] => CMS-Autotest 2
>> [kherkunft] => Wien
>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>> Test
>> )
>>
>> [2] => Array
>> (
>> [kid] => 3
>> [kaktiv] => 1
>> [ksid] => 1
>> [ks2id] => 0
>> [ktimestamp] => 1598022502
>> [kip] => 127.0.0.1
>> [kname] => CMS-Autotest 3
>> [kherkunft] => Wien
>> [kkommentar] => Test Test Test Test Test Test Test Test Test
>> Test
>> )
>>
>> )
>>
>> -
>>
>> diff of the expected application output:
>>
>> -Autotest> name="commentpart">> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
>> 1Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 2Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 3Test Test Test Test Test Test
>> Test Test Test
>> Test
>>
>> +Autotest> name="commentpart">> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
>> 3Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 2Test Test Test Test Test Test
>> Test Test Test TestCMS-Autotest
>> 1Test Test Test Test Test Test
>> Test Test Test
>> Test

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-24 Thread Sergei Golubchik 
Hi, Reindl!

1. Is it 10.3 to 10.4 difference? There were some optimizations that
could have such an effect.
https://mariadb.com/kb/en/changes-improvements-in-mariadb-104/#optimizer
Can you share the table structure?

2. Technically, MyISAM only returns rows in the order of insertion if
you insert into an empty table. This was always the case, see this
simple test:

  create table t1 (a int) engine=myisam;
  insert t1 values (1),(2),(3),(4),(5); select * from t1;
  delete from t1 where a<10;select * from t1;
  insert t1 values (1),(2),(3),(4),(5); select * from t1;
  drop table t1;

I don't think this is your case though.

On Aug 21, Reindl Harald wrote:
> MyISAM:
> 
> in all previous versions this test was stable, so as the timestamps are
> identical and ordering is 'desc' the result was ordered by the creation
> time of the the records
> 
> now it's reverse which can break all sort of expectations in subtle ways
> 
> while we could discuss what is the expected output when order by
> 'ktimestamp' and all are identical it still worries me that something
> which didn't change over years and major versions now comes with the
> reverse ordering
> 
> unless someone tells me that's the result of some real peformance
> optimization i would perfer the known result given that it's hard to
> know how much other code depends implicit on the previous behavior
> 
> -
> 
> select SQL_NO_CACHE SQL_CALC_FOUND_ROWS * from `cl_autotest_comments`
> where ksid=1 and kaktiv=1  order by ktimestamp desc limit 0,10
> 
> -
> 
> Array
> (
> [0] => Array
> (
> [kid] => 1
> [kaktiv] => 1
> [ksid] => 1
> [ks2id] => 0
> [ktimestamp] => 1598022502
> [kip] => 127.0.0.1
> [kname] => CMS-Autotest 1
> [kherkunft] => Wien
> [kkommentar] => Test Test Test Test Test Test Test Test Test
> Test
> )
> 
> [1] => Array
> (
> [kid] => 2
> [kaktiv] => 1
> [ksid] => 1
> [ks2id] => 0
> [ktimestamp] => 1598022502
> [kip] => 127.0.0.1
> [kname] => CMS-Autotest 2
> [kherkunft] => Wien
> [kkommentar] => Test Test Test Test Test Test Test Test Test
> Test
> )
> 
> [2] => Array
> (
> [kid] => 3
> [kaktiv] => 1
> [ksid] => 1
> [ks2id] => 0
> [ktimestamp] => 1598022502
> [kip] => 127.0.0.1
> [kname] => CMS-Autotest 3
> [kherkunft] => Wien
> [kkommentar] => Test Test Test Test Test Test Test Test Test
> Test
> )
> 
> )
> 
> -
> 
> diff of the expected application output:
> 
> -Autotest name="commentpart"> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
> 1Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 2Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 3Test Test Test Test Test Test
> Test Test Test
> Test
> 
> +Autotest name="commentpart"> href="comments_add.php?s2id=1">VerfassenCMS-Autotest
> 3Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 2Test Test Test Test Test Test
> Test Test Test TestCMS-Autotest
> 1Test Test Test Test Test Test
> Test Test Test
> Test
> 
> ___
> Mailing list: https://launchpad.net/~maria-discuss
> Post to : maria-discuss@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~maria-discuss
> More help   : https://help.launchpad.net/ListHelp
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-21 Thread Rodrigo Severo 

‐‐‐ Original Message ‐‐‐
On Friday, August 21, 2020 12:18 PM, Reindl Harald  
wrote:

> MyISAM:
>
> in all previous versions this test was stable, so as the timestamps are
> identical and ordering is 'desc' the result was ordered by the creation
> time of the the records
>
> now it's reverse which can break all sort of expectations in subtle ways
>
> while we could discuss what is the expected output when order by
> 'ktimestamp' and all are identical it still worries me that something
> which didn't change over years and major versions now comes with the
> reverse ordering
>
> unless someone tells me that's the result of some real peformance
> optimization i would perfer the known result given that it's hard to
> know how much other code depends implicit on the previous behavior


I would say that the actual ordering of the records below obtained by the 
select command shown is undefined. I can very well change for whatever subtle 
change in code.

AFAIU, if you need repeatable results, you need to write a select command with 
a deterministic result.

If this test has been presenting repeatable results for a long time, that's 
just chance.

But that's just the opinion of a MariaDB user. I'm no developer. I would like 
to know the opinion of the developers.


Regards,

Rodrigo Severo


___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-21 Thread Reindl Harald 


Am 21.08.20 um 17:44 schrieb Rodrigo Severo:
> ‐‐‐ Original Message ‐‐‐
> On Friday, August 21, 2020 12:18 PM, Reindl Harald  
> wrote:
> 
>> MyISAM:
>>
>> in all previous versions this test was stable, so as the timestamps are
>> identical and ordering is 'desc' the result was ordered by the creation
>> time of the the records
>>
>> now it's reverse which can break all sort of expectations in subtle ways
>>
>> while we could discuss what is the expected output when order by
>> 'ktimestamp' and all are identical it still worries me that something
>> which didn't change over years and major versions now comes with the
>> reverse ordering
>>
>> unless someone tells me that's the result of some real peformance
>> optimization i would perfer the known result given that it's hard to
>> know how much other code depends implicit on the previous behavior
> 
> 
> I would say that the actual ordering of the records below obtained by the 
> select command shown is undefined. I can very well change for whatever subtle 
> change in code.
> 
> AFAIU, if you need repeatable results, you need to write a select command 
> with a deterministic result.
> 
> If this test has been presenting repeatable results for a long time, that's 
> just chance.
> 
> But that's just the opinion of a MariaDB user. I'm no developer. I would like 
> to know the opinion of the developers.

without explicit ordering the behavior of MyISAM is that you get results
in der ordering the are created and so 'desc' can be and was deterministic

you can even "later table" with ordering MyISAM tables so that the
records are phyisally stored in the ordering they are typically fetched
for the workload

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp

[Maria-discuss] MariaDB 10.4: subtle change of result ordering

2020-08-21 Thread Reindl Harald 
MyISAM:

in all previous versions this test was stable, so as the timestamps are
identical and ordering is 'desc' the result was ordered by the creation
time of the the records

now it's reverse which can break all sort of expectations in subtle ways

while we could discuss what is the expected output when order by
'ktimestamp' and all are identical it still worries me that something
which didn't change over years and major versions now comes with the
reverse ordering

unless someone tells me that's the result of some real peformance
optimization i would perfer the known result given that it's hard to
know how much other code depends implicit on the previous behavior

-

select SQL_NO_CACHE SQL_CALC_FOUND_ROWS * from `cl_autotest_comments`
where ksid=1 and kaktiv=1  order by ktimestamp desc limit 0,10

-

Array
(
[0] => Array
(
[kid] => 1
[kaktiv] => 1
[ksid] => 1
[ks2id] => 0
[ktimestamp] => 1598022502
[kip] => 127.0.0.1
[kname] => CMS-Autotest 1
[kherkunft] => Wien
[kkommentar] => Test Test Test Test Test Test Test Test Test
Test
)

[1] => Array
(
[kid] => 2
[kaktiv] => 1
[ksid] => 1
[ks2id] => 0
[ktimestamp] => 1598022502
[kip] => 127.0.0.1
[kname] => CMS-Autotest 2
[kherkunft] => Wien
[kkommentar] => Test Test Test Test Test Test Test Test Test
Test
)

[2] => Array
(
[kid] => 3
[kaktiv] => 1
[ksid] => 1
[ks2id] => 0
[ktimestamp] => 1598022502
[kip] => 127.0.0.1
[kname] => CMS-Autotest 3
[kherkunft] => Wien
[kkommentar] => Test Test Test Test Test Test Test Test Test
Test
)

)

-

diff of the expected application output:

-AutotestVerfassenCMS-Autotest
1Test Test Test Test Test Test
Test Test Test TestCMS-Autotest
2Test Test Test Test Test Test
Test Test Test TestCMS-Autotest
3Test Test Test Test Test Test
Test Test Test
Test

+AutotestVerfassenCMS-Autotest
3Test Test Test Test Test Test
Test Test Test TestCMS-Autotest
2Test Test Test Test Test Test
Test Test Test TestCMS-Autotest
1Test Test Test Test Test Test
Test Test Test
Test

___
Mailing list: https://launchpad.net/~maria-discuss
Post to : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp