Re: [rt-users] RT 4.4.1 and transaction isolation level on Postgres

[Václav Ovsík]

On Wed, Jan 04, 2017 at 04:50:11AM -0800, Alex Vandiver wrote:
>... 
> MySQL suffers from the exact same problem -- but, as it happens,
> both more silently and more catastrophically.  See
> https://github.com/bestpractical/rt/commit/e36364c5

Eh. I'm glad I did transition from Mysql to Postgres years ago :).

> > I can change isolation level in postgresql.conf to 'repeatable read'
> > and things are different.
> 
> I advise against doing that.  Upon inspection, RT is not prepared to
> deal with the "could not serialize access due to concurrent update"
> errors that arise from updates to rows in multiple transactions in
> Postgres' repeatable-read isolation.

OK, thanks!

> Repeatable-read is only possible in MySQL because it has a fascinating
> definition of "repeatable":
> ...
> > Should I change the default isolation level on Postgres for RT to
> > 'repeatable read'?
> 
> No.  You should try the 4.4/previewscrips-race branch, which I've just
> pushed:
> 
> https://github.com/bestpractical/rt/compare/4.4-trunk...4.4/previewscrips-race
> 
> The gory details are contained in the commits therein.

You have my respect! Wonderful job. I tried your changes on my test
instance first. The problem is solved I think.
I installed changes to production instance too.
Thanks for your time and effort!
-- 
Zito

Re: [rt-users] RT 4.4.1 and transaction isolation level on Postgres

[Alex Vandiver]

On Tue, 3 Jan 2017 17:06:47 +0100
Václav Ovsík  wrote:
> How about the Mysql don't have this problem - is this caused by
> the different default transaction isolation level or not?

MySQL suffers from the exact same problem -- but, as it happens,
both more silently and more catastrophically.  See
https://github.com/bestpractical/rt/commit/e36364c5

> I can change isolation level in postgresql.conf to 'repeatable read'
> and things are different.

I advise against doing that.  Upon inspection, RT is not prepared to
deal with the "could not serialize access due to concurrent update"
errors that arise from updates to rows in multiple transactions in
Postgres' repeatable-read isolation.

Repeatable-read is only possible in MySQL because it has a fascinating
definition of "repeatable":

- Process 1 
mysql> set transaction isolation level repeatable read;
Query OK, 0 rows affected (0.00 sec)

mysql> start transaction with consistent snapshot;
Query OK, 0 rows affected (0.00 sec)

mysql> select id, Subject from Tickets where id = 1;
++-+
| id | Subject |
++-+
|  1 | foo |
++-+
1 row in set (0.00 sec)

- Process 2 

mysql> set transaction isolation level repeatable read;
Query OK, 0 rows affected (0.00 sec)

mysql> start transaction with consistent snapshot;
Query OK, 0 rows affected (0.00 sec)

mysql> update Tickets set Subject = 'bar' where id = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

- Process 1 

mysql> select id, Subject from Tickets where id = 1;
++-+
| id | Subject |
++-+
|  1 | foo |
++-+
1 row in set (0.00 sec)

mysql> select id, Subject from Tickets where id = 1 FOR UPDATE;
++-+
| id | Subject |
++-+
|  1 | bar |
++-+
1 row in set (0.00 sec)



Contrast this with PostgreSQL, whose definition of repeatable read
acknowledges that fully consistent updates are not possible in all
cases:

- Process 1 
rt4=# start transaction;
START TRANSACTION
rt4=# set transaction isolation level repeatable read;
SET
rt4=# select id, Subject from Tickets where id = 1;
 id | subject 
+-
  1 | foo
(1 row)

- Process 2 
rt4=# start transaction;
START TRANSACTION
rt4=# set transaction isolation level repeatable read;
SET
rt4=# update Tickets set Subject = 'bar' where id = 1;
UPDATE 1
rt4=# commit;
COMMIT

- Process 1 
rt4=# select id, Subject from Tickets where id = 1;
 id | subject 
+-
  1 | foo
(1 row)

rt4=# select id, Subject from Tickets where id = 1 FOR UPDATE;
ERROR:  could not serialize access due to concurrent update



 ( Yes, MySQL requires SET TRANSACTION ISOLATION _outside_ the
   transaction, and PostgreSQL requires it to be _inside_.  See 
   https://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
   https://www.postgresql.org/docs/9.1/static/sql-set-transaction.html )


> Should I change the default isolation level on Postgres for RT to
> 'repeatable read'?

No.  You should try the 4.4/previewscrips-race branch, which I've just
pushed:

https://github.com/bestpractical/rt/compare/4.4-trunk...4.4/previewscrips-race

The gory details are contained in the commits therein.
 - Alex

Re: [rt-users] RT 4.4.1 and transaction isolation level on Postgres

[Václav Ovsík]

On Mon, Jan 02, 2017 at 08:10:15PM -0800, Alex Vandiver wrote:
> On Mon, 2 Jan 2017 17:12:29 +0100
> Václav Ovsík  wrote:
> > Can anybody confirm on different system?
> 
> Thanks for the detailed replication instructions.  I can replicate, and
> have tracked down a minimal replication case.  I'll drop my findings
> and suggestion on your ticket.
> ...

Great, thank you very much for your time and effort!

How about the Mysql don't have this problem - is this caused by
the different default transaction isolation level or not?

I can change isolation level in postgresql.conf to 'repeatable read'
and things are different.

Should I change the default isolation level on Postgres for RT to
'repeatable read'?

-- 
Zito

Re: [rt-users] RT 4.4.1 and transaction isolation level on Postgres

[Alex Vandiver]

On Mon, 2 Jan 2017 17:12:29 +0100
Václav Ovsík  wrote:
> Can anybody confirm on different system?

Thanks for the detailed replication instructions.  I can replicate, and
have tracked down a minimal replication case.  I'll drop my findings
and suggestion on your ticket.

The short form is that this is due to 4.4's new PreviewScrips
functionality, which simulates (then rolls back) all of the changes,
which is racing with the actual change.  The bad news is that it's the
real change, not the dry run, which gets killed in the deadlock
detector, meaning that the owner does go unchanged.  At least this is
bubbled up to the user in the ticket display page, but this is still
quite unfortunate.

Amusingly, there are some strong parallels to the canonical Therac-25
case[1] -- the race here requires that one trigger a PreviewScrips very
quickly before submitting, which is rare except with users quite
accustomed to the UI.  In the Therac-25 case, only skilled users
could navigate to the bottom of the form within 8 seconds and thus
deliver lethal doses of radiation to their patients.

...RT is designed to be robust, but there's a reason it doesn't rate
itself as meant to be used for safety- or life-critical applications.

 - Alex


[1] https://en.wikipedia.org/wiki/Therac-25