[Yahoo-eng-team] [Bug 1722404] Re: Database transactions can fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" because of scatter_gather_cells

2018-02-01 Thread Matt Riedemann 
** Also affects: nova/ocata
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1722404

Title:
  Database transactions can fail with "TypeError: Can't upgrade a READER
  transaction to a WRITER mid-transaction" because of
  scatter_gather_cells

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  New
Status in OpenStack Compute (nova) pike series:
  Fix Committed

Bug description:
  I found this while working on a change to improve the usage of the
  CellDatabases test fixture by defaulting to the 'cell0' database
  instead of the 'cell1' database.

  In the set_target_cell method, we synchronize access to the cached cell
  database transaction context manager objects to prevent more than one 
  thread from using a cell's transaction context manager at the same
  time.

  In scatter_gather_cells, we're calling target_cell in such a way that
  the lock is acquired and released BEFORE the green thread actually
  accesses the database in the spawned function. So multiple threads can 
  access a cell's database transaction context manager and it's possible
  for a database transaction to fail with the error:

TypeError: Can't upgrade a READER transaction to a WRITER
   mid-transaction

  because the in-scope transaction context might be in the middle of a
  read when a concurrent green thread tries to do a write.

  I saw this happen in the test:

  
nova.tests.unit.compute.test_compute.DisabledInstanceTypesTestCase.test_can_resize_to_visible_instance_type

  where a parallel read of instances during a quota check accessed the
  cell's database transaction context manager at the same time as an
  instance.save() in the compute/api, causing the instance.save() to
  fail with "TypeError: Can't upgrade a READER transaction to a WRITER
  mid-transaction."

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1722404/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722404] Re: Database transactions can fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" because of scatter_gather_cells

2017-10-12 Thread Matt Riedemann 
** Also affects: nova/pike
   Importance: Undecided
   Status: New

** Changed in: nova/pike
   Status: New => In Progress

** Changed in: nova/pike
   Importance: Undecided => High

** Changed in: nova/pike
 Assignee: (unassigned) => melanie witt (melwitt)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1722404

Title:
  Database transactions can fail with "TypeError: Can't upgrade a READER
  transaction to a WRITER mid-transaction" because of
  scatter_gather_cells

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  In Progress

Bug description:
  I found this while working on a change to improve the usage of the
  CellDatabases test fixture by defaulting to the 'cell0' database
  instead of the 'cell1' database.

  In the set_target_cell method, we synchronize access to the cached cell
  database transaction context manager objects to prevent more than one 
  thread from using a cell's transaction context manager at the same
  time.

  In scatter_gather_cells, we're calling target_cell in such a way that
  the lock is acquired and released BEFORE the green thread actually
  accesses the database in the spawned function. So multiple threads can 
  access a cell's database transaction context manager and it's possible
  for a database transaction to fail with the error:

TypeError: Can't upgrade a READER transaction to a WRITER
   mid-transaction

  because the in-scope transaction context might be in the middle of a
  read when a concurrent green thread tries to do a write.

  I saw this happen in the test:

  
nova.tests.unit.compute.test_compute.DisabledInstanceTypesTestCase.test_can_resize_to_visible_instance_type

  where a parallel read of instances during a quota check accessed the
  cell's database transaction context manager at the same time as an
  instance.save() in the compute/api, causing the instance.save() to
  fail with "TypeError: Can't upgrade a READER transaction to a WRITER
  mid-transaction."

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1722404/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722404] Re: Database transactions can fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" because of scatter_gather_cells

2017-10-12 Thread OpenStack Infra 
Reviewed:  https://review.openstack.org/510691
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=3cc3cc453dc16e22365bea597c1be5bb0be57aeb
Submitter: Jenkins
Branch:master

commit 3cc3cc453dc16e22365bea597c1be5bb0be57aeb
Author: melanie witt 
Date:   Mon Oct 9 20:36:59 2017 +

Fix target_cell usage for scatter_gather_cells

In the set_target_cell method, we synchronize access to the cached cell
database transaction context manager objects to prevent more than one
thread from using a cell's transaction context manager at the same
time.

In scatter_gather_cells, we're calling target_cell in such a way that
the lock is acquired and released BEFORE the green thread actually
accesses the database in the spawned function. So multiple threads can
access a cell's database transaction context manager and it's possible
for a database transaction to fail with the error:

  TypeError: Can't upgrade a READER transaction to a WRITER
 mid-transaction

because the in-scope transaction context might be in the middle of a
read when a concurrent green thread tries to do a write.

Closes-Bug: #1722404

Change-Id: I07dd4d5aebdc82e343ec2035dc94c744e4754c96


** Changed in: nova
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1722404

Title:
  Database transactions can fail with "TypeError: Can't upgrade a READER
  transaction to a WRITER mid-transaction" because of
  scatter_gather_cells

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  I found this while working on a change to improve the usage of the
  CellDatabases test fixture by defaulting to the 'cell0' database
  instead of the 'cell1' database.

  In the set_target_cell method, we synchronize access to the cached cell
  database transaction context manager objects to prevent more than one 
  thread from using a cell's transaction context manager at the same
  time.

  In scatter_gather_cells, we're calling target_cell in such a way that
  the lock is acquired and released BEFORE the green thread actually
  accesses the database in the spawned function. So multiple threads can 
  access a cell's database transaction context manager and it's possible
  for a database transaction to fail with the error:

TypeError: Can't upgrade a READER transaction to a WRITER
   mid-transaction

  because the in-scope transaction context might be in the middle of a
  read when a concurrent green thread tries to do a write.

  I saw this happen in the test:

  
nova.tests.unit.compute.test_compute.DisabledInstanceTypesTestCase.test_can_resize_to_visible_instance_type

  where a parallel read of instances during a quota check accessed the
  cell's database transaction context manager at the same time as an
  instance.save() in the compute/api, causing the instance.save() to
  fail with "TypeError: Can't upgrade a READER transaction to a WRITER
  mid-transaction."

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1722404/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp