Re: [galaxy-dev] functional tests / database lock

2015-06-17 Thread Ryan G
I've migrated my staging instance to use postgres and ran the functional
tests.  I see lots failing.  I know this is known and perhaps expected, but
it still bothers me.  I would expect to see all the tests pass on a clean
instance.  If the tests aren't passing, to me, that indicates a problem
somewhere.  What's the point in having the tests if they always fail?



On Tue, Jun 16, 2015 at 1:12 PM, Ryan G ngsbioinformat...@gmail.com wrote:

 I am using postgres in production but want to validate my staging instance
 before pushing it into production.  I do this with the functional tests.  I
 guess I'll have to set up a postgres database for the staging instance as
 well.


 On Mon, Jun 15, 2015 at 10:32 AM, John Chilton jmchil...@gmail.com
 wrote:

 Not exactly - I feel like sqlite should be able to do the kinds of
 things Galaxy does without this occurring - it should just be
 inefficient. We are probably doing something across multiple threads
 that we should not be - presumably related to job handling. After a
 recent sqlalchemy upgrade - there is now a new error message in the
 logs that should help track down the problem:

 sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
 during reset or similar
 Traceback (most recent call last):
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
 _finalize_fairy
 fairy._reset(pool)
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705, in
 _reset
 pool._dialect.do_rollback(self)
   File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
 line 412, in do_rollback
 dbapi_connection.rollback()
 ProgrammingError: SQLite objects created in a thread can only be used
 in that same thread.The object was created in thread id
 140442561648384 and this is thread id 140442989479680

 At any rate - we recommend using postgres in production for a reason -
 these sqlite errors are pretty common.

 -John


 On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com
 wrote:
  Is there a known cause for this?
 
  On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo nsora...@tiscali.it
  wrote:
 
  Hi Ryan,
  there is no know fix for this, apart from moving to PostgreSQL.
 
  Cheers,
  Nicola
 
  Il 12.06.2015 16:43 Ryan G ha scritto:
 
  Hi all - I'm running functional tests on a set of tools using
  'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back and
 run
  the tool tests individually, they succeed.  So I looked into why they
 fail
  when running them in batch, and it turns out to be a database lock.
 I'm
  using the default sqllite database that run_tests.sh creates.  Does
 this
  need to be tuned somehow?
 
 
  galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
  Error - : (OperationalError) database is locked u'SELECT api_keys.id
 AS
  api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id
  AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys
 \nWHERE
  api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
  URL:
 
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
  line 149 in __call__
app_iter = self.application(environ, sr_checker)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
  line 84 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
  line 633 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 132 in __call__
return self.handle_request( environ, start_response )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 159 in handle_request
trans = self.transaction_factory( environ )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 71 in
self.set_transaction_factory( lambda e: self.transaction_chooser( e,
  galaxy_app, session_cookie ) )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 102 in transaction_chooser
return GalaxyWebTransaction( environ, galaxy_app, self,
 session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 194 in __init__
self.error_message = self._authenticate_api( session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 283 in _authenticate_api
provided_key = self.sa_session.query( self.app.model.APIKeys
 ).filter(
  self.app.model.APIKeys.table.c.key == api_key ).one()
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184
 in
  one
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227
 in
  __iter__
  

Re: [galaxy-dev] functional tests / database lock

2015-06-17 Thread Dannon Baker
Thanks, I'll take a look.  Non-tool-based functional tests should *not*
fail out of the box, and we'll fix them :)

On Wed, Jun 17, 2015 at 12:04 PM Ryan G ngsbioinformat...@gmail.com wrote:

 All the functional.test_admin_features
 Most of the functional.test_data_security
 functional.test_history_functions

 I'm attaching a PDF printout of the run_functional_tests.html



 On Wed, Jun 17, 2015 at 12:01 PM, Dannon Baker dannon.ba...@gmail.com
 wrote:

 Which tests are failing, specifically?  Many tests will fail without the
 underlying tools installed.

 On Wed, Jun 17, 2015 at 12:01 PM Ryan G ngsbioinformat...@gmail.com
 wrote:

 I've migrated my staging instance to use postgres and ran the functional
 tests.  I see lots failing.  I know this is known and perhaps expected, but
 it still bothers me.  I would expect to see all the tests pass on a clean
 instance.  If the tests aren't passing, to me, that indicates a problem
 somewhere.  What's the point in having the tests if they always fail?



 On Tue, Jun 16, 2015 at 1:12 PM, Ryan G ngsbioinformat...@gmail.com
 wrote:

 I am using postgres in production but want to validate my staging
 instance before pushing it into production.  I do this with the functional
 tests.  I guess I'll have to set up a postgres database for the staging
 instance as well.


 On Mon, Jun 15, 2015 at 10:32 AM, John Chilton jmchil...@gmail.com
 wrote:

 Not exactly - I feel like sqlite should be able to do the kinds of
 things Galaxy does without this occurring - it should just be
 inefficient. We are probably doing something across multiple threads
 that we should not be - presumably related to job handling. After a
 recent sqlalchemy upgrade - there is now a new error message in the
 logs that should help track down the problem:

 sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
 during reset or similar
 Traceback (most recent call last):
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
 _finalize_fairy
 fairy._reset(pool)
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705, in
 _reset
 pool._dialect.do_rollback(self)
   File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
 line 412, in do_rollback
 dbapi_connection.rollback()
 ProgrammingError: SQLite objects created in a thread can only be used
 in that same thread.The object was created in thread id
 140442561648384 and this is thread id 140442989479680

 At any rate - we recommend using postgres in production for a reason -
 these sqlite errors are pretty common.

 -John


 On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com
 wrote:
  Is there a known cause for this?
 
  On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo 
 nsora...@tiscali.it
  wrote:
 
  Hi Ryan,
  there is no know fix for this, apart from moving to PostgreSQL.
 
  Cheers,
  Nicola
 
  Il 12.06.2015 16:43 Ryan G ha scritto:
 
  Hi all - I'm running functional tests on a set of tools using
  'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back
 and run
  the tool tests individually, they succeed.  So I looked into why
 they fail
  when running them in batch, and it turns out to be a database
 lock.  I'm
  using the default sqllite database that run_tests.sh creates.  Does
 this
  need to be tuned somehow?
 
 
  galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
  Error - : (OperationalError) database is locked u'SELECT
 api_keys.id AS
  api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id
  AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys
 \nWHERE
  api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
  URL:
 
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
  line 149 in __call__
app_iter = self.application(environ, sr_checker)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
  line 84 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
  line 633 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 132 in __call__
return self.handle_request( environ, start_response )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 159 in handle_request
trans = self.transaction_factory( environ )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 71 in
self.set_transaction_factory( lambda e: self.transaction_chooser(
 e,
  galaxy_app, session_cookie ) )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 102 in transaction_chooser
return 

Re: [galaxy-dev] functional tests / database lock

2015-06-17 Thread Ryan G
Thanks.  Let me know how I can help.

On Wed, Jun 17, 2015 at 12:08 PM, Dannon Baker dannon.ba...@gmail.com
wrote:

 Thanks, I'll take a look.  Non-tool-based functional tests should *not*
 fail out of the box, and we'll fix them :)

 On Wed, Jun 17, 2015 at 12:04 PM Ryan G ngsbioinformat...@gmail.com
 wrote:

 All the functional.test_admin_features
 Most of the functional.test_data_security
 functional.test_history_functions

 I'm attaching a PDF printout of the run_functional_tests.html



 On Wed, Jun 17, 2015 at 12:01 PM, Dannon Baker dannon.ba...@gmail.com
 wrote:

 Which tests are failing, specifically?  Many tests will fail without the
 underlying tools installed.

 On Wed, Jun 17, 2015 at 12:01 PM Ryan G ngsbioinformat...@gmail.com
 wrote:

 I've migrated my staging instance to use postgres and ran the
 functional tests.  I see lots failing.  I know this is known and perhaps
 expected, but it still bothers me.  I would expect to see all the tests
 pass on a clean instance.  If the tests aren't passing, to me, that
 indicates a problem somewhere.  What's the point in having the tests if
 they always fail?



 On Tue, Jun 16, 2015 at 1:12 PM, Ryan G ngsbioinformat...@gmail.com
 wrote:

 I am using postgres in production but want to validate my staging
 instance before pushing it into production.  I do this with the functional
 tests.  I guess I'll have to set up a postgres database for the staging
 instance as well.


 On Mon, Jun 15, 2015 at 10:32 AM, John Chilton jmchil...@gmail.com
 wrote:

 Not exactly - I feel like sqlite should be able to do the kinds of
 things Galaxy does without this occurring - it should just be
 inefficient. We are probably doing something across multiple threads
 that we should not be - presumably related to job handling. After a
 recent sqlalchemy upgrade - there is now a new error message in the
 logs that should help track down the problem:

 sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
 during reset or similar
 Traceback (most recent call last):
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
 _finalize_fairy
 fairy._reset(pool)
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705,
 in _reset
 pool._dialect.do_rollback(self)
   File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
 line 412, in do_rollback
 dbapi_connection.rollback()
 ProgrammingError: SQLite objects created in a thread can only be used
 in that same thread.The object was created in thread id
 140442561648384 and this is thread id 140442989479680

 At any rate - we recommend using postgres in production for a reason -
 these sqlite errors are pretty common.

 -John


 On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com
 wrote:
  Is there a known cause for this?
 
  On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo 
 nsora...@tiscali.it
  wrote:
 
  Hi Ryan,
  there is no know fix for this, apart from moving to PostgreSQL.
 
  Cheers,
  Nicola
 
  Il 12.06.2015 16:43 Ryan G ha scritto:
 
  Hi all - I'm running functional tests on a set of tools using
  'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back
 and run
  the tool tests individually, they succeed.  So I looked into why
 they fail
  when running them in batch, and it turns out to be a database
 lock.  I'm
  using the default sqllite database that run_tests.sh creates.
 Does this
  need to be tuned somehow?
 
 
  galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job
 dispatched
  Error - : (OperationalError) database is locked u'SELECT
 api_keys.id AS
  api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id
  AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM
 api_keys \nWHERE
  api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
  URL:
 
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
  line 149 in __call__
app_iter = self.application(environ, sr_checker)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
  line 84 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
  line 633 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 132 in __call__
return self.handle_request( environ, start_response )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 159 in handle_request
trans = self.transaction_factory( environ )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 71 in
self.set_transaction_factory( lambda e:
 self.transaction_chooser( e,
  galaxy_app, session_cookie ) )
  File
 
 

Re: [galaxy-dev] functional tests / database lock

2015-06-17 Thread Dannon Baker
Which tests are failing, specifically?  Many tests will fail without the
underlying tools installed.

On Wed, Jun 17, 2015 at 12:01 PM Ryan G ngsbioinformat...@gmail.com wrote:

 I've migrated my staging instance to use postgres and ran the functional
 tests.  I see lots failing.  I know this is known and perhaps expected, but
 it still bothers me.  I would expect to see all the tests pass on a clean
 instance.  If the tests aren't passing, to me, that indicates a problem
 somewhere.  What's the point in having the tests if they always fail?



 On Tue, Jun 16, 2015 at 1:12 PM, Ryan G ngsbioinformat...@gmail.com
 wrote:

 I am using postgres in production but want to validate my staging
 instance before pushing it into production.  I do this with the functional
 tests.  I guess I'll have to set up a postgres database for the staging
 instance as well.


 On Mon, Jun 15, 2015 at 10:32 AM, John Chilton jmchil...@gmail.com
 wrote:

 Not exactly - I feel like sqlite should be able to do the kinds of
 things Galaxy does without this occurring - it should just be
 inefficient. We are probably doing something across multiple threads
 that we should not be - presumably related to job handling. After a
 recent sqlalchemy upgrade - there is now a new error message in the
 logs that should help track down the problem:

 sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
 during reset or similar
 Traceback (most recent call last):
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
 _finalize_fairy
 fairy._reset(pool)
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705, in
 _reset
 pool._dialect.do_rollback(self)
   File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
 line 412, in do_rollback
 dbapi_connection.rollback()
 ProgrammingError: SQLite objects created in a thread can only be used
 in that same thread.The object was created in thread id
 140442561648384 and this is thread id 140442989479680

 At any rate - we recommend using postgres in production for a reason -
 these sqlite errors are pretty common.

 -John


 On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com
 wrote:
  Is there a known cause for this?
 
  On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo nsora...@tiscali.it
  wrote:
 
  Hi Ryan,
  there is no know fix for this, apart from moving to PostgreSQL.
 
  Cheers,
  Nicola
 
  Il 12.06.2015 16:43 Ryan G ha scritto:
 
  Hi all - I'm running functional tests on a set of tools using
  'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back
 and run
  the tool tests individually, they succeed.  So I looked into why they
 fail
  when running them in batch, and it turns out to be a database lock.
 I'm
  using the default sqllite database that run_tests.sh creates.  Does
 this
  need to be tuned somehow?
 
 
  galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
  Error - : (OperationalError) database is locked u'SELECT api_keys.id
 AS
  api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id
  AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys
 \nWHERE
  api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
  URL:
 
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
  line 149 in __call__
app_iter = self.application(environ, sr_checker)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
  line 84 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
  line 633 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 132 in __call__
return self.handle_request( environ, start_response )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 159 in handle_request
trans = self.transaction_factory( environ )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 71 in
self.set_transaction_factory( lambda e: self.transaction_chooser( e,
  galaxy_app, session_cookie ) )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 102 in transaction_chooser
return GalaxyWebTransaction( environ, galaxy_app, self,
 session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 194 in __init__
self.error_message = self._authenticate_api( session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 283 in _authenticate_api
provided_key = self.sa_session.query( self.app.model.APIKeys
 ).filter(
  self.app.model.APIKeys.table.c.key == 

Re: [galaxy-dev] functional tests / database lock

2015-06-16 Thread Ryan G
I am using postgres in production but want to validate my staging instance
before pushing it into production.  I do this with the functional tests.  I
guess I'll have to set up a postgres database for the staging instance as
well.


On Mon, Jun 15, 2015 at 10:32 AM, John Chilton jmchil...@gmail.com wrote:

 Not exactly - I feel like sqlite should be able to do the kinds of
 things Galaxy does without this occurring - it should just be
 inefficient. We are probably doing something across multiple threads
 that we should not be - presumably related to job handling. After a
 recent sqlalchemy upgrade - there is now a new error message in the
 logs that should help track down the problem:

 sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
 during reset or similar
 Traceback (most recent call last):
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
 _finalize_fairy
 fairy._reset(pool)
   File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705, in
 _reset
 pool._dialect.do_rollback(self)
   File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
 line 412, in do_rollback
 dbapi_connection.rollback()
 ProgrammingError: SQLite objects created in a thread can only be used
 in that same thread.The object was created in thread id
 140442561648384 and this is thread id 140442989479680

 At any rate - we recommend using postgres in production for a reason -
 these sqlite errors are pretty common.

 -John


 On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com
 wrote:
  Is there a known cause for this?
 
  On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo nsora...@tiscali.it
  wrote:
 
  Hi Ryan,
  there is no know fix for this, apart from moving to PostgreSQL.
 
  Cheers,
  Nicola
 
  Il 12.06.2015 16:43 Ryan G ha scritto:
 
  Hi all - I'm running functional tests on a set of tools using
  'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back and
 run
  the tool tests individually, they succeed.  So I looked into why they
 fail
  when running them in batch, and it turns out to be a database lock.  I'm
  using the default sqllite database that run_tests.sh creates.  Does this
  need to be tuned somehow?
 
 
  galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
  Error - : (OperationalError) database is locked u'SELECT api_keys.id AS
  api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id
  AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys
 \nWHERE
  api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
  URL:
 
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
  line 149 in __call__
app_iter = self.application(environ, sr_checker)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
  line 84 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
  line 633 in __call__
return self.application(environ, start_response)
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 132 in __call__
return self.handle_request( environ, start_response )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
  line 159 in handle_request
trans = self.transaction_factory( environ )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 71 in
self.set_transaction_factory( lambda e: self.transaction_chooser( e,
  galaxy_app, session_cookie ) )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 102 in transaction_chooser
return GalaxyWebTransaction( environ, galaxy_app, self,
 session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 194 in __init__
self.error_message = self._authenticate_api( session_cookie )
  File
 
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
  line 283 in _authenticate_api
provided_key = self.sa_session.query( self.app.model.APIKeys ).filter(
  self.app.model.APIKeys.table.c.key == api_key ).one()
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184
 in
  one
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227
 in
  __iter__
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242
 in
  _execute_and_instances
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449
  in execute
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1584
  in _execute_clauseelement
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698
  in _execute_context
  File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', 

Re: [galaxy-dev] functional tests / database lock

2015-06-15 Thread John Chilton
Not exactly - I feel like sqlite should be able to do the kinds of
things Galaxy does without this occurring - it should just be
inefficient. We are probably doing something across multiple threads
that we should not be - presumably related to job handling. After a
recent sqlalchemy upgrade - there is now a new error message in the
logs that should help track down the problem:

sqlalchemy.pool.NullPool ERROR 2015-06-15 09:53:47,547 Exception
during reset or similar
Traceback (most recent call last):
  File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 571, in
_finalize_fairy
fairy._reset(pool)
  File build/bdist.linux-x86_64/egg/sqlalchemy/pool.py, line 705, in _reset
pool._dialect.do_rollback(self)
  File build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py,
line 412, in do_rollback
dbapi_connection.rollback()
ProgrammingError: SQLite objects created in a thread can only be used
in that same thread.The object was created in thread id
140442561648384 and this is thread id 140442989479680

At any rate - we recommend using postgres in production for a reason -
these sqlite errors are pretty common.

-John


On Fri, Jun 12, 2015 at 11:15 AM, Ryan G ngsbioinformat...@gmail.com wrote:
 Is there a known cause for this?

 On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo nsora...@tiscali.it
 wrote:

 Hi Ryan,
 there is no know fix for this, apart from moving to PostgreSQL.

 Cheers,
 Nicola

 Il 12.06.2015 16:43 Ryan G ha scritto:

 Hi all - I'm running functional tests on a set of tools using
 'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back and run
 the tool tests individually, they succeed.  So I looked into why they fail
 when running them in batch, and it turns out to be a database lock.  I'm
 using the default sqllite database that run_tests.sh creates.  Does this
 need to be tuned somehow?


 galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
 Error - : (OperationalError) database is locked u'SELECT api_keys.id AS
 api_keys_id, api_keys.create_time AS api_keys_create_time, api_keys.user_id
 AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys \nWHERE
 api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
 URL:
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
 line 149 in __call__
   app_iter = self.application(environ, sr_checker)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
 line 84 in __call__
   return self.application(environ, start_response)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
 line 633 in __call__
   return self.application(environ, start_response)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
 line 132 in __call__
   return self.handle_request( environ, start_response )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
 line 159 in handle_request
   trans = self.transaction_factory( environ )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 71 in
   self.set_transaction_factory( lambda e: self.transaction_chooser( e,
 galaxy_app, session_cookie ) )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 102 in transaction_chooser
   return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 194 in __init__
   self.error_message = self._authenticate_api( session_cookie )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 283 in _authenticate_api
   provided_key = self.sa_session.query( self.app.model.APIKeys ).filter(
 self.app.model.APIKeys.table.c.key == api_key ).one()
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184 in
 one
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227 in
 __iter__
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242 in
 _execute_and_instances
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449
 in execute
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1584
 in _execute_clauseelement
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698
 in _execute_context
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1691
 in _execute_context
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py', line 331
 in do_execute
 OperationalError: (OperationalError) database is locked u'SELECT
 api_keys.id AS api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM
 api_keys \nWHERE api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)





 Connetti 

Re: [galaxy-dev] functional tests / database lock

2015-06-12 Thread Nicola Soranzo
  Hi Ryan,
there is no know fix for this, apart from moving to
PostgreSQL.

Cheers,
Nicola

Il 12.06.2015 16:43 Ryan G ha scritto: 


Hi all - I'm running functional tests on a set of tools using
'run_tests.sh -sid ...'. A few of the tests fail. When I go back and run
the tool tests individually, they succeed. So I looked into why they
fail when running them in batch, and it turns out to be a database lock.
I'm using the default sqllite database that run_tests.sh creates. Does
this need to be tuned somehow?
 
 galaxy.jobs.handler INFO 2015-06-12
10:28:35,854 (14) Job dispatched
 Error - : (OperationalError) database
is locked u'SELECT api_keys.id [1] AS api_keys_id, api_keys.create_time
AS api_keys_create_time, api_keys.user_id AS api_keys_user_id,
api_keys.key AS api_keys_key nFROM api_keys nWHERE api_keys.key = ?'
(u'e4fe9bf787edb9ef746714beb6212bc8',)
 URL:
http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
[2]
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
line 149 in __call__
 app_iter = self.application(environ,
sr_checker)
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
line 84 in __call__
 return self.application(environ,
start_response)
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
line 633 in __call__
 return self.application(environ,
start_response)
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
line 132 in __call__
 return self.handle_request( environ,
start_response )
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
line 159 in handle_request
 trans = self.transaction_factory( environ
)
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
line 71 in 
 self.set_transaction_factory( lambda e:
self.transaction_chooser( e, galaxy_app, session_cookie ) )
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
line 102 in transaction_chooser
 return GalaxyWebTransaction( environ,
galaxy_app, self, session_cookie )
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
line 194 in __init__
 self.error_message = self._authenticate_api(
session_cookie )
 File
'/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
line 283 in _authenticate_api
 provided_key = self.sa_session.query(
self.app.model.APIKeys ).filter( self.app.model.APIKeys.table.c.key ==
api_key ).one()
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184 in
one
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line
2227 in __iter__
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242 in
_execute_and_instances
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449 in
execute
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py',
line 1584 in _execute_clauseelement
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698 in
_execute_context
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1691 in
_execute_context
 File
'build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py', line 331 in
do_execute
 OperationalError: (OperationalError) database is locked
u'SELECT api_keys.id [3] AS api_keys_id, api_keys.create_time AS
api_keys_create_time, api_keys.user_id AS api_keys_user_id,
api_keys.key AS api_keys_key nFROM api_keys nWHERE api_keys.key = ?'
(u'e4fe9bf787edb9ef746714beb6212bc8',)
  


Connetti gratis il mondo con la nuova indoona:  hai la chat, le chiamate, le 
video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] functional tests / database lock

2015-06-12 Thread Ryan G
Is there a known cause for this?

On Fri, Jun 12, 2015 at 10:52 AM, Nicola Soranzo nsora...@tiscali.it
wrote:

  Hi Ryan,
 there is no know fix for this, apart from moving to PostgreSQL.

 Cheers,
 Nicola

 Il 12.06.2015 16:43 Ryan G ha scritto:

 Hi all - I'm running functional tests on a set of tools using
 'run_tests.sh -sid ...'.  A few of the tests fail.  When I go back and run
 the tool tests individually, they succeed.  So I looked into why they fail
 when running them in batch, and it turns out to be a database lock.  I'm
 using the default sqllite database that run_tests.sh creates.  Does this
 need to be tuned somehow?


 galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched
 Error - : (OperationalError) database is locked u'SELECT api_keys.id AS
 api_keys_id, api_keys.create_time AS api_keys_create_time, api_keys.user_id
 AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM api_keys \nWHERE
 api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)
 URL:
 http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef746714beb6212bc8
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py',
 line 149 in __call__
   app_iter = self.application(environ, sr_checker)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
 line 84 in __call__
   return self.application(environ, start_response)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
 line 633 in __call__
   return self.application(environ, start_response)
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
 line 132 in __call__
   return self.handle_request( environ, start_response )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py',
 line 159 in handle_request
   trans = self.transaction_factory( environ )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 71 in
   self.set_transaction_factory( lambda e: self.transaction_chooser( e,
 galaxy_app, session_cookie ) )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 102 in transaction_chooser
   return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 194 in __init__
   self.error_message = self._authenticate_api( session_cookie )
 File
 '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py',
 line 283 in _authenticate_api
   provided_key = self.sa_session.query( self.app.model.APIKeys ).filter(
 self.app.model.APIKeys.table.c.key == api_key ).one()
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184 in
 one
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227 in
 __iter__
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242 in
 _execute_and_instances
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449
 in execute
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1584
 in _execute_clauseelement
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698
 in _execute_context
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1691
 in _execute_context
 File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py', line 331
 in do_execute
 OperationalError: (OperationalError) database is locked u'SELECT
 api_keys.id AS api_keys_id, api_keys.create_time AS api_keys_create_time,
 api_keys.user_id AS api_keys_user_id, api_keys.key AS api_keys_key \nFROM
 api_keys \nWHERE api_keys.key = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)





 Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate,
 le video chiamate e persino le chiamate di gruppo.
 E chiami gratis anche i numeri fissi e mobili nel mondo!
 Scarica subito l’app Vai su https://www.indoona.com/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/