Re: [galaxy-dev] bioblend problem trying to use run_tool function
Hello John,
Sorry to bother you again with this. I saw the many improvments added
but I still can't run my tool using bioblend.
The inputs names are 'input' and 'prefix' and the tool is very simple.
The problem is still a mapping issue. Parameters are not assigned
correctly in the command line. options are there but no values are there.
Could you give me a use case, so I can mime it to my project ?
Thanks in advance,
--
Olivia Doppelt-Azeroual
CIB, Institut Pasteur, Paris
Le 27/02/14 14:46, John Chilton a écrit :
I think you are right, and I am right, but you more so :). I bet the
inputs need to be dumped to json if files are being attached (i.e.
upload is being used) but should not be otherwise (because
content-type will be different). I wrote the client just to test
uploads - so this is probably the problem.
I have created a Trello card (https://trello.com/c/67HsrgVm) to fix
bioblend to handle these details transparently and (per your other
email) to test and provide examples of using a repeat with data
parameters. I am certain the Galaxy API allows this - I am not sure
the bioblend tool client is quite there yet - it shouldn't be hard to
update it or provide examples - but I will need to find some time
which can be difficult.
-John
On Tue, Feb 25, 2014 at 4:41 AM, Olivia Doppelt-Azeroual
olivia.dopp...@pasteur.fr wrote:
Dear John,
I am doing tests on a local version of the last galaxy-dist version before
updating our Galaxy server. And I have a question concernant the second
remark you made in your last email.
I've written a small script to test the run_tool function. When i put back
the dumps(inputs_dict), the script fails whereas the type of my input_dict
is a dictionnary.
The script is the following:
--
from bioblend.galaxy import GalaxyInstance
giLocal=GalaxyInstance('http://127.0.0.1:8080/',
key='43359c833b8977b01d6237d9b839dc27')
tool_inputs_loc={'input':'f597429621d6eb2b','name':data}
print type(tool_inputs_loc)
giLocal.tools.run_tool(history_id='1cd8e2f6b131e891',tool_id='export',tool_inputs=tool_inputs_loc)
---
when i print the payload it is:
{'tool_id': 'export', 'history_id': '1cd8e2f6b131e891', 'inputs': '{input:
f597429621d6eb2b, name: data}'}
On the server side, the error is the following:
--
galaxy.web.framework ERROR 2014-02-25 11:35:34,865 Uncaught exception in
exposed API method:
Traceback (most recent call last):
File
/home/odoppelt/localStuff_VM/tools/new_galaxy_disr/galaxy-dist/lib/galaxy/web/framework/__init__.py,
line 200, in decorator
rval = func( self, trans, *args, **kwargs)
File
/home/odoppelt/localStuff_VM/tools/new_galaxy_disr/galaxy-dist/lib/galaxy/webapps/galaxy/api/tools.py,
line 104, in create
for k, v in inputs.iteritems():
AttributeError: 'unicode' object has no attribute 'iteritems'
Do you have an idea ?
Have a nice day,
Thank you for your help,
--
Olivia
Le 24/02/14 16:18, John Chilton a écrit :
Yes, the get_intial_value bug was a bug that has been fixed in a
recent Galaxy release (I think the most recent one, but maybe the
previous one). Unfortunately though, I fixed other bugs related to
running more sophisticated tools through the API at the same time - so
I would again strongly recommend upgrading before proceeding or you
going to find all the same bugs I did. The tool running API has
significantly improved over the last couple releases.
As for removing the dumps statement in bioblend - I think (though am
not certain) that is the wrong thing to do. I think you are likely
somehow doing this at a level above the client call and shouldn't be -
i.e. my guess is you are passing in a string instead of a dictionary
and the result is that dumps it dumping an escaped string - instead of
jsonifing a dictionary.
For instance, the API driven test framework runs by calling this API
and does almost this exact dumps.
https://bitbucket.org/galaxy/galaxy-central/src/f573461f3667e8f120dc05bbd11a019d5fae1960/test/base/interactor.py?at=default#cl-251
Hope this helps.
-John
On Mon, Feb 24, 2014 at 8:51 AM, Olivia Doppelt-Azeroual
olivia.dopp...@pasteur.fr wrote:
I detected another problem which enables me to go further, however
doesn't
resolve the whole problem yet.
the function get_initial value didn't use the history in the recursive
call,
which led to an empty history. I changed it juste adding history to the
input.get_initial_value
def get_initial_value( self, trans, context, history=None ):
rval = []
for i in range( self.default ):
rval_dict = { '__index__': i}
for input in self.inputs.itervalues():
rval_dict[ input.name ] = input.get_initial_value(
trans,
context, history )
rval.append( rval_dict )
return rval
There is another problem though.
On the bioblend side:
_
Traceback (most recent call last):
File
Re: [galaxy-dev] workflow_execute_parameters.py
Dear Dannon,
when the parameter (that I need to fix) is 'set at runtime', the value
that it gets is: __lt__galaxy.tools.parameters.basic.RuntimeValue
python workflow_execute_parameters.py 43359c833b8977b01d6237d9b839dc27
http://127.0.0.1:8080/api/workflows f2db41e1fa331b3e a799d38679e985db
'14=hda=52ea4fd71e9f7c3f' 'param=export_sbw=prefix=test'
the parameter prefix needs to be set at the value test (I tried
without the quotes also)
---
Here is what you asked for:
{
id: f2db41e1fa331b3e,
inputs: {
14: {
label: input Dataset,
value:
}
},
model_class: StoredWorkflow,
name: export_WF,
published: false,
steps: {
14: {
id: 14,
input_steps: {},
tool_id: null,
type: data_input
},
15: {
id: 15,
input_steps: {
inputs_0|input: {
source_step: 14,
step_output: output
}
},
tool_id: export_sbw,
type: tool
}
},
tags: [],
url: /api/workflows/f2db41e1fa331b3e
}
as you can see, the parameter called prefix, is not there.
Thank you for your help
--
Olivia
Le 25/04/14 19:49, Dannon Baker a écrit :
Olivia,
Is the parameter set to 'set at runtime' when you view the workflow in
the editor? Could you send me the workflow json (just the
http://127.0.0.1:8080/api/workflows/workflow_id/ bit)?
-Dannon
On Fri, Apr 25, 2014 at 5:26 AM, Olivia Doppelt-Azeroual
olivia.dopp...@pasteur.fr mailto:olivia.dopp...@pasteur.fr wrote:
Dear Dannon,
Thank you for your answer, it works better now that I know the
steps ID for my workflow.
I have another question though.
The second step of my WF uses a parameter input named prefix. I
really need to specify it when I launch the WF using the API.
When i display the WF using the url you gave me, the parameters
prefix, is not there and when I use this in my command line, the
specified Value is not read by Galaxy.
python workflow_execute_parameters.py
43359c833b8977b01d6237d9b839dc27
http://127.0.0.1:8080/api/workflows f2db41e1fa331b3e
a799d38679e985db '10=hda=52ea4fd71e9f7c3f'
'param=export_sbw=prefix=test'
Could you help me on that ? How do I give a value to a parameter
if it does not appear in the WF description ??
Thanks in advance,
--
Olivia
Le 23/04/14 18:40, Dannon Baker a écrit :
Hi Olivia!
It's probably talking about the workflowstep object with
identifier 8. If you request the description of this workflow
via the API (http://yourgalaxyinstance.org/api/workflows/the id
of this workflow), you'll be given the exact step identifiers
to use. In other words, unless this is the first workflow ever
created on a brand new galaxy instance, your step ids will not be
0 and 1.
-Dannon
On Wed, Apr 23, 2014 at 12:30 PM, Olivia Doppelt-Azeroual
olivia.dopp...@pasteur.fr mailto:olivia.dopp...@pasteur.fr wrote:
Dear developpers,
I'm trying to launch a small workflow using the api script
workflow_execute_parameters.py
I have 2 steps: the input data step and a second step which
needs a parameter called prefix.
The workflow is attached to this email.
I get an error on the parameters inputs which asks me to
specify information about an 8th step (which does not exist)
Is there a test case available to use this script ?
Please keep me informed on that.
Have a nice day,
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] workflow_execute_parameters.py
Dear Dannon, Thank you for your answer, it works better now that I know the steps ID for my workflow. I have another question though. The second step of my WF uses a parameter input named prefix. I really need to specify it when I launch the WF using the API. When i display the WF using the url you gave me, the parameters prefix, is not there and when I use this in my command line, the specified Value is not read by Galaxy. python workflow_execute_parameters.py 43359c833b8977b01d6237d9b839dc27 http://127.0.0.1:8080/api/workflows f2db41e1fa331b3e a799d38679e985db '10=hda=52ea4fd71e9f7c3f' 'param=export_sbw=prefix=test' Could you help me on that ? How do I give a value to a parameter if it does not appear in the WF description ?? Thanks in advance, -- Olivia Le 23/04/14 18:40, Dannon Baker a écrit : Hi Olivia! It's probably talking about the workflowstep object with identifier 8. If you request the description of this workflow via the API (http://yourgalaxyinstance.org/api/workflows/the id of this workflow), you'll be given the exact step identifiers to use. In other words, unless this is the first workflow ever created on a brand new galaxy instance, your step ids will not be 0 and 1. -Dannon On Wed, Apr 23, 2014 at 12:30 PM, Olivia Doppelt-Azeroual olivia.dopp...@pasteur.fr mailto:olivia.dopp...@pasteur.fr wrote: Dear developpers, I'm trying to launch a small workflow using the api script workflow_execute_parameters.py I have 2 steps: the input data step and a second step which needs a parameter called prefix. The workflow is attached to this email. I get an error on the parameters inputs which asks me to specify information about an 8th step (which does not exist) Is there a test case available to use this script ? Please keep me informed on that. Have a nice day, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] workflow_execute_parameters.py
Dear developpers,
I'm trying to launch a small workflow using the api script
workflow_execute_parameters.py
I have 2 steps: the input data step and a second step which needs a
parameter called prefix.
The workflow is attached to this email.
I get an error on the parameters inputs which asks me to specify
information about an 8th step (which does not exist)
Is there a test case available to use this script ?
Please keep me informed on that.
Have a nice day,
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
{
a_galaxy_workflow: true,
annotation: ,
format-version: 0.1,
name: export_WF,
steps: {
0: {
annotation: ,
id: 0,
input_connections: {},
inputs: [
{
description: ,
name: input Dataset
}
],
name: Input dataset,
outputs: [],
position: {
left: 289.8824539185,
top: 449.500984192
},
tool_errors: null,
tool_id: null,
tool_state: {\name\: \input Dataset\},
tool_version: null,
type: data_input,
user_outputs: []
},
1: {
annotation: ,
id: 1,
input_connections: {
inputs_0|input: {
id: 0,
output_name: output
}
},
inputs: [],
name: export data for SynBioWatch,
outputs: [
{
name: output,
type: text
}
],
position: {
left: 569.882453918,
top: 369.806159973
},
post_job_actions: {},
tool_errors: null,
tool_id: export_sbw,
tool_state: {\inputs\: \[{\\\__index__\\\: 0,
\\\prefix\\\: {\\\__class__\\\: \\\RuntimeValue\\\}, \\\input\\\:
null}]\, \__rerun_remap_job_id__\: null, \__page__\: 0},
tool_version: 1.0,
type: tool,
user_outputs: []
}
}
}___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] launching openmpi tool with Galaxy
Dear developers, I'm trying to launch a mpi version of test tool. To do so, I have a test xml called sleep.xml but all he does is write the hostname of the machine in an output file. (command: /usr/lib64/openmpi/bin/mpirun hostname $output) When I launch this tool as a local job (we use son of grid engine as the job scheduler), everything is ok. When i specify the following line in the universe_wsgi.xml: sleep = drmaa://-q galaxy -pe openmpi 8/ The program is launched but it is lost somewhere and juste doesn't do anything. It is just running and never ends. Does someone has an explanation for that ? Thanks in advance, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] bioblend run_tool function used with a tool containing repeats for inputs in its XML
Dear Developpers, I'm still trying to use the bioblend function run_tool and now that I've uptdated my version of galaxy, the problem comes from the input_dict used to define the inputs. The command line generated is right except for the inputs information, it is either None are sometimes, nothing is just written there. I'm starting to think that it is due to the fact that in my xml tool, there is a way to add as many inputs as needed using the repeats tag. This maybe confuses the link between bioblend and galaxy as the variables are named differently in the cheetah command line. Is there an example of a run_tool utilisation ? somewhere ? Thanks in advance for your help. Have a nice day, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bioblend problem trying to use run_tool function
Hello John,
Yes it helps a little but I still have the following error:
___
Traceback (most recent call last):
File script_testBioblend_distant.py, line 16, in module
giLocal.tools.run_tool(history_id='0a248a1f62a0cc04',tool_id='export',tool_inputs=tool_inputs_loc)
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxy/tools/__init__.py,
line 26, in run_tool
return Client._post(self, payload)
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxy/client.py,
line 181, in _post
r = self.gi.make_post_request(url, payload=payload,
files_attached=files_attached)
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxyclient.py,
line 94, in make_post_request
r.status_code, body=r.text) # @see self.body for HTTP response body
bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy:
500: html
headtitleInternal Server Error/title/head
body
h1Internal Server Error/h1
pThe server has either erred or is incapable of performing
the requested operation.
br/
!-- --/p
hr noshade
div align=rightWSGI Server/div
/body
/html
___
I'm currently trying to understand this error. Please tell me if you
know anything about this error.
Have a nice day,
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
Le 21/02/14 15:49, John Chilton a écrit :
I copied the HTML into a new document, opened it, and copied the
stacktrace produced by Galaxy - it looks like this:
Internal Server Error
Galaxy was unable to successfully complete your request
URL: http://galaxy.web.pasteur.fr/api/tools?key=c23d8cb98671911813305051710d4ae1
Module galaxy.web.framework.middleware.error:149 in __call__
app_iter = self.application(environ, sr_checker)
Module paste.recursive:84 in __call__
return self.application(environ, start_response)
Module galaxy.web.framework.middleware.remoteuser:96 in __call__
return self.app( environ, start_response )
Module paste.httpexceptions:633 in __call__
return self.application(environ, start_response)
Module galaxy.web.framework.base:132 in __call__
return self.handle_request( environ, start_response )
Module galaxy.web.framework.base:190 in handle_request
body = method( trans, **kwargs )
Module galaxy.web.framework:201 in decorator
log.exception( 'Uncaught exception in exposed API method:'+
str(err),exc_info=True)
TypeError: exception() got an unexpected keyword argument 'exc_info'
So what is happening here is and error is occurring when Galaxy is
trying to log another error - causing the API to break down. I don't
think the newest release of Galaxy has this line:
log.exception( 'Uncaught exception in exposed API method:'+
str(err),exc_info=True)
I am not sure what exc_info is - but it looks like it shouldn't be
there. You may want to remove that and see if you get a cleaner error
message that is easier to interpret.
Hope this helps.
-John
On Fri, Feb 21, 2014 at 3:50 AM, Olivia Doppelt
olivia.dopp...@pasteur.fr wrote:
Dear Developers,
I would like to use bioblend to export the result of a galaxy workflow.
To do that, we want to use an implemented tool called export_data; which
enables the copy of a result file from galaxy arborescence to a local
directory which belong to the user making the analysis and the export.
We tried to launch this tool using the run_tool function but it doesn't
work.
giOlivia.tools.run_tool(history_id='xxx',tool_id='/export_data/export/2.2',tool_inputs={'input':'id_data','name':data})
I get an error message which is very long and attached to this email.
Please help me on that it is the final step of several ones using bioblend.
Have a nice day,
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bioblend problem trying to use run_tool function
I detected another problem which enables me to go further, however
doesn't resolve the whole problem yet.
the function get_initial value didn't use the history in the recursive
call, which led to an empty history. I changed it juste adding *history*
to the input.get_initial_value
def get_initial_value( self, trans, context, history=None ):
rval = []
for i in range( self.default ):
rval_dict = { '__index__': i}
for input in self.inputs.itervalues():
rval_dict[ input.name ] = input.get_initial_value(
trans, context, *history* )
rval.append( rval_dict )
return rval
There is another problem though.
On the bioblend side:
_
Traceback (most recent call last):
File script_testBioblend_distant.py, line 16, in module
giLocal.tools.run_tool(history_id='0a248a1f62a0cc04',tool_id='export',tool_inputs={'input':'8ad39366bdad8699','name':data})
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxy/tools/__init__.py,
line 26, in run_tool
return Client._post(self, payload)
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxy/client.py,
line 181, in _post
r = self.gi.make_post_request(url, payload=payload,
files_attached=files_attached)
File
/home/odoppelt/localStuff_VM/galaxy_work/SBW_project/bioblend/bioblend/galaxyclient.py,
line 94, in make_post_request
r.status_code, body=r.text) # @see self.body for HTTP response body
bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy:
400: {
message: {
data: {
inputs: [
{
input: History does not include a dataset of the
required format / build
}
]
},
type: error
}
}
On the galaxy side, there is no error message.
Sorry for the many mails but I prefer to keep you updated so you dont
waste time.
--
Olivia
Le 24/02/14 15:29, Olivia Doppelt-Azeroual a écrit :
Yes, indeed,
the log from galaxy are the following:
The first error was due to the fact that bioblend function run_tools;
gave galaxy a dump of the input_dicts which type is string where
galaxy was waiting for an item structure.
I removed the dump:
payload[inputs] = tool_inputs
instead of
payload[inputs] = dumps(tool_inputs)
However, there is another error afterwards which is:
___
/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/types.py:215:
SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects
natively, and SQLAlchemy must convert from floating point - rounding
errors and other issues may occur. Please consider storing Decimal
numbers as strings or integers on this platform for lossless storage.
galaxy.web.framework ERROR 2014-02-24 15:15:15,374 Uncaught exception
in exposed API method:
Traceback (most recent call last):
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/web/framework/__init__.py,
line 203, in decorator
rval = func( self, trans, *args, **kwargs)
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/webapps/galaxy/api/tools.py,
line 123, in create
template, vars = tool.handle_input( trans, params.__dict__,
history=target_history )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/__init__.py,
line 1921, in handle_input
state = self.new_state( trans, history=history )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/__init__.py,
line 1846, in new_state
self.fill_in_new_state( trans, inputs, state.inputs,
history=history )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/__init__.py,
line 1855, in fill_in_new_state
state[ input.name ] = input.get_initial_value( trans, context,
history=history )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/parameters/grouping.py,
line 116, in get_initial_value
rval_dict[ input.name ] = input.get_initial_value( trans, context )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/parameters/basic.py,
line 1657, in get_initial_value
return self.get_initial_value_from_history_prevent_repeats(trans,
context, None, history=history);
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/lib/galaxy/tools/parameters/basic.py,
line 1670, in get_initial_value_from_history_prevent_repeats
assert history is not None, DataToolParameter requires a history
AssertionError: DataToolParameter requires a history
127.0.0.1 - - [24/Feb/2014:15:15:15 +0200] POST
/api/tools?key=cd065330048989af4ae266cf94685e6b HTTP/1.1 500 - -
python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-15-generic
___
Thanks for your help,
Olivia
Re: [galaxy-dev] BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Dear John,
Sorry to have disturb you with that problem actually it was linked to a
local fact.
We had in the past a problem with the api authentification so as a work
around, we have 2 galaxy servers. one for galaxy and another for the
api. Actually it is quite new (a few weeks), so i was not very familiar
with it.
The problem was because i only reboot galaxy and not galaxy api... so my
changes were not taken into account.
So your patch now works !! Thank you again, we can go on with our
project using Bioblend.
Have a nice day !
-
Olivia
Le 10/01/14 15:35, John Chilton a écrit :
I am stumped, I am not sure how this would happen. How did you apply
the patch against the -dist stable code? Can you send me the full
contents of the patched ..api/library_contents.py file?
-John
On Tue, Jan 7, 2014 at 9:48 AM, Olivia Doppelt
olivia.dopp...@pasteur.fr wrote:
Dear John,
Unfortunatly, I have another problem.
The galaxy version in use for our institute was last updated on December
10th.
Weirdly, when i put the patch in the library_contents.py file, I get another
error (which i don't have on my local version)
Actually, it never goes in the code i patched... and it doesn't underdand
the string which is the role.
the error in the following:
_
157.99.60.187 - - [07/Jan/2014:16:36:31 +0200] GET
/api/libraries/79449a84684b4136/contents?key=API_key HTTP/1.1 200 - -
python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-12-generic
sqlalchemy.pool.QueuePool WARNING 2014-01-07 16:36:32,212 Error closing
cursor: current transaction is aborted, commands ignored until end of
transaction block
galaxy.web.framework ERROR 2014-01-07 16:36:32,212 Uncaught exception in
exposed API method:
Traceback (most recent call last):
File //galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py, line
197, in decorator
rval = func( self, trans, *args, **kwargs)
File
//galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/library_contents.py,
line 183, in create
role = str( trans.security.decode_id( role ) )
File
//galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py,
line 843, in upload_library_dataset
trans.app.security_agent.derive_roles_from_access( trans, library.id,
cntrller, library=True, **vars )
File //galaxy/galaxy-dist/lib/galaxy/security/__init__.py, line 939,
in derive_roles_from_access
in_roles = [ self.sa_session.query( self.model.Role ).get( x ) for x in
listify( kwd.get( k + '_in', [] ) ) ]
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 775, in get
return self._load_on_ident(key)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2512, in _load_on_ident
return q.one()
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2184, in one
ret = list(self)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2227, in __iter__
return self._execute_and_instances(context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2242, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1449, in execute
params)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1584, in _execute_clauseelement
compiled_sql, distilled_params
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1698, in _execute_context
context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1691, in _execute_context
context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py,
line 331, in do_execute
cursor.execute(statement, parameters)
DataError: (DataError) invalid input syntax for integer:
odoppelt__at__pasteur.fr
LINE 3: WHERE role.id = E'odoppelt__at__pasteur.fr'
^
'SELECT role.id AS role_id, role.create_time AS role_create_time,
role.update_time AS role_update_time, role.name AS role_name,
role.description AS role_description, role.type AS role_type, role.deleted
AS role_deleted \nFROM role \nWHERE role.id = %(param_1)s' {'param_1':
'odoppelt__at__pasteur.fr'}
_
Le 06/01/14 18:34, Olivia Doppelt a écrit :
Hello John,
Thank you very much for the patch. It works now with the argument 'roles'.
I tested with the wrong role in order to be sure that it filters when the
role is not the right one and it does get filtered :)
However, I need to add a few lines to the patch in order to avoid the use
Re: [galaxy-dev] BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Dear John,
Unfortunatly, I have another problem.
The galaxy version in use for our institute was last updated on December
10th.
Weirdly, when i put the patch in the library_contents.py file, I get
another error (which i don't have on my local version)
Actually, it never goes in the code i patched... and it doesn't
underdand the string which is the role.
the error in the following:
_
157.99.60.187 - - [07/Jan/2014:16:36:31 +0200] GET
/api/libraries/79449a84684b4136/contents?key=API_key HTTP/1.1 200 -
- python-requests/2.0.1 CPython/2.7.5+ Linux/3.11.0-12-generic
sqlalchemy.pool.QueuePool WARNING 2014-01-07 16:36:32,212 Error
closing cursor: current transaction is aborted, commands ignored until
end of transaction block
galaxy.web.framework ERROR 2014-01-07 16:36:32,212 Uncaught exception
in exposed API method:
Traceback (most recent call last):
File
//galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py, line
197, in decorator
rval = func( self, trans, *args, **kwargs)
File
//galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/library_contents.py,
line 183, in create
role = str( trans.security.decode_id( role ) )
File
//galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py,
line 843, in upload_library_dataset
trans.app.security_agent.derive_roles_from_access( trans,
library.id, cntrller, library=True, **vars )
File //galaxy/galaxy-dist/lib/galaxy/security/__init__.py,
line 939, in derive_roles_from_access
in_roles = [ self.sa_session.query( self.model.Role ).get( x ) for
x in listify( kwd.get( k + '_in', [] ) ) ]
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 775, in get
return self._load_on_ident(key)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2512, in _load_on_ident
return q.one()
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2184, in one
ret = list(self)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2227, in __iter__
return self._execute_and_instances(context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py,
line 2242, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1449, in execute
params)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1584, in _execute_clauseelement
compiled_sql, distilled_params
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1698, in _execute_context
context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py,
line 1691, in _execute_context
context)
File
//galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py,
line 331, in do_execute
cursor.execute(statement, parameters)
DataError: (DataError) invalid input syntax for integer:
odoppelt__at__pasteur.fr
LINE 3: WHERE role.id = E'odoppelt__at__pasteur.fr'
^
'SELECT role.id AS role_id, role.create_time AS role_create_time,
role.update_time AS role_update_time, role.name AS role_name,
role.description AS role_description, role.type AS role_type,
role.deleted AS role_deleted \nFROM role \nWHERE role.id =
%(param_1)s' {'param_1': 'odoppelt__at__pasteur.fr'}
_
Le 06/01/14 18:34, Olivia Doppelt a écrit :
Hello John,
Thank you very much for the patch. It works now with the argument 'roles'.
I tested with the wrong role in order to be sure that it filters when
the role is not the right one and it does get filtered :)
However, I need to add a few lines to the patch in order to avoid the
use of this method without a defined role.
basically, the complete patch is:
+roles = payload.get(roles, None)
+if roles:
+roles = util.listify(roles)
+
+def to_id_as_str(role):
+role = role.replace(__at__, @)
+if @ in role:
+role = str( trans.sa_session.query( trans.app.model.Role
).filter( trans.app.model.Role.table.c.name == role ).first().id )
+else:
+role = str( trans.security.decode_id( role ) )
+return role
+
+payload[roles] = map( to_id_as_str, roles )
+
+else:
+trans.response.status = 400
+return Missing required roles definition
I don't know if you (could/want to) add this to your commit.
Thank you very much for your help.
--
Olivia
Le 06/01
Re: [galaxy-dev] BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Hello again, I just reminded a capital thing concerning the api script library_upload_from_import_dir.py. Actually it only works for admin users. simple users don't have the authorization to upload anything in their galaxy library using this script. It is the main reason why we choose to user bioblend. -- Olivia Le 04/01/14 22:10, Olivia Doppelt a écrit : Le 03/01/14 15:44, John Chilton a écrit : I am not sure I understand the problem. Can you clarify a few points: Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing? Dear John, In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our user_library_import_dir and one in an output directory to enable the users to export their results from the instance. The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script) For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use. Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are uploadable. However, using bioblend function from libraries class upload_file_from_server, a user with its key can upload within its galaxy library, the data of anyone. #giOlivia.libraries.upload_file_from_server(OLIVIA_API_key,/xxx/links/otheru...@pasteur.fr/,F0045a9ffd77d1226) To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error. giOlivia.libraries.upload_file_from_server(OLIVIA_API_key,//links/oli...@pasteur.fr/, F0045a9ffd77d1226, roles='oli...@pasteur.fr') and I get the following error: Traceback (most recent call last): File input, line 1, in module File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py, line 242, in upload_file_from_server return self._do_upload(**vars) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py, line 190, in _do_upload files_attached=files_attached) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py, line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py, line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: html headtitleInternal Server Error/title/head body h1Internal Server Error/h1 pThe server has either erred or is incapable of performing the requested operation. br/ !-- --/p hr noshade div align=rightWSGI Server/div /body /html If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind? elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message the api does not produce any error when the api key corresponds to the user which tries to import its own data. Has your institute made modifications Galaxy to enforce this constraint - (it can't be possible that any user may upload any other user's data in their Galaxy Data library)? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API? I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key) I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work. Tell me if you need more information. I'll look further on monday but i'm kind of stuck. Thank you for your help, Cheers, -- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur On Fri, Jan 3, 2014 at 5:36
Re: [galaxy-dev] BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Le 03/01/14 15:44, John Chilton a écrit : I am not sure I understand the problem. Can you clarify a few points: Is bioblend letting users associate roles with data libraries that they shouldn't be able to and the operation is succeeding -or- are you trying to add roles and the operation is failing? Dear John, In our Galaxy instance, each user connecting to Galaxy has a library associated to its user role which is created automatically as well as two directories; one in the links directory corresponding to our user_library_import_dir and one in an output directory to enable the users to export their results from the instance. The user directories are readable by Galaxy and by themselves (unix rights are set using a root cron script) For a project, we need to do basic galaxy stuff using either the API scripts or bioblend api. We have tried both and bioblend is a little simpler to use. Using the api script library_upload_from_import_dir.py, using a user api key, only files from the user directory are uploadable. However, using bioblend function from libraries class upload_file_from_server, a user with its key can upload within its galaxy library, the data of anyone. #giOlivia.libraries.upload_file_from_server(OLIVIA_API_key,/xxx/links/otheru...@pasteur.fr/,F0045a9ffd77d1226) To disable this, i'm trying to add an associate role when I launch the upload_file_from_server command but I get an error. giOlivia.libraries.upload_file_from_server(OLIVIA_API_key,//links/oli...@pasteur.fr/, F0045a9ffd77d1226, roles='oli...@pasteur.fr') and I get the following error: Traceback (most recent call last): File input, line 1, in module File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py, line 242, in upload_file_from_server return self._do_upload(**vars) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/libraries/__init__.py, line 190, in _do_upload files_attached=files_attached) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxy/client.py, line 71, in _post r = self.gi.make_post_request(url, payload=payload, files_attached=files_attached) File /usr/local/lib/python2.7/dist-packages/bioblend-0.4.2_dev-py2.7.egg/bioblend/galaxyclient.py, line 94, in make_post_request r.status_code, body=r.text) # @see self.body for HTTP response body ConnectionError: Unexpected response from galaxy: 500: html headtitleInternal Server Error/title/head body h1Internal Server Error/h1 pThe server has either erred or is incapable of performing the requested operation. br/ !-- --/p hr noshade div align=rightWSGI Server/div /body /html If the API is producing an error message can you share that: in particular this code is probably resulting in an error or some kind? elif roles: # Check to see if the user selected roles to associate with the DATASET_ACCESS permission # on the dataset that would cause accessibility issues. vars = dict( DATASET_ACCESS_in=roles ) permissions, in_roles, error, message = \ trans.app.security_agent.derive_roles_from_access( trans, library.id, cntrller, library=True, **vars ) if error: if cntrller == 'api': return 400, message the api does not produce any error when the api key corresponds to the user which tries to import its own data. Has your institute made modifications Galaxy to enforce this constraint - (it can't be possible that any user may upload any other user's data in their Galaxy Data library)? Also is the thing you are trying to do something that is working through the web ui but not through the API, or have you only tried through the API? I did a few tests yesterday, and to resume, it works as it should using the api script but there is a problem using bioblend. I'm not sure that roles are really needed here, i only thought that it could guide galaxy (and bioblend) to not upload data that does not belong to user (corresponding to the API key) I looked at the sources, and my major doubt was that bioblend didn't pass all the arguments but it does, everything is transmitted to galaxy do it should work. Tell me if you need more information. I'll look further on monday but i'm kind of stuck. Thank you for your help, Cheers, -- Olivia Doppelt-Azeroual, PhD CIB - Institut Pasteur On Fri, Jan 3, 2014 at 5:36 AM, Olivia Doppelt olivia.dopp...@pasteur.fr wrote: Hello to all, I'm trying to use the function upload_file_from_server from the galaxy libraries api of Bioblend. It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's
[galaxy-dev] BioBlend AND Galaxy: Problem using upload_file_from_server function with an associated role
Hello to all, I'm trying to use the function /upload_file_from_server/ from the galaxy libraries api of Bioblend. It's working correctly when i don't associate any role to the arguments. However, in our architecture (Institut Pasteur, Paris) it can't be possible that any user may upload any other user's data in their Galaxy Data library. Thus, it is how /upload_file_from_server/ is working at the moment. As I understand the problem, it doesn't come from Bioblend itself, as all arguments (as well as roles) are well transmitted on the server side. could somebody help me with that ? Happy new Year !!! and Have a nice day, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] api script import_library_dataset_to_history.py problem
Hello to all,
I'm trying to use the script import_library_dataset_to_history.py in
order to upload a file from a dataset in a history.
I get an error, but I'm not sure that I use it the right way.
The command line I use is:
python import_library_dataset_to_history.py
cd065330048989af4ae266cf94685e6b http://127.0.0.1:8080
/api/libraries/f597429621d6eb2b/contents/f597429621d6eb2b
I'm not sur if the url (http://127.0.0.1:8080) is the one to use.
I'm not sure if the library_file_id is the right one.
At the url
http://127.0.0.1:8080/api/libraries/f597429621d6eb2b/contents/f597429621d6eb2b
The Json:
{
data_type: fasta,
date_uploaded: 2012-11-16T13:29:37.051910,
file_name:
/home/odoppelt/localStuff_VM/tools/galaxy-dist/database/files/000/dataset_2.dat,
file_size: 5492113,
folder_id: f597429621d6eb2b,
genome_build: ?,
id: f597429621d6eb2b,
ldda_id: f597429621d6eb2b,
message: ,
metadata_data_lines: 90357,
metadata_dbkey: ?,
metadata_sequences: 246,
misc_blurb: 246 sequences,
misc_info: uploaded fasta file,
model_class: LibraryDataset,
name: Galaxy64-[Fasta_file_containing_result_contigs].fasta,
template_data: {},
uploaded_by: odopp...@pasteur.fr,
uuid: null
}
could you please tell me if I use it the wrong way or if it doesn't work.
FYI,
the error i'm getting is:
Traceback (most recent call last):
File import_library_dataset_to_history.py, line 16, in module
submit( sys.argv[1], sys.argv[2], data )
File common.py, line 105, in submit
r = post( api_key, url, data )
File common.py, line 47, in post
return simplejson.loads( urllib2.urlopen( req ).read() )
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/__init__.py,
line 384, in loads
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py,
line 402, in decode
File
/home/odoppelt/localStuff_VM/tools/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py,
line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded:
line 1 column 0 (char 0)
Thanks in advance,
Cheers,
--
Olivia Doppelt-Azeroual, PhD
Tel: 92 15
CIB - Institut Pasteur
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] conditions within workflows
Dear developpers, I have a question concerning the insertion of conditions inside workflows ? Here, at the pasteur Institute, we would like to use galaxy to create a microbiome analysis workflow. However, it requires at least 10 different tools which may be or may not be used in several cases. For the moment, all I see is to create a giant wrapper for this workflow but it would be loosing the great interest of galaxy. I see on bitbucket that there is a branch on galaxy-central named galaxy-central-conditional-toolbox-elements are there new elements since february ? Best regards, -- Olivia Doppelt-Azeroual, PhD Tel: 92 15 CIB - Institut Pasteur ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] workflow multiple inputs
Dear developpers, Last week we made an update on the default branch of galaxy-dist. Some problems concerning the api were detected and resolved thanks to Dannon Baker by pulling the stable branch. However, a very interesting fonctionality was working well on this default branch, concerning the multiple inputs for workflows. There were two icons next to the input dataset window, a multiple file icon and a link icon to link paired input. Why is this functionnality not included in the stable branch ? isn't stable yet? It is a very useful though. Please keep me informed about that. Cheers, Olivia Doppelt-Azeroual Pasteur Institute, CIB, PARIS ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
