[galaxy-dev] Running additional services
We're running cloudman and I would like to run a couple of additional services alongside galaxy. Is there a elegant way/place to add/run these services to existing galaxy code/scripts? Cheers, Rob Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Creating multiple datasets in a libset
Hi Neil,
I've attached my class function for uploading multiple files.
def upload_files(self, fullpaths):
Uploads files from a disk location to a Galaxy library
Accepts an array of full path filenames
Example: fullpaths = ['/home/username/file1.txt',
'/home/username/files2.txt']
if self.jsonstring == None:
self.get_library()
library_id = self.library_id
library_folder_id = self.library_folder_id
api_key = self.api_key
api_url = self.api_url
#Galaxy needs to read the pathnames as a new line delimited string
#so we do that transformation here
fullpaths_string =
for path in fullpaths:
fullpaths_string = fullpaths_string + path + \n
fullpaths_string = fullpaths_string[:-1]
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
data['filesystem_paths'] = fullpaths_string
data['create_type'] = 'file'
#Start the upload. This will return right away, but it may take
awhile
libset = submit(api_key, api_url + libraries/%s/contents %
library_id, data, return_formatted = False)
#Iterate through each dataset we just uploaded and block until all
files have been written to the Galaxy database
for ds in libset:
last_filesize = 0
while True:
#If file_size != 0 and the file_size is different after a
second iteration, then we assume the disk write is finished
ds_id = ds['id']
uploaded_file = display(api_key, api_url +
'libraries/%s/contents/%s' %(library_id, ds_id), return_formatted=False)
print uploaded_file
if uploaded_file['file_size'] != 0 and
uploaded_file['file_size'] == last_filesize:
break
else:
last_filesize = uploaded_file['file_size']
time.sleep(2)
self.libset = libset
return libset
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Wed, May 29, 2013 at 12:45 AM, neil.burd...@csiro.au wrote:
Hi Guys,
Did you manage to get multiple datasets working? I can't seem to
upload multiple files. Only the last file appears in the history. I changed
my code as mentioned in the thread below in example_watch_folder.py to
add multiple files separated by a new line and increased the sleep time:
for fname in os.listdir(in_folder):
fullpath = os.path.join(in_folder, fname)
print ' fullpath is [%s] ' % fullpath
if os.path.isfile(fullpath):
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
data['filesystem_paths'] =
/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz\n
/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz
print ' data is [%s] ' % str(data['filesystem_paths'])
data['create_type'] = 'file'
libset = submit(api_key, api_url + libraries/%s/contents
% library_id, data, return_formatted = False)
#TODO Handle this better, but the datatype isn't always
# set for the followup workflow execution without this
# pause.
time.sleep(65)
However, I get the following crash:
./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c
http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input
/home/galaxy/galaxy-drop/output This One f2db41e1fa331b3e
fullpath is
[/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz]
data is [/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz
/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz]
url is :
http://barium-rbh/csiro/api/libraries/33b43b4e7093c91f/contents?key=64f3209856a3cf4f2d034a1ad5bf851c
data is : {'file_type': 'auto', 'dbkey': '', 'create_type': 'file',
'folder_id': 'F33b43b4e7093c91f', 'upload_option': 'upload_paths',
'filesystem_paths':
'/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz\n
/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz'}
url is :
http://barium-rbh/csiro/api/workflows?key=64f3209856a3cf4f2d034a1ad5bf851c
data is : {'workflow_id': 'f2db41e1fa331b3e', 'ds_map': {'14': {'src':
'ld', 'id': 'ff5476bcf6c921fa'}}, 'history':
'141_S_0851_MRI_T2_Screening.nii.gz - apiFullCTE'}
{'outputs': ['daecbdd824e1c349', '358eb58cd5463e0d', 'c0279aab05812500'],
'history': '3cc0effd29705aa3'}
url is :
http://barium-rbh/csiro/api/workflows?key
[galaxy-dev] Cloudman / galaxyIndicies
We'd like to trim the the galaxyIndicies/genome directory to human and blastdb. As we may have several CM instances, we would like to share this filesystem across all instances. I would like to simply copy these data to a new volume and mount this filesystem with all our CM instances. Do I just need to rename specified filesystem in pd.yaml to ourGalaxyIndicies and then mount our CMs to /mnt/ourGalaxyIndicies/, or are their other locations I need to modify as well? Are the size and snap_id required? Cheers! Rob Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Creating multiple datasets in a libset
Hi Dannon,
Thanks for the response. Sorry to be pedantic, but just to make sure that I
understand the interpretation of this field on the other side of the API, I
would need to have something like the following:
data['filesystem_paths'] = /home/me/file1.vcf \n /home/me/file2.vcf /n
/home/me/file3.vcf
I assume I should also increase the time.sleep() to reflect the uploading
of extra files?
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 9:15 AM, Dannon Baker dannon.ba...@gmail.comwrote:
Hey Rob,
That example_watch_folder.py does just submit exactly one at a time,
executes the workflow, and then does the next all in separate transactions.
If you wanted to upload multiple filepaths at once, you'd just append more
to the ''filesystem_paths' field (newline separated paths).
-Dannon
On Fri, Apr 26, 2013 at 11:54 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
I'm looking at example_watch_folder.py and it's not clear from the
example how you submit multiple datasets to a library. In the example, the
first submit returns a libset [] with only a single entry and then proceeds
to iterate through each dataset in the libset in the following section:
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
*data['filesystem_paths'] = fullpath*
data['create_type'] = 'file'
libset = submit(api_key, api_url + libraries/%s/contents %
library_id, data, return_formatted = False)
time.sleep(5)
for ds in libset:
if 'id' in ds:
wf_data = {}
wf_data['workflow_id'] = workflow['id']
wf_data['history'] = %s - %s % (fname,
workflow['name'])
wf_data['ds_map'] = {}
for step_id, ds_in in workflow['inputs'
].iteritems():
wf_data['ds_map'][step_id] = {'src':'ld',
'id':ds['id']}
res = submit( api_key, api_url + 'workflows',
wf_data, return_formatted=False)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Creating multiple datasets in a libset
Hi Dannon,
I've written some code to (i) query a dataset to ensure that it's been
uploaded after a submit and (ii) to ensure a resulting dataset has been
written to the file.
*#Block until all datasets have been uploaded*
libset = submit(api_key, api_url + libraries/%s/contents % library_id,
data, return_formatted = False)
for ds in libset:
while True:
uploaded_file = display(api_key, api_url +
'libraries/%s/contents/%s' %(library_id, ds['id']), return_formatted=False)
if uploaded_file['misc_info'] == None:
time.sleep(1)
else:
break
*#Block until all result datasets have been saved to the filesystem*
result_ds_url = api_url + 'histories/' + history_id + '/contents/' +
dsh['id'];
while True:
result_ds = display(api_key, result_ds_url, return_formatted=False)
if result_ds[state] == 'ok':
break
else:
time.sleep(1)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 11:18 AM, Dannon Baker dannon.ba...@gmail.comwrote:
Yep, that example filesystem_paths you suggest should work fine. The
sleep() bit was a complete hack from the start, for simplicity in
demonstrating a very basic pipeline, but what you probably want to do for a
real implementation is query the dataset in question via the API, verify
that the datatype/etc have been set, and only after that execute the
workflow; instead of relying on sleep.
On Mon, Apr 29, 2013 at 9:24 AM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon,
Thanks for the response. Sorry to be pedantic, but just to make sure that
I understand the interpretation of this field on the other side of the API,
I would need to have something like the following:
data['filesystem_paths'] = /home/me/file1.vcf \n /home/me/file2.vcf /n
/home/me/file3.vcf
I assume I should also increase the time.sleep() to reflect the uploading
of extra files?
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 9:15 AM, Dannon Baker dannon.ba...@gmail.comwrote:
Hey Rob,
That example_watch_folder.py does just submit exactly one at a time,
executes the workflow, and then does the next all in separate transactions.
If you wanted to upload multiple filepaths at once, you'd just append more
to the ''filesystem_paths' field (newline separated paths).
-Dannon
On Fri, Apr 26, 2013 at 11:54 PM, Rob Leclerc
robert.lecl...@gmail.comwrote:
I'm looking at example_watch_folder.py and it's not clear from the
example how you submit multiple datasets to a library. In the example, the
first submit returns a libset [] with only a single entry and then proceeds
to iterate through each dataset in the libset in the following section:
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
*data['filesystem_paths'] = fullpath*
data['create_type'] = 'file'
libset = submit(api_key, api_url + libraries/%s/contents %
library_id, data, return_formatted = False)
time.sleep(5)
for ds in libset:
if 'id' in ds:
wf_data = {}
wf_data['workflow_id'] = workflow['id']
wf_data['history'] = %s - %s % (fname,
workflow['name'])
wf_data['ds_map'] = {}
for step_id, ds_in in workflow['inputs'
].iteritems():
wf_data['ds_map'][step_id] = {'src':'ld',
'id':ds['id']}
res = submit( api_key, api_url + 'workflows',
wf_data, return_formatted=False)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Creating multiple datasets in a libset
Correction. The above were not reliable methods to ensure the file was
copied into the data library. Checking for file_size != 0 was also not
effective for large files.
Dannon, can you tell me which field we should query and what state/message
which will allow us to avoid race conditions?
The only solution that I can see that when the file_size != 0, you then
ensure it that the file_size has not changed after a a short delay.
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 4:11 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon,
I've written some code to (i) query a dataset to ensure that it's been
uploaded after a submit and (ii) to ensure a resulting dataset has been
written to the file.
*#Block until all datasets have been uploaded*
libset = submit(api_key, api_url + libraries/%s/contents % library_id,
data, return_formatted = False)
for ds in libset:
while True:
uploaded_file = display(api_key, api_url +
'libraries/%s/contents/%s' %(library_id, ds['id']), return_formatted=False)
if uploaded_file['misc_info'] == None:
time.sleep(1)
else:
break
*#Block until all result datasets have been saved to the filesystem*
result_ds_url = api_url + 'histories/' + history_id + '/contents/' +
dsh['id'];
while True:
result_ds = display(api_key, result_ds_url, return_formatted=False)
if result_ds[state] == 'ok':
break
else:
time.sleep(1)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 11:18 AM, Dannon Baker dannon.ba...@gmail.comwrote:
Yep, that example filesystem_paths you suggest should work fine. The
sleep() bit was a complete hack from the start, for simplicity in
demonstrating a very basic pipeline, but what you probably want to do for a
real implementation is query the dataset in question via the API, verify
that the datatype/etc have been set, and only after that execute the
workflow; instead of relying on sleep.
On Mon, Apr 29, 2013 at 9:24 AM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon,
Thanks for the response. Sorry to be pedantic, but just to make sure
that I understand the interpretation of this field on the other side of the
API, I would need to have something like the following:
data['filesystem_paths'] = /home/me/file1.vcf \n /home/me/file2.vcf /n
/home/me/file3.vcf
I assume I should also increase the time.sleep() to reflect the
uploading of extra files?
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Mon, Apr 29, 2013 at 9:15 AM, Dannon Baker dannon.ba...@gmail.comwrote:
Hey Rob,
That example_watch_folder.py does just submit exactly one at a time,
executes the workflow, and then does the next all in separate transactions.
If you wanted to upload multiple filepaths at once, you'd just append more
to the ''filesystem_paths' field (newline separated paths).
-Dannon
On Fri, Apr 26, 2013 at 11:54 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm looking at example_watch_folder.py and it's not clear from the
example how you submit multiple datasets to a library. In the example, the
first submit returns a libset [] with only a single entry and then
proceeds
to iterate through each dataset in the libset in the following section:
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
*data['filesystem_paths'] = fullpath*
data['create_type'] = 'file'
libset = submit(api_key, api_url + libraries/%s/contents %
library_id, data, return_formatted = False)
time.sleep(5)
for ds in libset:
if 'id' in ds:
wf_data = {}
wf_data['workflow_id'] = workflow['id']
wf_data['history'] = %s - %s % (fname,
workflow['name'])
wf_data['ds_map'] = {}
for step_id, ds_in in workflow['inputs'
].iteritems():
wf_data['ds_map'][step_id] = {'src':'ld',
'id':ds['id']}
res = submit( api_key, api_url + 'workflows',
wf_data, return_formatted=False)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
___
Please keep all
[galaxy-dev] Creating multiple datasets in a libset
I'm looking at example_watch_folder.py and it's not clear from the example
how you submit multiple datasets to a library. In the example, the first
submit returns a libset [] with only a single entry and then proceeds to
iterate through each dataset in the libset in the following section:
data = {}
data['folder_id'] = library_folder_id
data['file_type'] = 'auto'
data['dbkey'] = ''
data['upload_option'] = 'upload_paths'
*data['filesystem_paths'] = fullpath*
data['create_type'] = 'file'
libset = submit(api_key, api_url + libraries/%s/contents % library_id,
data, return_formatted = False)
time.sleep(5)
for ds in libset:
if 'id' in ds:
wf_data = {}
wf_data['workflow_id'] = workflow['id']
wf_data['history'] = %s - %s % (fname, workflow[
'name'])
wf_data['ds_map'] = {}
for step_id, ds_in in workflow['inputs'
].iteritems():
wf_data['ds_map'][step_id] = {'src':'ld', 'id'
:ds['id']}
res = submit( api_key, api_url + 'workflows',
wf_data, return_formatted=False)
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] directory copies as zero-length file from job_working_directory?
Hi Nikhil, I've encountered this issue as well when using the API. Some scripts were using a 5 second sleep default to allow for copying the file to the output directory, but I found that I needed up to 15 seconds (occasionally that's not even enough). Cheers, Rob Rob Leclerc, PhD P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu On Mar 28, 2013, at 10:06 PM, Nikhil Joshi najo...@ucdavis.edu wrote: Hi all, So I have a tool that I am writing a wrapper script for that produces a directory of images as one of its outputs. This directory gets created just fine in the job_working_directory, but then when the job finishes and the contents of the job_working_directory are copied, the copy of the images directory is a zero-length *file*. I seem to recall having this problem before and finding a solution to it, but now I can't seem to find it. Any ideas? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] snpEff on Galaxy
Does anyone have any experience running snpEff on Galaxy? We installed snpEff from the toolshed and now have the latest changeset (c052639fa666). We modified the config files to correct the pathnames but now we're running into a series of errors when we try to run '*Compute SNP effect*' with a file using the hg19 dataset *The first error was:* java.lang.RuntimeException: Property: 'hg37.genome' not found at ca.mcgill.mcb.pcingola.interval.Genome.init(Genome.java:78) at ca.mcgill.mcb.pcingola.snpEffect.Config.readGenomeConfig(Config.java:334) at ca.mcgill.mcb.pcingola.snpEffect.Config.read(Config.java:301) at ca.mcgill.mcb.pcingola.snpEffect.Config.init(Config.java:67) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:582) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77 This first error was confusing because the input file was for hg19, so it's not clear why there was a reference to hg37. Next, we then edited snpEff.config and added the line *hg37.genome : Homo_sapiens* * * with the other hg37.XX : Homo_sapiens keypairs. *This led to the second error:* ERROR: Cannot read file '/home/galaxy//snpEff/data//hg37/snpEffectPredictor.bin'. You can try to download the database by running the following command: java -jar snpEff.jar download hg37 java.lang.RuntimeException: java.io.FileNotFoundException: /home/galaxy/snpEff/data/hg37/snpEffectPredictor.bin (No such file or directory) at ca.mcgill.mcb.pcingola.snpEffect.SnpEffectPredictor.load(SnpEffectPredictor.java:58) at ca.mcgill.mcb.pcingola.snpEffect.Config.loadSnpEffectPredictor(Config.java:240) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:595) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77) Caused by: java.io.FileNotFoundException: /home/galaxy/snpEff/data/hg37/snpEffectPredictor.bin (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:137) at java.io.FileInputStream.init(FileInputStream.java:96) at ca.mcgill.mcb.pcingola.util.Gpr.readFileSerializedGzThrow(Gpr.java:468) at ca.mcgill.mcb.pcingola.snpEffect.SnpEffectPredictor.load(SnpEffectPredictor.java:55) ... 4 more The problem now is that is is looking for the hg37 directory, which doesn't exist but GRCh37.65 does exist. So we then change the directory name to hg37 and reran, resulting in the third error: VcfFileIterator.parseVcfLine(276): Fatal error reading file '/mnt/galaxyData/files/000/dataset_432.dat' (line: 1): 1 2338204 2338204 G/T + java.lang.RuntimeException: java.lang.RuntimeException: WARNING: Unkown IUB code for SNP '+' at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.parseVcfLine(VcfFileIterator.java:277) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.readNext(VcfFileIterator.java:331) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.readNext(VcfFileIterator.java:1) at ca.mcgill.mcb.pcingola.fileIterator.FileIterator.hasNext(FileIterator.java:69) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.iterateVcf(SnpEffCmdEff.java:215) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.runAnalysis(SnpEffCmdEff.java:698) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:648) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77) Caused by: java.lang.RuntimeException: WARNING: Unkown IUB code for SNP '+' at ca.mcgill.mcb.pcingola.vcf.VcfEntry.parseAlts(VcfEntry.java:530) at ca.mcgill.mcb.pcingola.vcf.VcfEntry.parse(VcfEntry.java:445) at ca.mcgill.mcb.pcingola.vcf.VcfEntry.init(VcfEntry.java:66) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.parseVcfLine(VcfFileIterator.java:274) ... 8 more I'm going to assume that this is all related to the first issue, but I'm not sure how to fix the issue. Cheers! Rob Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Setting up SSL on CloudMan EC2
Has anyone configured SSL for nginx on CloudMan (AWS)? If so, can you share
your nginx.conf file as I'm having some issues.
*Details*
I tried adopting the nginx.conf for CloudMan+CloudBioLiniux as described
herehttps://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readthedocs.org/en/latest/
but
ran into a problem with the line (in red):
server {
* listen 443 default_server ssl;*
ssl_certificate /usr/nginx/conf/cert;
ssl_certificate_key /usr/nginx/conf/key;
}
According to *nginx.conf.default* the configuration for HTTPS looks like:
*server {*
*listen 443;*
*server_name localhost;*
*...*
*}*
This suggested that the documentation was incorrect, and that I needed to
insert a semicolon after '*listen 443*'.
*[emerg]: unknown directive default_server in
/opt/galaxy/pkg/nginx/conf/nginx.conf:123*
*
*
*
*Returning again to nginx.conf.default I 'replaced default_server ssl' with:
*server {*
*listen 443;*
*server_name localhost;*
*...*
*}
*
After restarting the galaxy server, I hit out server and then directed to *
http://localhost*
On the one hand, I'm not surprised that I am directed to localhost, however
I see the same usage for describing the service for port 80.
Nevertheless, in the CloudMan+CloudBioLinux docs following section was to
be added to the port 80 section.
server {
listen 80;
server_namemygalaxy.example.com;
rewrite^ https://$server_name$request_uri? permanent;
}
so I changed $server_name - $host. Redirection worked, but there was no
response on the other end. There are several avenues I could take from
here. Any suggestions?
-Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Production Servers - Best Practices
I need a comprehensive understanding of the Best Practices for a Galaxy Production Server. (i) I've read Running Galaxy in a Production Environment and associated wikis (ii) I saw the presentations Building Scalable Galaxy (2010), Deploying Galaxy on the Cloud (2010), and Cloudman: Galaxy on the Cloud. (iii) I've searched the mailing list. Are there any other *undocumented pitfalls/tips *that we should be aware of for a Cloudman/Amazon install? Security would be a high priority for this list. Below are a list of references I've compiled for my own list: Running Galaxy in a Production Environmenthttp://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer Sildes: Building Scalable Galaxy 2010http://wiki.galaxyproject.org/Events/GDC2010?action=AttachFiledo=viewtarget=GDC2010_building_scalable.pdf Deploying Galaxy on the Cloud, 2010https://bytebucket.org/galaxy/galaxy-central/wiki/DevConf2010/galaxy_devconf_2010_cloud.pdf Configure Apache Proxy to Galaxyhttp://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy Configure nginx Proxy to Galaxyhttp://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy Running Tests http://wiki.galaxyproject.org/Admin/Running%20Tests Presentation: CloudMan: Galaxy on the Cloud, 2011http://vimeo.com/24870202 Cloudman Capacity Planning for AWShttp://wiki.galaxyproject.org/CloudMan/AWS/CapacityPlanning Using Amazon EBS Volumes http://aws.amazon.com/ebs/ Copying an Amazon EBS Snapshothttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-copy-snapshot.html ** Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy - FTP Download Problem
We have a CloudMan Galaxy install on Amazon and I'm able to upload FTP files from FireZilla (or programatically through a Java service I wrote). However, I'm unable to download that same file once uploaded. The 550 response is a Requested action not take. File unavailable (e.g., file not found, no access). I have rwx permissions on the directory and rw permissions on the file.* * * * *Status:* Starting download of /TAF1_ChIP-2.txt *Command: *TYPE A *Response:* 200 Type set to A *Command: *PASV *Response: *227 Entering Passive Mode (10,40,11,236,117,50). *Status: *Server sent passive reply with unroutable address. Using server address instead. *Command: *RETR TAF1_ChIP-2.txt *Response: *550 TAF1_ChIP-2.txt: Operation not permitted *Error: Critical file transfer error* Thoughts? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Trouble Shooting example_watch_folder.py
There were several issues and bugs I encountered getting this script to
work. I've written down some notes for a quickstart for anyone else who
might be following in my footsteps.
1. Create input and output directores where you will drop and emit files.
In my example I used:
/Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output
2. Edit the galaxy-dist configuration script (universe_wsgi.ini) and set:
allow_library_path_paste = true
3. In example_watch_folder.py, comment out line 30 and replace it with:
#library_id = library[0]['id] This causes a key error : 0 -- remove
the indexing
library_id = library['id']
4. In example_watch_folder.py change the sleep duration on line 59 to 15
seconds. If you're still getting zero byte .dat files, then you'll need to
extend the duration even more.
time.sleep(15)
5. CREATE A SAMPLE Workflow Dataset:
1. Start Galaxy
2. Go to USCS and upload the Exons from chr22 to your clean history file
3. Use the Text Manipulation tool to select the first 5 lines of the
gene exon file
4. Extract the workflow
5. IMPORTANT: If you don't do the following step you'll get a 500
Internal Server.
6. Change the output filename in the workflow and then save the
workflow.
7. Test that the workflow is accessable by visiting
'http://galaxy_server/api/workflows'
in your browser, and then visit 'http://
galaxy_server/api/workflows/workflow_id
8. Download the exons file to local
6. In your browser go to: http://localhost:8080/api/workflows and identify
the id of the workflow you've just created. Let's assume it is
(4b187121143038ff) and that your api_key for admin is api_key
7. Run the script:
./example_watch_folder.py api_key http://localhost:8080/api/
/Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output My API Import 4b187121143038ff
8. Copy the exons file to /Users/Rob/Documents/galaxy-drop/input when the
process is complete the file will be placed in
/Users/Rob/Documents/galaxy-drop/output
9. In your browser go to: http://localhost:8080/api/histories locate the
recently created history and then navigate to:
http://localhost:8080/api/histories/history_id/contents
where history_id is the id of the history that was created when you ran
the dataset on the last workflow.
This is my result. The second file is the output file.
[
{
id: 0d16186aaff7cbfd,
name: Galaxy1-[Exons_chr22].bed,
type: file,
url: /api/histories/2d9035b3fc152403/contents/0d16186aaff7cbfd
},
{
id: 81991b32733a4fc4,
name: top200,
type: file,
url: /api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
]
10. You can navigate to the JSON description of the file in your browser:
http://localhost:8080/api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
and the output file can be downloaded here:
http:localhost:8080/datasets/81991b32733a4fc4/display?to_ext=data
Cheers!
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
Hi Dannon,
Thanks. I've moved to a full size monitor, apparently my eyes are going!
They *key error: 0* seems to be the result of a bug in the code.
print(library)
results in:
{'url': '/api/libraries/4b187121143038ff', 'id': '4b187121143038ff',
'name': 'API Import11'}
So library[0]['id'] doesn't point to anything, resulting in the key error.
I changed the code on line 30
library_id = library['id']
This seems to fix the issue. I've now got to the section of code where I am
watching the in_folder, but there doesn't seem to be any subsequent
activity. I'll post my findings or questions as I make progress or lack
thereof.
Cheers,
Rob
On Wed, Mar 13, 2013 at 7:35 AM, Dannon Baker dannon.ba...@gmail.comwrote:
It should be set on line 29 as the result of that submit function. Can
you add (between 29 and 30) a 'print library' to see what that call
actually results in?
Off the top of my head, this might be happening if your user account (that
the API key is associated with) isn't an admin account -- only admins can
create Data Libraries.
On Tue, Mar 12, 2013 at 6:35 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to
add the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the
API Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow )
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if it
doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries', lib_create_data,
return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0]['id']' is set or is in scope, since this seems to be the item in
the iterator above this section.
Cheers,
Rob
On Tue, Mar 12, 2013 at 6:14 PM, Dannon Baker dannon.ba...@gmail.comwrote:
The second argument there should point to the url of Galaxy's API --
in this case http://localhost:8080/api, I'm guessing.
-Dannon
On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm trying to get the example_watch_folder.py to work with a simple
example,
but I am running into an error (Caveat, python is not my native
language)
I call the following script, but get the subsequent error:
$ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
URL did not return JSON data
The preconditions I have in place are:
1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f
2. I created a new history on http://localhost:8080
3. I downloaded the the exons from hg19/chr22 from UCSC
3. On the dataset I performed a Text Manipulation Select first (20
lines)
4. I extracted a workflow from the history and recorded the workflow id
(2a56795cad3c7db3).
5. I created directories:
/Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/
My args may be malformed. Can someone provide me an example of a
theirs?
Cheers,
Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
I was able to get the script to work. I've written down some notes for a
quickstart for anyone else who might be following in my footsteps.
1. Create input and output directores where you will drop and emit files.
In my example I used:
/Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output
2. Edit the galaxy-dist configuration script (universe_wsgi.ini) and set:
allow_library_path_paste = true
3. In example_watch_folder.py, comment out line 30 and replace it with:
#library_id = library[0]['id] This causes a key error : 0
library_id = library['id']
4. CREATE A SAMPLE Workflow Dataset:
1. Start Galaxy
2. Go to USCS and upload the Exons from chr22 to your clean history file
3. Use the Text Manipulation tool to select the first 5 lines of the
gene exon file
4. Extract the workflow
5. *RESTART Galaxy* (You'll get a 500 Internal Server Error if you try
to run this without restarting)
6. Download the exons file to local
5. In your browser go to: http://localhost:8080/api/workflows and identify
the id of the workflow you've just created. Let's assume it is
(4b187121143038ff) and that your api_key for admin is
(8c25bc83f6f9e4001dd21eb7b64f063f)
6. Run the script:
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output My API Import 4b187121143038ff
7. Copy the exons file to /Users/Rob/Documents/galaxy-drop/input when the
process is complete the file will be placed in
/Users/Rob/Documents/galaxy-drop/output
8. In your browser go to: http://localhost:8080/api/histories locate the
recently created history and then navigate to:
http://localhost:8080/api/histories/history_id/contents
where history_id is the id of the history that was created when you ran
the dataset on the last workflow.
This is my result. The second file is the output file.
[
{
id: 0d16186aaff7cbfd,
name: Galaxy1-[Exons_chr22].bed,
type: file,
url: /api/histories/2d9035b3fc152403/contents/0d16186aaff7cbfd
},
{
id: 81991b32733a4fc4,
name: top200,
type: file,
url: /api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
]
9. You can navigate to the JSON description of the file in your browser:
http://localhost:8080/api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
and the output file can be downloaded here:
http:localhost:8080/datasets/81991b32733a4fc4/display?to_ext=data
Cheers!
Rob
On Wed, Mar 13, 2013 at 12:15 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon,
Thanks. I've moved to a full size monitor, apparently my eyes are going!
They *key error: 0* seems to be the result of a bug in the code.
print(library)
results in:
{'url': '/api/libraries/4b187121143038ff', 'id': '4b187121143038ff',
'name': 'API Import11'}
So library[0]['id'] doesn't point to anything, resulting in the key error.
I changed the code on line 30
library_id = library['id']
This seems to fix the issue. I've now got to the section of code where I
am watching the in_folder, but there doesn't seem to be any subsequent
activity. I'll post my findings or questions as I make progress or lack
thereof.
Cheers,
Rob
On Wed, Mar 13, 2013 at 7:35 AM, Dannon Baker dannon.ba...@gmail.comwrote:
It should be set on line 29 as the result of that submit function. Can
you add (between 29 and 30) a 'print library' to see what that call
actually results in?
Off the top of my head, this might be happening if your user account
(that the API key is associated with) isn't an admin account -- only admins
can create Data Libraries.
On Tue, Mar 12, 2013 at 6:35 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to
add the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the
API Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow
)
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if
it doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries',
lib_create_data, return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0
[galaxy-dev] Internal Server Error - Creating Workflows
If you create a Workflow and then try to access it through the API, https://main.g2.bx.psu.edu/api/workflows/workflow_id the server will emit a 500 Internal Server Error. Changing the output filename on the workflow and then saving seems to fix the issue. Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] example_watch_folder.py
If you're having problems with example_watch_folder.py you may need to increase the delay hack on line 55 to 15-20 seconds. Without the increased delay, it would upload a zerobyte file for me. It's not clear why, but there is a delay between the creation of a file name and the setting of the file to the dataset. Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] example_watch_folder.py : URL did not return JSON data
I'm trying to get the example_watch_folder.py to work with a simple example, but I am running into an error (Caveat, python is not my native language) I call the following script, but get the subsequent error: $ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/ /Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3 * URL did not return JSON data* The preconditions I have in place are: 1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f 2. I created a new history on http://localhost:8080 3. I downloaded the the exons from hg19/chr22 from UCSC 3. On the dataset I performed a Text Manipulation Select first (20 lines) 4. I extracted a workflow from the history and recorded the workflow id (2a56795cad3c7db3). 5. I created directories: - /Users/Rob/Documents/galaxy-db/input/ - /Users/Rob/Documents/galaxy-db/output/ My args may be malformed. Can someone provide me an example of a theirs? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to add
the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the API
Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow )
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if it
doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries',
lib_create_data, return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0]['id']' is set or is in scope, since this seems to be the
item in the iterator above this section.
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Tue, Mar 12, 2013 at 6:14 PM, Dannon Baker dannon.ba...@gmail.comwrote:
The second argument there should point to the url of Galaxy's API --
in this case http://localhost:8080/api, I'm guessing.
-Dannon
On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm trying to get the example_watch_folder.py to work with a simple
example,
but I am running into an error (Caveat, python is not my native language)
I call the following script, but get the subsequent error:
$ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
URL did not return JSON data
The preconditions I have in place are:
1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f
2. I created a new history on http://localhost:8080
3. I downloaded the the exons from hg19/chr22 from UCSC
3. On the dataset I performed a Text Manipulation Select first (20
lines)
4. I extracted a workflow from the history and recorded the workflow id
(2a56795cad3c7db3).
5. I created directories:
/Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/
My args may be malformed. Can someone provide me an example of a theirs?
Cheers,
Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] API - Status 403 Forbidden
I'm trying to use the blend4j to access the Galaxy API, but I'm getting a 403 error, however when I pass the same request string through my browser I get a json response (Status 200). //Here is the URL string being passed in both cases: http://localhost:8080/api/histories?key=c5954f0e726e52da297cdad5ad4aa249 //Below are the Galaxy logs emitted on my localhost: 127.0.0.1 - - [07/Mar/2013:21:12:01 -0400] GET /api/histories?key=c5954f0e726e52da297cdad5ad4aa249 HTTP/1.1 200 - - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.155 Safari/537.22 127.0.0.1 - - [07/Mar/2013:21:12:24 -0400] GET /api/histories?key=c5954f0e726e52da297cdad5ad4aa249 HTTP/1.1 403 - - Java/1.6.0_41 I'm not sure if this is related or not, because I can logout in my browser and still hit the local server without getting a 403, but how do I login (and logout) through the API to set up a session because I assume that the API key is only valid for the logged in user? Is there an API request for the login? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] BioBlend: Problem Running Example file
I had trouble running blend4j, so I tried to jump into python (a language I have little experience with). I tried running the example run_import_workflow.py *% python run_imported_workflow.py http://localhost:80808c25bc83f6f9e4001dd21eb7b64f063f * but I get an error: Initiating Galaxy connection Importing workflow Creating data library 'Imported data for API demo' Traceback (most recent call last): File run_imported_workflow.py, line 53, in module library_dict = gi.libraries.create_library(library_name) File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/libraries/__init__.py, line 27, in create_library File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/client.py, line 53, in _post File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/__init__.py, line 132, in make_post_request File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-1.1.0-py2.7.egg/requests/models.py, line 604, in json return json.loads(self.text or self.content) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/__init__.py, line 454, in loads return _default_decoder.decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 374, in decode obj, end = self.raw_decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 393, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Is there anything I am missing from the stock configuration which would cause this not to run out of the box? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] API - Status 403 Forbidden
Hi John, Thanks! I restarted the server, regenerated the API key, and can confirm that it's now working. As you mentioned, it's possible that I was missing a value on the end of the API key and was caught by the bug which allowed me to check find histories when I was logged out. Cheers! Rob On Fri, Mar 8, 2013 at 9:33 AM, John Chilton chil0...@umn.edu wrote: I can confirm this is a Galaxy bug. Steps to reproduce: - Log into clean galaxy. - Create an API key. - Logout - Got to localhost:8080/api/histories You should get a 403 and don't. This doesn't explain why the API isn't working for you, my best guess would be the key is off by a character or you created a new one and overwrote that one maybe. I think your instinct to try to diagnose this outside of blend4j is a good one, if you can get the API working without a browser session (perhaps from wget or curl) then I am confident the blend4j part will work (or at least we can get it to). -John On Thu, Mar 7, 2013 at 9:09 PM, Rob Leclerc robert.lecl...@gmail.com wrote: I'm trying to use the blend4j to access the Galaxy API, but I'm getting a 403 error, however when I pass the same request string through my browser I get a json response (Status 200). //Here is the URL string being passed in both cases: http://localhost:8080/api/histories?key=c5954f0e726e52da297cdad5ad4aa249 //Below are the Galaxy logs emitted on my localhost: 127.0.0.1 - - [07/Mar/2013:21:12:01 -0400] GET /api/histories?key=c5954f0e726e52da297cdad5ad4aa249 HTTP/1.1 200 - - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.155 Safari/537.22 127.0.0.1 - - [07/Mar/2013:21:12:24 -0400] GET /api/histories?key=c5954f0e726e52da297cdad5ad4aa249 HTTP/1.1 403 - - Java/1.6.0_41 I'm not sure if this is related or not, because I can logout in my browser and still hit the local server without getting a 403, but how do I login (and logout) through the API to set up a session because I assume that the API key is only valid for the logged in user? Is there an API request for the login? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] BioBlend: Problem Running Example file
I'm have the same error when I run library_create_library.py with the documentation laid out in the galaxy-central/tip//scrips/api/README from: /galaxy-dist/scripts/api/ *% ./library_create_library.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/libraries api_test 'API Test Library'* Traceback (most recent call last): File ./library_create_library.py, line 19, in module submit( sys.argv[1], sys.argv[2], data ) File ./common.py, line 100, in submit r = post( api_key, url, data ) File ./common.py, line 42, in post return simplejson.loads( urllib2.urlopen( req ).read() ) File build/bdist.macosx-10.6-intel/egg/simplejson/__init__.py, line 384, in loads File build/bdist.macosx-10.6-intel/egg/simplejson/decoder.py, line 402, in decode File build/bdist.macosx-10.6-intel/egg/simplejson/decoder.py, line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) -Rob On Fri, Mar 8, 2013 at 11:23 AM, Sytchev, Ilya isytc...@hsph.harvard.eduwrote: JSONDecodeError will be raised if the given JSON document is not valid. http://simplejson.readthedocs.org/en/latest/ Ilya From: Rob Leclerc robert.lecl...@gmail.com Date: Friday, March 8, 2013 12:01 AM To: galaxy-...@bx.psu.edu galaxy-...@bx.psu.edu Subject: [galaxy-dev] BioBlend: Problem Running Example file I had trouble running blend4j, so I tried to jump into python (a language I have little experience with). I tried running the example run_import_workflow.py *% python run_imported_workflow.py http://localhost:80808c25bc83f6f9e4001dd21eb7b64f063f * but I get an error: Initiating Galaxy connection Importing workflow Creating data library 'Imported data for API demo' Traceback (most recent call last): File run_imported_workflow.py, line 53, in module library_dict = gi.libraries.create_library(library_name) File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/libraries/__init__.py, line 27, in create_library File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/client.py, line 53, in _post File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/__init__.py, line 132, in make_post_request File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-1.1.0-py2.7.egg/requests/models.py, line 604, in json return json.loads(self.text or self.content) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/__init__.py, line 454, in loads return _default_decoder.decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 374, in decode obj, end = self.raw_decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 393, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Is there anything I am missing from the stock configuration which would cause this not to run out of the box? Cheers, Rob -- Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] BioBlend: Problem Running Example file
Dannon, *Thank you* you're absolutely right! ...Now I just have to deal with the permission denied errors. (The I set the galaxy admin to my userID/email). Cheers! Rob Rob Leclerc, PhD http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu On Fri, Mar 8, 2013 at 3:46 PM, Dannon Baker dannon.ba...@gmail.com wrote: Rob, The request below library_create_library looks like it has the wrong url, it should be: ./library_create_library.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/api/libraries api_test 'API Test Library' -Dannon On Fri, Mar 8, 2013 at 11:40 AM, Rob Leclerc robert.lecl...@gmail.comwrote: I'm have the same error when I run library_create_library.py with the documentation laid out in the galaxy-central/tip//scrips/api/README from: /galaxy-dist/scripts/api/ *% ./library_create_library.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/libraries api_test 'API Test Library'* Traceback (most recent call last): File ./library_create_library.py, line 19, in module submit( sys.argv[1], sys.argv[2], data ) File ./common.py, line 100, in submit r = post( api_key, url, data ) File ./common.py, line 42, in post return simplejson.loads( urllib2.urlopen( req ).read() ) File build/bdist.macosx-10.6-intel/egg/simplejson/__init__.py, line 384, in loads File build/bdist.macosx-10.6-intel/egg/simplejson/decoder.py, line 402, in decode File build/bdist.macosx-10.6-intel/egg/simplejson/decoder.py, line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) -Rob On Fri, Mar 8, 2013 at 11:23 AM, Sytchev, Ilya isytc...@hsph.harvard.edu wrote: JSONDecodeError will be raised if the given JSON document is not valid. http://simplejson.readthedocs.org/en/latest/ Ilya From: Rob Leclerc robert.lecl...@gmail.com Date: Friday, March 8, 2013 12:01 AM To: galaxy-...@bx.psu.edu galaxy-...@bx.psu.edu Subject: [galaxy-dev] BioBlend: Problem Running Example file I had trouble running blend4j, so I tried to jump into python (a language I have little experience with). I tried running the example run_import_workflow.py *% python run_imported_workflow.py http://localhost:80808c25bc83f6f9e4001dd21eb7b64f063f * but I get an error: Initiating Galaxy connection Importing workflow Creating data library 'Imported data for API demo' Traceback (most recent call last): File run_imported_workflow.py, line 53, in module library_dict = gi.libraries.create_library(library_name) File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/libraries/__init__.py, line 27, in create_library File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/client.py, line 53, in _post File build/bdist.macosx-10.6-intel/egg/bioblend/galaxy/__init__.py, line 132, in make_post_request File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-1.1.0-py2.7.egg/requests/models.py, line 604, in json return json.loads(self.text or self.content) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/__init__.py, line 454, in loads return _default_decoder.decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 374, in decode obj, end = self.raw_decode(s) File /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/simplejson-3.1.0-py2.7-macosx-10.6-intel.egg/simplejson/decoder.py, line 393, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Is there anything I am missing from the stock configuration which would cause this not to run out of the box? Cheers, Rob -- Rob Leclerc, PhD http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.lecl...@aya.yale.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
