Re: [galaxy-dev] Displaying tool select list according to user

2017-09-21 Thread SAPET, Frederic 
Hi,

I have created 3 issues!

Use different .loc files depending on user's parameters
https://github.com/galaxyproject/galaxy/issues/4666

Access DataLibrary files from tool form
https://github.com/galaxyproject/galaxy/issues/4667

File upload enhancement
https://github.com/galaxyproject/galaxy/issues/4668

thank you Peter for pointing me out http://www.sequenceserver.com/

and of course as I also develop and maintain tools, I understand that all 
‘craziest’ ideas cannot become a reality.

Fred


De : Peter Cock [mailto:p.j.a.c...@googlemail.com]
Envoyé : jeudi 21 septembre 2017 14:46
À : SAPET, Frederic 
Cc : Laure QUINTRIC ; Galaxy-dev 

Objet : Re: [galaxy-dev] Displaying tool select list according to user



On Thu, Sep 21, 2017 at 12:50 PM, SAPET, Frederic 
mailto:frederic.sa...@biogemma.com>> wrote:
Hi Peter,

Good idea, I will keep that in mind for other needs.
I'm not sure that it will be suitable for what I want to do. Users will have to 
first import database to their history (yet another step as they ask me for 
less, even the upload process is sometimes one step they would like to avoid !)

But it could be a way of achieving what I want, if Galaxy could get files (or 
.loc files) from a data library, within the tool form itself !

Maybe I will create an issue on the Galaxy git in order to discuss about that. 
I will be glad to help if something can be done.

Good idea! I wonder if the Galaxy UI team can think of a good way to integrate 
this, which would make Shared Data library files available to any tool without 
the import to current history step? Can you reply with a link to the new GitHub 
issue for this?

Maybe some explanations about the fact that my users ask me for "less steps".
I really want Galaxy to be the main tool of our analysis platform.
Right now I still maintain a plain old perl/php/CGI web interface. With a blast 
tool (among a lot of others tools). This proprietary Blast interface is really 
like you will find on NCBI or EBI sites. And you can quickly paste your 
sequences, pick up a database, a Blast algorythm and then launch. Few seconds 
later I have a 'htmlized' Blast report, with a nice picture.
This use case (is this sequence a part of this database ?) is still faster on 
this plain old interface. And I afraid that I will have to keep it until they 
found Galaxy better than it...

We used to run wwwBLAST locally for similar reasons. If I was trying to setup 
up something similar now, I'd look at http://www.sequenceserver.com/ for this - 
but making the Galaxy alternative easier to use would be neater.

Maybe it could be another (or two) idea(s)  : the user can drag and drop his 
file within the tool form, Galaxy will handle the creation of this file within 
the history and then launch the tool. Or user can copy/paste a sequence within 
a text-area, and Galaxy will handle the creation of a file from these data 
within the history and then launch the tool.
This skips upload process.

The Galaxy upload does now support drag and drop, so again in principle the 
Galaxy team might be able to extend the tool interface as well. The tool 
input's expected file formats could be used to guide the data-type sniffer 
and/or short list the choices shown to the use.

As well or instead, I could imagine having an upload icon associated with each 
tool input (again, creating a history item and selecting it for you)? This 
could take advantage of the fact that each tool's input parameter specifies the 
expected input file formats, so that might be mapped to known file extensions 
for a file picker?


thank you again for sharing your ideas, it helps!

Fred

These are great usability ideas you are putting forward here - as a Galaxy Tool 
Author they sound positive, but I cannot comment on how complicated it would be 
for the Galaxy UI team. Or indeed if the UI would actually be practical - or 
just overly complicate things which is always a risk.

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] resubmission on out of memory

2017-09-21 Thread John Chilton 
Something like this is possible with some caveats. It is possible to
detect memory and walltime errors - but not based on regex in tools
but instead by the job runner. So the SLURM runner implements
detection of out of memory errors and timeout I think - I don't think
most of the other runners do.

When I started hacking on this feature, there was no documentation for
it and I wanted to understand how it worked and verify that it worked
so I wrote a test case. The problem is the test case tests a bunch of
different features all at once - so it will be a lot to walk through
and you will need to understand dynamic job destinations and such:

https://github.com/galaxyproject/galaxy/blob/dev/test/integration/resubmission_job_conf.xml
https://github.com/galaxyproject/galaxy/commit/0559cff6e94b250ddd98275b119ab51b36491e34

That said let me see if I can come up with a simple example:








  



  
  

https://github.com/galaxyproject/galaxy/issues/3107). It is a medium
priority for me - so I may get to it at some point. This sort of thing
is important when scaling up analyses.

-John



On Tue, Sep 19, 2017 at 4:26 PM, Matthias Bernt  wrote:
> Dear list,
>
> I recall that its possible to configure a tool can such that out of memory
> conditions (and run time) can be recognized (by regexp matching on
> stadout/stderr). Can this be used to trigger job resubmission on the
> cluster?
>
> Could someone please point me to some kind of documentation, if this is the
> case?
>
> Best,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig
> Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Displaying tool select list according to user

2017-09-21 Thread Peter Cock 
On Thu, Sep 21, 2017 at 12:50 PM, SAPET, Frederic <
frederic.sa...@biogemma.com> wrote:

> Hi Peter,
>
> Good idea, I will keep that in mind for other needs.
> I'm not sure that it will be suitable for what I want to do. Users will
> have to first import database to their history (yet another step as they
> ask me for less, even the upload process is sometimes one step they would
> like to avoid !)


>
But it could be a way of achieving what I want, if Galaxy could get files
> (or .loc files) from a data library, within the tool form itself !


>
Maybe I will create an issue on the Galaxy git in order to discuss about
> that. I will be glad to help if something can be done.
>

Good idea! I wonder if the Galaxy UI team can think of a good way to
integrate this, which would make Shared Data library files available to any
tool without the import to current history step? Can you reply with a link
to the new GitHub issue for this?


> Maybe some explanations about the fact that my users ask me for "less
> steps".
> I really want Galaxy to be the main tool of our analysis platform.
> Right now I still maintain a plain old perl/php/CGI web interface. With a
> blast tool (among a lot of others tools). This proprietary Blast interface
> is really like you will find on NCBI or EBI sites. And you can quickly
> paste your sequences, pick up a database, a Blast algorythm and then
> launch. Few seconds later I have a 'htmlized' Blast report, with a nice
> picture.
> This use case (is this sequence a part of this database ?) is still faster
> on this plain old interface. And I afraid that I will have to keep it until
> they found Galaxy better than it...
>

We used to run wwwBLAST locally for similar reasons. If I was trying to
setup up something similar now, I'd look at http://www.sequenceserver.com/
for this - but making the Galaxy alternative easier to use would be neater.


> Maybe it could be another (or two) idea(s)  : the user can drag and drop
> his file within the tool form, Galaxy will handle the creation of this file
> within the history and then launch the tool. Or user can copy/paste a
> sequence within a text-area, and Galaxy will handle the creation of a file
> from these data within the history and then launch the tool.
> This skips upload process.
>

The Galaxy upload does now support drag and drop, so again in principle the
Galaxy team might be able to extend the tool interface as well. The tool
input's expected file formats could be used to guide the data-type sniffer
and/or short list the choices shown to the use.

As well or instead, I could imagine having an upload icon associated with
each tool input (again, creating a history item and selecting it for you)?
This could take advantage of the fact that each tool's input parameter
specifies the expected input file formats, so that might be mapped to known
file extensions for a file picker?


>
> thank you again for sharing your ideas, it helps!
>
> Fred
>

These are great usability ideas you are putting forward here - as a Galaxy
Tool Author they sound positive, but I cannot comment on how complicated it
would be for the Galaxy UI team. Or indeed if the UI would actually be
practical - or just overly complicate things which is always a risk.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Displaying tool select list according to user

2017-09-21 Thread Gildas Le Corguillé 
As it was already mention, about the filtering we presented at Baltimore this 
project : https://github.com/abims-sbr/galaxy-blastprivate 


And about the report, we also have something about that 
:https://github.com/abims-sbr/galaxy-blastvisu 


But shame on us, there is a lot of dust on them.
Although, if we can build a little group around this project, we are stil 
interesting in. Our goal was also to drop out our W3Blast server. 

Gildas

-
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr 

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org 

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: +33 2 98 29 23 81
--



> Le 21 sept. 2017 à 13:50, SAPET, Frederic  a 
> écrit :
> 
> Hi Peter,
> 
> Good idea, I will keep that in mind for other needs.
> I'm not sure that it will be suitable for what I want to do. Users will have 
> to first import database to their history (yet another step as they ask me 
> for less, even the upload process is sometimes one step they would like to 
> avoid !)
> 
> But it could be a way of achieving what I want, if Galaxy could get files (or 
> .loc files) from a data library, within the tool form itself !
> 
> Maybe I will create an issue on the Galaxy git in order to discuss about 
> that. I will be glad to help if something can be done.
> 
> Maybe some explanations about the fact that my users ask me for "less steps".
> I really want Galaxy to be the main tool of our analysis platform. 
> Right now I still maintain a plain old perl/php/CGI web interface. With a 
> blast tool (among a lot of others tools). This proprietary Blast interface is 
> really like you will find on NCBI or EBI sites. And you can quickly paste 
> your sequences, pick up a database, a Blast algorythm and then launch. Few 
> seconds later I have a 'htmlized' Blast report, with a nice picture.
> This use case (is this sequence a part of this database ?) is still faster on 
> this plain old interface. And I afraid that I will have to keep it until they 
> found Galaxy better than it...
> 
> Maybe it could be another (or two) idea(s)  : the user can drag and drop his 
> file within the tool form, Galaxy will handle the creation of this file 
> within the history and then launch the tool. Or user can copy/paste a 
> sequence within a text-area, and Galaxy will handle the creation of a file 
> from these data within the history and then launch the tool.
> This skips upload process.
> 
> thank you again for sharing your ideas, it helps!
> 
> Fred
> 
> -Message d'origine-
> De : Peter Cock [mailto:p.j.a.c...@googlemail.com] 
> Envoyé : mercredi 20 septembre 2017 12:37
> À : SAPET, Frederic 
> Cc : Laure QUINTRIC ; Galaxy-dev 
> 
> Objet : Re: [galaxy-dev] Displaying tool select list according to user
> 
> Have you tried creating the BLAST databases as files in your history (by 
> running makeblastdb within Galaxy), and then importing them into a shared 
> data library (which can then be configured with role based permissions)?
> 
> Peter
> 
> 
> On Wed, Sep 20, 2017 at 9:39 AM, SAPET, Frederic 
>  wrote:
>> Hello Laure
>> 
>> 
>> 
>> this idea cannot be applied in my use case.
>> 
>> indeed I don’t want that users can choose which data to display. I 
>> want to hide some data based on their name/group/address (anything 
>> that can be useful to clasify my users).
>> 
>> 
>> 
>> Thank you for sharing your ideas :)
>> 
>> 
>> 
>> Fred
>> 
>> De : Laure QUINTRIC [mailto:laure.quint...@ifremer.fr]
>> Envoyé : mercredi 13 septembre 2017 13:54 À : Peter Cock 
>> ; SAPET, Frederic 
>> 
>> 
>> 
>> Cc : Galaxy-dev 
>> Objet : Re: [galaxy-dev] Displaying tool select list according to user
>> 
>> 
>> 
>> A workaround would be to do this :
>> 
>> - first, create a tool which will output a tabular loc file with all 
>> the custom banks for the connected user.
>> - then add in the ncbi_blast wrapper a select list with the attribute 
>> from_dataset to load the loc file generated by the previous tool  : 
>> see the example here :
>> https://docs.galaxyproject.org/en/latest/dev/schema.html#from-dataset
>> 
>> I have something like this in the ncbi_blast tool:
>>> label="My custom dbs"/>
>>> multiple="true" display="checkboxes" label="Select our custom db">
>>
>>
>>
>>
>>
>> 
>> and I set in param userdb_loc_file the loc file which is in my history 
>> and looks like this :
>> NCBI_ntNCBI_nt
>> 2017-05-19/path/to/biobanks/n/NCBI_nt/current/NCBI_nt/NCBI_nt
>> 
>> Laure
>> 
>> Le 12

Re: [galaxy-dev] Displaying tool select list according to user

2017-09-21 Thread SAPET, Frederic 
Hi Peter,

Good idea, I will keep that in mind for other needs.
I'm not sure that it will be suitable for what I want to do. Users will have to 
first import database to their history (yet another step as they ask me for 
less, even the upload process is sometimes one step they would like to avoid !)

But it could be a way of achieving what I want, if Galaxy could get files (or 
.loc files) from a data library, within the tool form itself !

Maybe I will create an issue on the Galaxy git in order to discuss about that. 
I will be glad to help if something can be done.

Maybe some explanations about the fact that my users ask me for "less steps".
I really want Galaxy to be the main tool of our analysis platform. 
Right now I still maintain a plain old perl/php/CGI web interface. With a blast 
tool (among a lot of others tools). This proprietary Blast interface is really 
like you will find on NCBI or EBI sites. And you can quickly paste your 
sequences, pick up a database, a Blast algorythm and then launch. Few seconds 
later I have a 'htmlized' Blast report, with a nice picture.
This use case (is this sequence a part of this database ?) is still faster on 
this plain old interface. And I afraid that I will have to keep it until they 
found Galaxy better than it...

Maybe it could be another (or two) idea(s)  : the user can drag and drop his 
file within the tool form, Galaxy will handle the creation of this file within 
the history and then launch the tool. Or user can copy/paste a sequence within 
a text-area, and Galaxy will handle the creation of a file from these data 
within the history and then launch the tool.
This skips upload process.

thank you again for sharing your ideas, it helps!

Fred

-Message d'origine-
De : Peter Cock [mailto:p.j.a.c...@googlemail.com] 
Envoyé : mercredi 20 septembre 2017 12:37
À : SAPET, Frederic 
Cc : Laure QUINTRIC ; Galaxy-dev 

Objet : Re: [galaxy-dev] Displaying tool select list according to user

Have you tried creating the BLAST databases as files in your history (by 
running makeblastdb within Galaxy), and then importing them into a shared data 
library (which can then be configured with role based permissions)?

Peter


On Wed, Sep 20, 2017 at 9:39 AM, SAPET, Frederic  
wrote:
> Hello Laure
>
>
>
> this idea cannot be applied in my use case.
>
> indeed I don’t want that users can choose which data to display. I 
> want to hide some data based on their name/group/address (anything 
> that can be useful to clasify my users).
>
>
>
> Thank you for sharing your ideas :)
>
>
>
> Fred
>
> De : Laure QUINTRIC [mailto:laure.quint...@ifremer.fr]
> Envoyé : mercredi 13 septembre 2017 13:54 À : Peter Cock 
> ; SAPET, Frederic 
> 
>
>
> Cc : Galaxy-dev 
> Objet : Re: [galaxy-dev] Displaying tool select list according to user
>
>
>
> A workaround would be to do this :
>
> - first, create a tool which will output a tabular loc file with all 
> the custom banks for the connected user.
> - then add in the ncbi_blast wrapper a select list with the attribute 
> from_dataset to load the loc file generated by the previous tool  : 
> see the example here :
> https://docs.galaxyproject.org/en/latest/dev/schema.html#from-dataset
>
> I have something like this in the ncbi_blast tool:
>  label="My custom dbs"/>
>  multiple="true" display="checkboxes" label="Select our custom db">
> 
> 
> 
> 
> 
>
> and I set in param userdb_loc_file the loc file which is in my history 
> and looks like this :
> NCBI_ntNCBI_nt
> 2017-05-19/path/to/biobanks/n/NCBI_nt/current/NCBI_nt/NCBI_nt
>
> Laure
>
> Le 12/09/2017 à 17:25, Peter Cock a écrit :
>
> Thanks Fred,
>
>
>
> That's the previous discussion I was trying to find -
>
> sadly no clear solution but some ideas to explore.
>
>
>
> It does seem filtering example.loc by user or role
>
> is not a niche request. I wonder if this could be
>
> linked to the Galaxy data library permissions
>
> structure as a way to manage the access rights?
>
>
>
> Peter
>
>
>
> On Tue, Sep 12, 2017 at 4:20 PM, SAPET, Frederic 
>  wrote:
>
> Hi Laure,
>
>
>
> There are others users (including me! ) that need this kind of function :
>
> http://dev.list.galaxyproject.org/Blast-db-permission-td4670440.html
>
>
>
> @Galaxy team:
>
> Do we need to add an issue in GitHub ?
>
> Do you think that it could be enabled one day ?
>
>
>
> I think this something that could be nice to develop !
>
> I would be glad to help.
>
>
>
> Thank a lot
>
>
>
> Fred
>
>
>
> De : galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] De 
> la part de Peter Cock Envoyé : mardi 12 septembre 2017 14:56 À : Laure 
> QUINTRIC  Cc : Galaxy-dev 
> 
> Objet : Re: [galaxy-dev] Displaying tool select list according to user
>
>
>
> Hi Laure,
>
>
>
> I don't think there is any easy to use mechanism for this in Galaxy.
>
> Unfortunately Galaxy loads all the *.loc files at startup and treats
>
> them as global