[galaxy-dev] Galaxy Feature Request: Input naming deficiencies

[Ziphozakhe Mashologu]

Hi All

In using Galaxy, I would like to be able to keep track of the input dataset
name down the analysis chain.

This is a similar if not the same issue raised by Eric in
https://github.com/galaxyproject/galaxy/issues/2752, to which he proposes
in the context of a workflow that, "the rename dataset action should
support usage of all the dataset names from Input Dataset elements, rather
than just input files to the particular tool." Please see the issue for
more details.

There are other similar issues raise scattered all over regarding the same
issue:

Enhancements:
1. Ability to name datasets using the element identifier:
https://github.com/galaxyproject/galaxy/issues/2006
2. Ability to define the collection name in a workflow:
https://github.com/galaxyproject/galaxy/issues/2398
3. Name datasets according to both collection name and element identifier:
https://github.com/galaxyproject/galaxy/issues/2140,
https://github.com/galaxyproject/galaxy/issues/2023

Bug Fixes:
1. Renaming using the input from paired dataset collections:
https://github.com/galaxyproject/galaxy/issues/1675,
https://github.com/galaxyproject/galaxy/issues/1686

I have created a github issue regarding the above, in effort to try and
consolidate all these related above issues.

https://github.com/galaxyproject/galaxy/issues/2980

Can we use the above issue to discuss the best approach on these issues,
point out work being done on the area and perhaps define clear milestones
for each?

Galaxy Devs please provide me some guidance to best tackle this.
(particularly of tracking the input dataset names along the analysis chain).

Thanks,
Zipho
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Seeking Job Metrics Clarity

[Ziphozakhe Mashologu]

Hi Nate

Thanks for the response.

It was indeed part of the problem. I needed collectl installed on the
individual cluster instances.(job runners)

Seems to be working properly now. I plan to work on using the metrics to
dynamical & intelligently choose job destination of mapping jobs for
example.

Thanks
Z
On 26 Feb 2016 15:21, "Nate Coraor" <n...@bx.psu.edu> wrote:

> On Wed, Feb 24, 2016 at 5:37 AM, Ziphozakhe Mashologu <zi...@sanbi.ac.za>
> wrote:
>
>> Hi All
>>
>> I have enabled job metrics on a local galaxy install, with latest code
>> from dev branch. I have " " uncommented in job_metrics_conf.xml
>> file:
>>
>> Following are the errors in the logs:
>>
>> galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job
>> properties for plugin
>> > 0x7ff0b6af3a50>
>> Traceback (most recent call last):
>>   File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in
>> collect_properties
>> properties = plugin.job_properties( job_id, job_directory )
>>   File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py",
>> line 105, in job_properties
>> raise Exception( message )
>> Exception: Failed to find collectl log in directory
>>
>> It seems like collectl is looking for the following files:
>>
>> ['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json',
>> 'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC',
>> 'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots',
>> 'metadata_results_HistoryDatasetAssociation_2_33gTDj',
>> '__instrument_core_epoch_start', 'galaxy_2.sh',
>> 'metadata_out_HistoryDatasetAssociation_2_G5bZYP',
>> '__instrument_core_epoch_end', '__instrument_meminfo_meminfo',
>> '__instrument_uname_uname',
>> 'metadata_override_HistoryDatasetAssociation_2_krldDM',
>> 'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t',
>> '__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec']
>>
>> Which are suppose to exist under
>> galaxy/database/job_working_directory/000 job directory.
>>
>> Has anybody faced anything similar issue or is it a configuration issue,
>> or perhaps a known issue with job_metrics (collectl)?
>>
>> BTW: I installed collectl packages on the host.
>>
>
> Hi Zipho,
>
> If this Galaxy instance runs jobs on a cluster, you would also need
> collectl to be installed on the cluster.
>
> --nate
>
>
>
>>
>> Regards
>> Zipho
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Seeking Job Metrics Clarity

[Ziphozakhe Mashologu]

Hi All

I have enabled job metrics on a local galaxy install, with latest code from
dev branch. I have " " uncommented in job_metrics_conf.xml file:

Following are the errors in the logs:

galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job
properties for plugin

Traceback (most recent call last):
  File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in
collect_properties
properties = plugin.job_properties( job_id, job_directory )
  File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py", line
105, in job_properties
raise Exception( message )
Exception: Failed to find collectl log in directory

It seems like collectl is looking for the following files:

['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json',
'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC',
'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots',
'metadata_results_HistoryDatasetAssociation_2_33gTDj',
'__instrument_core_epoch_start', 'galaxy_2.sh',
'metadata_out_HistoryDatasetAssociation_2_G5bZYP',
'__instrument_core_epoch_end', '__instrument_meminfo_meminfo',
'__instrument_uname_uname',
'metadata_override_HistoryDatasetAssociation_2_krldDM',
'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t',
'__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec']

Which are suppose to exist under galaxy/database/job_working_directory/000
job directory.

Has anybody faced anything similar issue or is it a configuration issue, or
perhaps a known issue with job_metrics (collectl)?

BTW: I installed collectl packages on the host.

Regards
Zipho
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Working on the admin interface: JS vs Mako

[Ziphozakhe Mashologu]

Super, thanks Martin. I will dive into these links soon and please excuse
any obvious questions from me in the process.

Thanks,
Z

On Mon, Feb 1, 2016 at 10:36 PM, Martin Čech <mar...@bx.psu.edu> wrote:

> Hi,
>
> Galaxy is implementing frontend with the help of these libraries:
> https://github.com/galaxyproject/galaxy/tree/dev/client/galaxy/scripts/libs
>
> Notably the MV-paradigm is being implemented in Backbone.
>
> Good examples of applications/pieces that are written in JS are at
> https://github.com/galaxyproject/galaxy/tree/dev/client/galaxy/scripts/mvc
>
> The easiest one to look into is probably the shortest one: `citation`. If
> you want to look into full backbone superpower check out `history`
>
> Martin
>
>
> On Mon, Feb 1, 2016 at 3:26 PM Ziphozakhe Mashologu <zi...@sanbi.ac.za>
> wrote:
>
>> Hi All
>>
>> In terms of the future, are we using or considering using any JS SPA
>> frameworks or the route is plain JavaScript?
>>
>> I ask as we aim to satisfy our pursued RESTful architecture.
>>
>> Thanks
>> Zipho
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 1, 2016 at 7:30 PM, Peter van Heusden <p...@sanbi.ac.za>
>> wrote:
>>
>>> Yeah I noticed that about the DependencyResolversView.
>>>
>>> In terms of writing JS - where's a good example to look at? I tried
>>> digging around the client/ directories but couldn't make sense of how it
>>> works.
>>>
>>> Thanks,
>>> Peter
>>>
>>> On 1 February 2016 at 16:48, John Chilton <jmchil...@gmail.com> wrote:
>>>
>>>> Peter,
>>>>
>>>>   We would like to replace all the mako with JS, if I was going to put
>>>> a bunch of effort into admin pages I'd start by reworking what was
>>>> there to use JavaScript and the API. That is me however, I have lots
>>>> of time to put into Galaxy fundamentals and refactoring. This is more
>>>> work and I am happy to field PRs that modify mako for the time
>>>> being.(https://github.com/galaxyproject/galaxy/pull/1632/files looks
>>>> great for instance).
>>>>
>>>>   A middle ground between rewriting everything and still depending on
>>>> mako is to dump dictionaries to JavaScript in the mako and render
>>>> things in JS. This will make it easier to adapt your changes once the
>>>> pages are ultimately rewritten to JavaScript SPAs. In the case 1632 -
>>>> that DependencyResolversView class produces dictionaries that can be
>>>> dumped as JSON easily into the page and JavaScript could be used to
>>>> build up the tables in
>>>> templates/webapps/tool_shed/repository/common.mako.
>>>>
>>>> -John
>>>>
>>>>
>>>> On Mon, Feb 1, 2016 at 11:51 AM, Peter van Heusden <p...@sanbi.ac.za>
>>>> wrote:
>>>> > Hi there
>>>> >
>>>> > Zipho and I'd would like to make some changes to the admin pages,
>>>> including
>>>> > the tool info page and the local data page. These are currently
>>>> rendered
>>>> > from mako templates - should we change these or is the interface
>>>> going to be
>>>> > rewritten in client side / javascript in the near(ish) future?
>>>> >
>>>> > Thanks,
>>>> > Peter
>>>> >
>>>> > ___
>>>> > Please keep all replies on the list by using "reply all"
>>>> > in your mail client.  To manage your subscriptions to this
>>>> > and other Galaxy lists, please use the interface at:
>>>> >   https://lists.galaxyproject.org/
>>>> >
>>>> > To search Galaxy mailing lists use the unified search at:
>>>> >   http://galaxyproject.org/search/mailinglists/
>>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Working on the admin interface: JS vs Mako

[Ziphozakhe Mashologu]

Hi All

In terms of the future, are we using or considering using any JS SPA
frameworks or the route is plain JavaScript?

I ask as we aim to satisfy our pursued RESTful architecture.

Thanks
Zipho







On Mon, Feb 1, 2016 at 7:30 PM, Peter van Heusden  wrote:

> Yeah I noticed that about the DependencyResolversView.
>
> In terms of writing JS - where's a good example to look at? I tried
> digging around the client/ directories but couldn't make sense of how it
> works.
>
> Thanks,
> Peter
>
> On 1 February 2016 at 16:48, John Chilton  wrote:
>
>> Peter,
>>
>>   We would like to replace all the mako with JS, if I was going to put
>> a bunch of effort into admin pages I'd start by reworking what was
>> there to use JavaScript and the API. That is me however, I have lots
>> of time to put into Galaxy fundamentals and refactoring. This is more
>> work and I am happy to field PRs that modify mako for the time
>> being.(https://github.com/galaxyproject/galaxy/pull/1632/files looks
>> great for instance).
>>
>>   A middle ground between rewriting everything and still depending on
>> mako is to dump dictionaries to JavaScript in the mako and render
>> things in JS. This will make it easier to adapt your changes once the
>> pages are ultimately rewritten to JavaScript SPAs. In the case 1632 -
>> that DependencyResolversView class produces dictionaries that can be
>> dumped as JSON easily into the page and JavaScript could be used to
>> build up the tables in
>> templates/webapps/tool_shed/repository/common.mako.
>>
>> -John
>>
>>
>> On Mon, Feb 1, 2016 at 11:51 AM, Peter van Heusden 
>> wrote:
>> > Hi there
>> >
>> > Zipho and I'd would like to make some changes to the admin pages,
>> including
>> > the tool info page and the local data page. These are currently rendered
>> > from mako templates - should we change these or is the interface going
>> to be
>> > rewritten in client side / javascript in the near(ish) future?
>> >
>> > Thanks,
>> > Peter
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/