Re: [galaxy-dev] Seeking Job Metrics Clarity

2016-02-26 Thread Ziphozakhe Mashologu 
Hi Nate

Thanks for the response.

It was indeed part of the problem. I needed collectl installed on the
individual cluster instances.(job runners)

Seems to be working properly now. I plan to work on using the metrics to
dynamical & intelligently choose job destination of mapping jobs for
example.

Thanks
Z
On 26 Feb 2016 15:21, "Nate Coraor"  wrote:

> On Wed, Feb 24, 2016 at 5:37 AM, Ziphozakhe Mashologu 
> wrote:
>
>> Hi All
>>
>> I have enabled job metrics on a local galaxy install, with latest code
>> from dev branch. I have " " uncommented in job_metrics_conf.xml
>> file:
>>
>> Following are the errors in the logs:
>>
>> galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job
>> properties for plugin
>> > 0x7ff0b6af3a50>
>> Traceback (most recent call last):
>>   File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in
>> collect_properties
>> properties = plugin.job_properties( job_id, job_directory )
>>   File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py",
>> line 105, in job_properties
>> raise Exception( message )
>> Exception: Failed to find collectl log in directory
>>
>> It seems like collectl is looking for the following files:
>>
>> ['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json',
>> 'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC',
>> 'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots',
>> 'metadata_results_HistoryDatasetAssociation_2_33gTDj',
>> '__instrument_core_epoch_start', 'galaxy_2.sh',
>> 'metadata_out_HistoryDatasetAssociation_2_G5bZYP',
>> '__instrument_core_epoch_end', '__instrument_meminfo_meminfo',
>> '__instrument_uname_uname',
>> 'metadata_override_HistoryDatasetAssociation_2_krldDM',
>> 'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t',
>> '__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec']
>>
>> Which are suppose to exist under
>> galaxy/database/job_working_directory/000 job directory.
>>
>> Has anybody faced anything similar issue or is it a configuration issue,
>> or perhaps a known issue with job_metrics (collectl)?
>>
>> BTW: I installed collectl packages on the host.
>>
>
> Hi Zipho,
>
> If this Galaxy instance runs jobs on a cluster, you would also need
> collectl to be installed on the cluster.
>
> --nate
>
>
>
>>
>> Regards
>> Zipho
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Seeking Job Metrics Clarity

2016-02-26 Thread Nate Coraor 
On Wed, Feb 24, 2016 at 5:37 AM, Ziphozakhe Mashologu 
wrote:

> Hi All
>
> I have enabled job metrics on a local galaxy install, with latest code
> from dev branch. I have " " uncommented in job_metrics_conf.xml
> file:
>
> Following are the errors in the logs:
>
> galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job
> properties for plugin
>  0x7ff0b6af3a50>
> Traceback (most recent call last):
>   File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in
> collect_properties
> properties = plugin.job_properties( job_id, job_directory )
>   File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py",
> line 105, in job_properties
> raise Exception( message )
> Exception: Failed to find collectl log in directory
>
> It seems like collectl is looking for the following files:
>
> ['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json',
> 'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC',
> 'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots',
> 'metadata_results_HistoryDatasetAssociation_2_33gTDj',
> '__instrument_core_epoch_start', 'galaxy_2.sh',
> 'metadata_out_HistoryDatasetAssociation_2_G5bZYP',
> '__instrument_core_epoch_end', '__instrument_meminfo_meminfo',
> '__instrument_uname_uname',
> 'metadata_override_HistoryDatasetAssociation_2_krldDM',
> 'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t',
> '__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec']
>
> Which are suppose to exist under galaxy/database/job_working_directory/000
> job directory.
>
> Has anybody faced anything similar issue or is it a configuration issue,
> or perhaps a known issue with job_metrics (collectl)?
>
> BTW: I installed collectl packages on the host.
>

Hi Zipho,

If this Galaxy instance runs jobs on a cluster, you would also need
collectl to be installed on the cluster.

--nate



>
> Regards
> Zipho
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Seeking Job Metrics Clarity

2016-02-24 Thread Ziphozakhe Mashologu 
Hi All

I have enabled job metrics on a local galaxy install, with latest code from
dev branch. I have " " uncommented in job_metrics_conf.xml file:

Following are the errors in the logs:

galaxy.jobs.metrics ERROR 2016-02-24 12:28:08,741 Failed to collect job
properties for plugin

Traceback (most recent call last):
  File "dev/galaxy/lib/galaxy/jobs/metrics/__init__.py", line 101, in
collect_properties
properties = plugin.job_properties( job_id, job_directory )
  File "dev/galaxy/lib/galaxy/jobs/metrics/instrumenters/collectl.py", line
105, in job_properties
raise Exception( message )
Exception: Failed to find collectl log in directory

It seems like collectl is looking for the following files:

['__instrument_collectl_pid', '__instrument_env_vars', 'galaxy.json',
'metadata_kwds_HistoryDatasetAssociation_2_1rh1WC',
'set_metadata_N05Q24.py', '__instrument_core_galaxy_slots',
'metadata_results_HistoryDatasetAssociation_2_33gTDj',
'__instrument_core_epoch_start', 'galaxy_2.sh',
'metadata_out_HistoryDatasetAssociation_2_G5bZYP',
'__instrument_core_epoch_end', '__instrument_meminfo_meminfo',
'__instrument_uname_uname',
'metadata_override_HistoryDatasetAssociation_2_krldDM',
'set_metadata_lpIXS5.py', 'metadata_in_HistoryDatasetAssociation_2_iiYW0t',
'__instrument_cpuinfo_cpuinfo', 'tool_script.sh', 'galaxy_2.ec']

Which are suppose to exist under galaxy/database/job_working_directory/000
job directory.

Has anybody faced anything similar issue or is it a configuration issue, or
perhaps a known issue with job_metrics (collectl)?

BTW: I installed collectl packages on the host.

Regards
Zipho
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/